Household equipment control method and device, equipment and medium

Abstract

The embodiment of the invention provides a household equipment control method and device, equipment and a medium. The method comprises the following steps: acquiring user posture data and historical interaction information of at least one user; determining an interaction intention of the at least one user according to the user posture data and the historical interaction information of the at least one user; determining an execution priority of the interaction intention of the at least one user; determining a target user in the at least one user according to the execution priority of the interaction intention of the at least one user; and determining and controlling target equipment according to the interaction intention of the target user. By optimizing the detection mode of the interaction intentions and determining the execution priorities of the interaction intentions, the execution accuracy of the smart home equipment in response to the multiple interaction intentions is effectively improved. Risks of lagging execution of important interaction intentions and execution conflicting interaction intentions are avoided, and equipment response accuracy and user interaction experience are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smart home, and in particular to a method, device, equipment and medium for controlling home equipment. Background Art

[0002] The core goal of smart home interaction technology is to reduce the user operation cost and improve the scene adaptability. Early systems relied on physical switches for control, and then gradually developed into voice control interactions such as voice assistants, mobile terminal touch control, and preset sensor trigger modes.

[0003] However, when multiple users are in the same space at the same time and multiple interaction intentions are proposed simultaneously, the existing technology can only capture multiple interaction intentions from multiple users, but cannot reasonably determine which interaction intention should be executed first among these multiple interaction intentions. In this scenario, the response of smart home devices to multiple interaction intentions is full of randomness, which may lead to the risk of important interaction intentions being executed later or conflicting interaction intentions being executed when executing multiple interaction intentions issued by multiple users, thereby reducing the user's interaction experience. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method, device, equipment and medium for controlling home equipment that can overcome the above problems or at least partially solve the above problems.

[0005] To solve the above problems, embodiments of the present invention disclose a method for controlling home equipment, the method includes: Obtain user posture data and historical interaction information of at least one user; Determine the interaction intention of the at least one user according to the user posture data and the historical interaction information of the at least one user; Determine the execution priority of the interaction intention of the at least one user; Determine a target user among the at least one user according to the execution priority of the interaction intention of the at least one user; Determine and control a target device according to the interaction intention of the target user.

[0006] Optionally, the obtaining user posture data of at least one user includes: Obtain the original user posture data of the at least one user; For each user among the at least one user, when it is determined that the posture of the user is blocked according to the original user posture data of the user, generate complementary posture data; Use the complementary posture data as the user posture data of the user.

[0007] Optionally, the user posture data includes user body posture data. Obtaining the user posture data and historical interaction information of at least one user includes: Determine the user identity according to the user body posture data of the at least one user; Obtain the historical interaction information corresponding to the user identity.

[0008] Optionally, the historical interaction information includes historical posture data and corresponding historical interaction intentions. Determining the interaction intention of the at least one user according to the user posture data of the at least one user and the historical interaction information includes: Determine the similarity between the user posture data of the at least one user and the historical posture data; Use the historical interaction intention corresponding to the historical posture data with the highest similarity as the interaction intention of the at least one user.

[0009] Optionally, determining the execution priority of the interaction intention of the at least one user includes: Determine the urgency level of the interaction intention of the at least one user; Determine the intention priority of the interaction intention according to the urgency level of the interaction intention of the at least one user.

[0010] Optionally, the user posture data includes user posture images. Determining the execution priority of the interaction intention of the at least one user includes: Obtain the image clarity of the user posture image; Determine the execution priority of the interaction intention of the at least one user according to the image clarity of the user posture image of the at least one user.

[0011] Optionally, determining and controlling the target device according to the interaction intention of the target user includes: Determine at least one candidate device according to the interaction intention of the at least one user; Obtain the location data of the target user and the location data of the candidate device; Determine the distance between the at least one candidate device and the target user according to the location data of the target user and the location data of the candidate device; Use the candidate device with the closest distance as the target device, and control the target device according to the interaction intention of the target user.

[0012] On the other hand, an embodiment of the present invention also discloses a home device control device, and the device includes: A data acquisition module, configured to acquire user posture data and historical interaction information of at least one user; An intention determination module, configured to determine the interaction intention of the at least one user according to the user posture data of the at least one user and the historical interaction information; A priority determination module, configured to determine the execution priority of the interaction intention of the at least one user; A target user determination module, configured to determine a target user among the at least one user according to the execution priority of the interaction intention of the at least one user; A device control module, configured to determine and control a target device according to the interaction intention of the target user.

[0013] Optionally, the data acquisition module includes: A raw data acquisition sub-module, configured to acquire the raw user posture data of the at least one user; A posture completion sub-module, configured to generate completion posture data for each user among the at least one user when it is determined according to the raw user posture data of the user that the posture of the user is occluded; A posture data determination sub-module, configured to use the completion posture data as the user posture data of the user.

[0014] Optionally, the user posture data includes user body posture data, and the data acquisition module includes: A user identity determination sub-module, configured to determine the user identity according to the user body posture data of the at least one user; An interaction information acquisition sub-module, configured to acquire the historical interaction information corresponding to the user identity.

[0015] Optionally, the historical interaction information includes historical posture data and corresponding historical interaction intentions, and the intention determination module includes: A similarity determination sub-module, configured to determine the similarity between the user posture data of the at least one user and the historical posture data; An interaction intention determination sub-module, configured to use the historical interaction intention corresponding to the historical posture data with the highest similarity as the interaction intention of the at least one user.

[0016] Optionally, the priority determination module includes: An intention urgency determination sub-module, configured to determine the intention urgency of the interaction intention of the at least one user; A first priority determination sub-module, configured to determine the intention priority of the interaction intention according to the intention urgency of the interaction intention of the at least one user.

[0017] Optionally, the user posture data includes user posture images, and the priority determination module includes: A clarity determination sub-module, configured to obtain the image clarity of the user posture image; A second-priority determination sub-module, configured to determine the execution priority of the interaction intention of the at least one user according to the image clarity of the user posture images of the at least one user.

[0018] Optionally, the device control module includes: A candidate device determination sub-module, configured to determine at least one candidate device according to the interaction intention of the at least one user; A location data acquisition sub-module, configured to acquire the location data of the target user and the location data of the candidate device; A distance determination sub-module, configured to determine the distance between the at least one candidate device and the target user according to the location data of the target user and the location data of the candidate device; A target device control sub-module, configured to use the candidate device with the shortest distance as the target device, and control the target device according to the interaction intention of the target user.

[0019] Correspondingly, an embodiment of the present invention discloses an electronic device, including: a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, it implements each step of the above embodiment of the home device control method.

[0020] Correspondingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each step of the above embodiment of the home device control method.

[0021] The embodiments of the present invention have the following advantages: obtaining the user posture data and historical interaction information of at least one user, and accurately identifying the true needs of individual users according to the user posture data of at least one user and the historical interaction information, effectively avoiding the problem of misidentifying multiple-person instructions; at the same time, by determining the execution priority of the interaction intention of at least one user, it is possible to effectively avoid the intention conflict between at least one user or between at least one user. Then, according to the execution priority of the interaction intention, the target user is judged, realizing the automatic arbitration of the execution of the interaction intention, optimizing the response logic of the smart home device when facing multiple interaction intentions issued by multiple users. Finally, according to the interaction intention of the target user, the target device is determined and controlled. By optimizing the detection method of the interaction intention and determining the execution priority of the interaction intention, the execution accuracy of the smart home device when responding to multiple interaction intentions is effectively improved. It avoids the risks of lagging execution of important interaction intentions and executing conflicting interaction intentions, and significantly improves the device response accuracy and the user's interaction experience. Description of the Drawings

[0022] Figure 1 is a flowchart of the steps of an embodiment of a method for controlling a home device according to the present invention; Figure 2 is a structural block diagram of an embodiment of a home device control device according to the present invention. Detailed implementation manners

[0023] To make the above objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0024] The core logic of the prior art is passive response. For example: Voice control interaction relies on voice recognition technology to convert user instructions into operation codes executable by the device, but can only process sequential instructions of a single user; Mobile APP interaction sends instructions by the user actively touching the interface, and cannot distinguish the physical positions and intention directivities of multiple users in the same space; Preset sensor interaction is triggered based on environmental thresholds and lacks the ability to dynamically adapt to the personalized needs of users. The essential limitations of these solutions are: single-dimensional interaction, static intention recognition, and mechanical decision-making logic.

[0025] One of the core concepts of the embodiments of the present invention is to accurately determine the interaction intention by recognizing the user's posture and combining historical interaction information, and determine the priority of the interaction intention according to the user's posture data, so that the user's interaction intention transitions from the passive recognition of the system to the system actively detecting the user's posture, and at the same time relying on judging the priority of the interaction intention to assist in improving the accuracy of intention response and enhancing the convenience of user interaction.

[0026] Referring to Figure 1 , a flowchart of the steps of an embodiment of a method for controlling a home device according to the present invention is shown, which may specifically include the following steps: Step 101, obtaining user posture data and historical interaction information of at least one user; In the field of smart home, multiple scenarios will be faced. The specific scenarios may be multiple scenarios of single-user or multi-user including at least one user. In these scenarios, users are generally connected to smart home devices through a device with central control function; And the device with central control function may be a smart central control device such as a central control screen, a smart terminal such as a mobile phone, a tablet, and a cloud server such as a scenario of end-cloud collaboration, etc.; For example, a smart home sensing device transmits data to a device with central control function, and when a user wants to obtain corresponding data, it will be obtained through a device with central control function; Correspondingly, the user's interaction intention for smart home devices will also be forwarded to the corresponding smart home devices through a device with central control function; The execution subject of the method provided by the embodiments of the present invention may also be the above-mentioned device with central control function.

[0027] User posture data can be the human physical state information obtained through multi-modal sensors. This physical state information can include bone key point data, contour data, and action trajectory data. These data help to know the user's current posture and actions, and further infer the user's intention based on the posture and actions at that time. Exemplarily, the bone key point data can be determined by capturing the three-dimensional coordinates of the relevant joints of the user through a 3D depth camera. The action trajectory data can be captured according to devices such as millimeter wave radars, such as the action direction and action speed. The contour data can be further identified according to an infrared thermal imager.

[0028] Historical interaction information can refer to the historical interaction records between the user and relevant devices. For these interaction records, they can be classified and managed according to multiple dimensions. For example, they can be classified according to the corresponding sub-scenes in the home scene, such as classified according to the functions of the rooms. For example, a series of interaction behaviors occurring in the kitchen will be recorded under the data dimension of the kitchen room, and scenes such as the study, living room, and living room are all divided according to the following logic. In addition, they can also be classified according to the user-time dimension. For example, if user A triggers a certain interaction one or more times within a certain time range, it is determined that user A has this historical interaction record within that time range. For example, if user A triggers the operation of turning on the study light multiple times within a time window centered at 16:00 with a half-hour span before and after, it is considered that user A will have the historical interaction record of turning on the study light around 16:00 and record it. The reason for setting this way is to reduce the influence of accidental events and save storage resources. These information can be used for intention determination and priority calculation, and the acquisition method can be extracted from the local storage of the edge computing unit. And through the interaction process, the historical interaction information needs to be continuously updated. In addition, the historical interaction information can also refer to the interaction records of the user for a certain smart home scene.

[0029] In one embodiment, step 101 may include the following sub-steps: Sub-step S11, obtaining the original user posture data of the at least one user; The original user posture data refers to the uncorrected posture data. In some scenarios, due to the failure of the sensor device for obtaining data, or the user being blocked by something, the data acquisition may be incomplete. For such original data, it needs to be corrected before use.

[0030] Sub-step S12, for each user among the at least one user, when it is determined that the posture of the user is blocked according to the original user posture data of the user, generating complementary posture data; When it is confirmed that the posture of the user's presence is blocked, it is necessary to complete the posture data, and the technology used for completion can adopt the HRnet (High-Resolution Network) technology; HRnet is a human pose estimation model widely used in the field of computer vision. Its core features include multi-resolution parallel processing, repeated multi-scale fusion, and high-precision output, etc. When the user is blocked, it means that the user contour data obtained from the camera is missing. At this time, the user's action posture can be completed with the help of action trajectory data, and the user posture model can be constructed with the bone key point data, so as to obtain the completed user posture data.

[0031] The situation where the user's posture is determined to be blocked can include: when the missing amount of the user's posture data reaches a preset threshold, such as when the user's body is blocked in a large area, or when key data is missing, such as the missing hand movement trajectory; the preset threshold can be set by itself according to business needs, and the present invention does not limit this.

[0032] Sub-step S13, using the completed posture data as the user's posture data.

[0033] After the posture is completed, the completed posture data is used as the user's posture data for the next operation; combining HRnet and the fused multi-modal data for occlusion posture repair to improve the abnormal handling ability of the method and better adapt to the smart home scenario.

[0034] In one embodiment, the user's posture data includes the user's body posture data, and step 101 may include the following sub-steps: Sub-step S21, determining the user identity according to the user's body posture data of the at least one user; In the field of smart home, determining the user's identity to obtain the user's personal data is a key step in achieving personalized interaction; the user's body posture data can include: characteristic data such as height information, body shape information, and movement habits, etc. Here, taking one user as an example: the height information and body shape information of the user can be determined by confirming relevant information such as the user's height and shoulder width through the images captured by the camera, and the recognition rate of the identity information can be further improved through data such as preference action data and gait characteristics, making the information corresponding to the user's identity more accurate; the process of determining the user's identity based on the user's body posture data can be to calculate the similarity between the user's height information and body shape information and the relevant data in the historical body posture database, and determine the user's identity according to the similarity. For example, the data in the historical body posture database can include the historical body posture data corresponding to multiple user identities, and each group of historical body posture data includes historical height information, historical body shape information, and historical movement habit information; when determining the user's identity, the similarity between the height information and the historical height information can be calculated first to determine the range of preliminary historical body posture data, and then, within the range of preliminary user identity, the similarity between the body shape information and the historical body shape information can be calculated, and the range of historical body posture data can be further restricted according to the calculated similarity. Finally, the similarity between the movement habit data and the historical movement habit data within the restricted range of historical body posture data is calculated, and the user identity corresponding to the group of body posture data with the highest similarity is used as the user's identity.

[0035] In addition, after each determination of the user's identity, the historical body posture data corresponding to the identity will be updated according to the user's body posture data used this time, making the historical body posture data more in line with the user's characteristics.

[0036] Sub-step S22: Obtain the historical interaction information corresponding to the user identity.

[0037] When retaining the historical interaction information, the historical interaction information can be associated with the user identity to describe the historical interaction information of the user. When judging the user's interaction intention, the corresponding accuracy of judging the interaction intention can be further improved through the user identity to meet the user's personalized interaction intention; for example, for the same interaction intention of 'increasing the air conditioner temperature', the comfortable temperature of user A is 26 degrees Celsius while the comfortable temperature of user B is 24 degrees Celsius. Then, corresponding to their respective interaction intentions, the operation methods during the response are naturally different. Associating the user identity with the historical interaction information enables the historical interaction information to reflect the user's personalized needs and further improves the accuracy of judging the user's interaction intention.

[0038] Step 102: Determine the interaction intention of the at least one user according to the user posture data of the at least one user and the historical interaction information.

[0039] After obtaining the user's posture data, the user's interaction intention can be further determined through similarity calculation, mapping relation table, etc. based on the user's posture data and historical interaction information. The interaction intention can include the target device to be operated, the corresponding operation method, operation parameters, operation strategies, etc.

[0040] Exemplarily, based on the differences in the posture data, different posture data can be classified into relevant posture categories according to a preset classification standard. Each posture category corresponds to a type of interaction intention, and a posture-intention type mapping table is established according to this corresponding relationship. In this way, after obtaining the user's posture data, the type of intention corresponding to the user's current posture can be roughly determined through the preset posture-intention type mapping table. Determine the interaction intention in the historical interaction record according to the user's scenario or the current time point. For example, according to the user's posture data, it is determined that the user's current behavior belongs to the type of intention for controlling a lighting device, and the current time is 17:30. In the historical interaction record, this user usually turns on the kitchen light within the time window of 18:00. At this time, it is determined that the user's interaction intention is to turn on the kitchen light.

[0041] In one embodiment, the historical interaction information includes historical posture data and the corresponding historical interaction intention. Step 102 may include the following sub-steps: Sub-step S31, determine the similarity between the user posture data of the at least one user and the historical posture data.

[0042] In addition to determining the type of the user's interaction intention through the preset posture-intention type mapping table and then determining the user's interaction intention through subsequent judgment logic, the user's interaction intention can also be determined by determining the similarity between the current user posture and the historical posture data. The method for determining the similarity can use methods such as determining the Euclidean distance between data and determining the cosine similarity between data. The present invention does not limit this.

[0043] Sub-step S32, use the historical interaction intention corresponding to the historical posture data with the highest similarity as the interaction intention of the at least one user.

[0044] Use the interaction intention corresponding to the historical posture data with the highest similarity as the user's current interaction intention. Determine the user's interaction intention through posture similarity to improve the accuracy of judging the interaction intention.

[0045] Step 103, determine the execution priority of the interaction intention of the at least one user; In a smart home scenario with at least one user, merely determining the interaction intention cannot cover all home scenarios. For example, in scenarios where the same user has multiple interaction intentions, or where there are multiple users and each user has an interaction intention. Therefore, in addition to determining the interaction intention, it is also necessary to determine the execution priority of the interaction intention.

[0046] In one embodiment, step 103 may include the following sub-steps: Sub-step S41, determining the urgency level of the interaction intention of the at least one user; One way to determine the execution priority of the interaction intention can be through judging the urgency level of the interaction intention. For example, different urgency levels of the interaction intention can be assigned according to its attributes. For example, the urgency level can be divided from level 4 to level 1. As the level number decreases, the urgency level gradually increases. For ordinary query-type intentions such as checking whether a device is turned on, battery level, running duration, querying room temperature, humidity, air quality, and querying device usage records, etc., the lowest urgency level, i.e., level 4, will be assigned; for control-type intentions such as turning on / off devices like lights, air conditioners, TVs, etc., and adjusting device parameters such as temperature, brightness, volume, etc., a relatively low-middle urgency level, i.e., level 3, will be assigned; for intentions related to device fault query and maintenance, such as detecting device anomalies and prompting solutions, updating device software, security patches, restoring factory settings, and reconfiguring the network, etc., which are intentions regarding device faults or device maintenance, a relatively high-middle urgency level, i.e., level 2, will be assigned; for security-type intentions, such as turning on the camera, closing the gas valve, or when it is recognized according to the user's posture data that the user is in an abnormal state, the abnormal state can be immobility or the need to call for help, etc. In this scenario, the intention will be recognized as a call-for-help intention, and for the call-for-help intention, the highest urgency level, i.e., level 1, will be assigned.

[0047] Sub-step S42, determining the intention priority of the interaction intention according to the urgency level of the interaction intention of the at least one user.

[0048] After learning the urgency level of the user's interaction intention, the execution priority will be determined based on the urgency level. More urgent intentions will be executed more preferentially, so as to avoid the risk of causing important interaction intentions to be executed laggingly.

[0049] In one embodiment, the user posture data includes a user posture image, and step 103 may include the following sub-steps: Sub-step S51, obtaining the image clarity of the user posture image; In addition to using the urgency of the interaction intention to judge the execution priority, the execution priority can also be judged by judging the clarity of the user posture image; generally, when the user posture is blurred, the accuracy of the corresponding intention judgment result will also decrease, because the unclear user posture represents an inaccurate interaction intention. For inaccurate interaction intentions, the execution priority of this interaction intention will be reduced.

[0050] Sub-step S52, determine the execution priority of the interaction intention of the at least one user according to the image clarity of the user posture image of the at least one user.

[0051] Determining the execution priority based on the clarity of the user posture image and assigning a lower execution priority to the interaction intention with low credibility corresponding to the unclear posture can further improve the rationality and accuracy of executing the interaction intention.

[0052] In one embodiment, the above two methods can be combined. For example, although the user posture image is unclear, the result of intention matching shows that the user's intention at this time is a distress intention, then this intention will still be given a high execution priority; in addition, the intention execution priority can also be judged according to the user identity; for the user identity, it can be reflected in the way of family members, that is, the identity information can be divided into internal family members such as father, mother, grandfather, grandmother, children and external family members such as visitors, etc. Then, when sensitive intentions such as turning on the electric switch or gas switch from the child are detected synchronously in the same scene, this intention will not be responded to, and an alarm message will be output to other members of the family synchronously; when the intention to adjust the air conditioner parameters from the visitor and the intention to adjust the air conditioner parameters from the internal family members are detected, the air conditioner parameter adjustment instruction from the internal family members will be responded to, and a message will be sent to this member to confirm whether the intention to adjust the air conditioner parameters of the visitor needs to be responded to. Thus, the risk of facing conflicting interaction intentions is avoided, and the accuracy of executing interaction intentions is improved.

[0053] Step 104, determine the target user among the at least one user according to the execution priority of the interaction intention of the at least one user; After obtaining the execution priority of the user's interaction intention, the user with the highest execution priority among the at least one user is used as the target user, and the interaction intention of this user is prepared to be responded to.

[0054] Step 105, determine and control the target device according to the interaction intention of the target user.

[0055] After determining the target user whose intention is to be responded to, by analyzing the interaction intention of the target user, determine the target device to be operated by the interaction intention, and control the target device according to information such as the operation method, operation parameters and operation strategy in the interaction intention.

[0056] In one embodiment, step 105 may include the following sub-steps: Sub-step S61, determining at least one candidate device according to the interaction intention of the at least one user; Generally, the relevant devices that need to respond to the user's interaction intent can be determined. However, if there are multiple devices that can respond to the intent in the scenario, the relevant candidate devices need to be confirmed first. Sub-step S62, obtaining the location data of the target user and the location data of the candidate device; Sub-step S63, determining the distance between the at least one candidate device and the target user according to the location data of the target user and the location data of the candidate device; Sub-step S64, taking the candidate device closest to the target device as the target device, and controlling the target device according to the interaction intention of the target user.

[0057] After determining the distance between the candidate device and the target user, the device closest to the target user is used as the device that responds to the interaction intention. The distance between the target user and the candidate device is used as one of the conditions for determining the target device, and the device closest to the user is intentionally selected as the target device, so that the user's interaction intention can be responded to more promptly.

[0058] Optionally, when a user sets multiple smart scenes, the multiple smart scenes can be parsed into a collection of multiple interaction intentions with temporal features or spatial features, and the degree of association between the smart scenes can be judged according to the historical interaction records. When the degree of association meets the default conditions, the interaction intentions of the multiple smart scenes are connected in series according to the temporal features or spatial features; when parsing the user's interaction intention, if the interaction intention is the starting intention of multiple interaction intentions connected in series, the subsequent intention will be pre-executed; exemplarily, the user sets a smart scene for going home and a smart scene for taking a bath, and multiple historical interaction records show that after the user triggers the smart scene for going home, he will enter the smart scene for taking a bath 20 minutes later, and the triggering method of the smart scene for going home is to open the door after 17:00; if the user's intention to open the door is detected within the time window of 17:00, the subsequent intention will be pre-executed according to the temporal relationship of the interaction intentions after responding to the intention.

[0059] In addition to determining the user's intention using the user gesture data, it is also possible to synchronously receive various interaction information such as the user's voice interaction information, the intelligent terminal device, the interaction information of the App installed on the intelligent terminal device, and the touch interaction information, etc., to assist the user gesture data in jointly confirming the user's interaction intention, thereby improving the recognition accuracy of the user's intention. For example, when the user intends to turn on the air conditioner but there are multiple air conditioner devices at home, the target air conditioner device can be determined and controlled based on the user's voice information "turn on the air conditioner" and the user's location and movement information. Compared with only determining the target device to be controlled based on the voice information, combining the user gesture data and the voice information can reduce the number of interaction rounds with the user and more quickly locate the target device intended to be controlled by the user, improving the efficiency of the intention control process.

[0060] Obtain the user gesture data and historical interaction information of at least one user, and accurately identify the true needs of individual users based on the user gesture data and the historical interaction information of at least one user, effectively avoiding the problem of misidentifying multiple-person instructions. At the same time, by determining the execution priority of the interaction intention of at least one user, it is possible to effectively avoid the intention conflicts among at least one user or between at least one user. Then, based on the execution priority of the interaction intention, the target user is judged, realizing the automatic arbitration of the execution of the interaction intention, optimizing the response logic of the smart home device when facing multiple interaction intentions issued by multiple users. Finally, based on the interaction intention of the target user, the target device is determined and controlled. By optimizing the detection method of the interaction intention and determining the execution priority of the interaction intention, the execution accuracy of the smart home device in responding to multiple interaction intentions is effectively improved. It avoids the risks of lagging execution of important interaction intentions and conflicting interaction intentions, significantly enhancing the device response accuracy and the user's interaction experience.

[0061] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present invention.

[0062] Refer to Figure 2 , which shows the structural block diagram of an embodiment of a home device control device of the present invention, and specifically may include the following modules: The data acquisition module 201 is used to acquire the user gesture data and historical interaction information of at least one user; The execution subject of the method provided in the embodiment of the present invention may be a smart central control device, a smart terminal, a cloud server and other devices; the application scenario may be a single-user or multi-user smart home scenario including at least one user.

[0063] User posture data can be human body physical state information obtained through multimodal sensors. This physical state information can include skeletal key point data, contour data, and motion trajectory data. These data help to know the user's current posture and action, so as to further infer the user's intention based on the posture and action at that time. For example, skeletal key point data can be determined by capturing the three-dimensional coordinates of the user's relevant joints through a 3D depth camera. Motion trajectory data can be captured based on equipment such as millimeter wave radar, such as the direction and speed of movement. And contour data can be further identified based on infrared thermal imagers. Among them, the specific method of obtaining user posture data can refer to the existing technology and will not be repeated here.

[0064] Historical interaction information can refer to the historical interaction records between users and related devices. For these interaction records, they can be classified and managed according to multiple dimensions. For example, they can be classified according to the corresponding sub-scenes in the home scene, such as according to the room function classification. For example, a series of interactive behaviors occurring in the kitchen will be recorded under the data dimension of the kitchen room. The study, living room, living room and other scenes are divided according to the following logic; in addition, they can also be classified according to the user-time dimension. For example, if user A triggers a certain interaction or a certain type of interaction multiple times within a certain time range, it is determined that user A has such a historical interaction record within the time range; for example, user A triggers the operation of turning on the study light multiple times within a time window with 16 o'clock as the center and a span of half an hour before and after, it is considered that user A will have a historical interaction record of turning on the study light around 16 o'clock and record it. The reason for setting it is to reduce the impact of accidental events and save storage resources; this information can be used for intention determination and priority calculation, and the acquisition method can be extracted from the local storage of the edge computing unit, and through the interaction process, the historical interaction information needs to be continuously updated. In addition, historical interaction information can also refer to the user's interaction record for a smart home scene.

[0065] An intention determination module 202, configured to determine the interaction intention of the at least one user according to the user gesture data of the at least one user and the historical interaction information; The user's interaction intention is further determined through similarity calculation, mapping relationship table and other methods. The interaction intention may include the target device to be operated and the corresponding operation method, operation parameters and operation strategy.

[0066] Exemplarily, different posture data can be classified into relevant posture categories according to preset classification criteria based on the differences in the posture data. Each posture category corresponds to a type of interaction intention, and a posture-intention type mapping table can be established according to this corresponding relationship. In this way, after obtaining the user's posture data, the type of intention corresponding to the user's current posture can be roughly determined through the preset posture-intention type mapping table. Determine the interaction intention in the historical interaction record according to the user's scenario or the current time point. For example, according to the user's posture data, it is determined that the user's current behavior belongs to the type of intention for controlling a lighting device, and the current time is 17:30. In the historical interaction record, this user usually turns on the kitchen light within the time window of 18:00. At this time, it is determined that the user's interaction intention is to turn on the kitchen light; A priority determination module 203, configured to determine the execution priority of the interaction intention of the at least one user; In a smart home scenario, simply determining the interaction intention cannot cover all home scenarios. For example, when there is a scenario where the same user has multiple interaction intentions, or when there are multiple users and each user has an interaction intention. Therefore, in addition to determining the interaction intention, it is also necessary to determine the execution priority of the interaction intention.

[0067] A target user determination module 204, configured to determine a target user among the at least one user according to the execution priority of the interaction intention of the at least one user; After obtaining the execution priority of the user's interaction intention, the user with the highest execution priority is taken as the target user, and preparations are made to respond to the interaction intention of this user.

[0068] A device control module 205, configured to determine and control a target device according to the interaction intention of the target user.

[0069] After determining the target user whose intention is to be responded to, by parsing the interaction intention of the target user, the target device to be operated by the interaction intention is determined, and the target device is controlled according to information such as the operation method, operation parameters, and operation strategy in the interaction intention.

[0070] In one embodiment, the data acquisition module includes: A raw data acquisition sub-module, configured to acquire the raw user posture data of the at least one user; A posture completion sub-module, configured to generate completed posture data for each user among the at least one user when it is determined according to the raw user posture data of the user that the posture of the user is blocked; A posture data determination sub-module, configured to use the completed posture data as the user posture data of the user.

[0071] In one embodiment, the user posture data includes user body posture data, and the data acquisition module includes: A user identity determination sub-module, configured to determine the user identity according to the user body posture data of the at least one user; An interaction information acquisition sub-module, configured to acquire the historical interaction information corresponding to the user identity.

[0072] In one embodiment, the historical interaction information includes historical posture data and corresponding historical interaction intentions, and the intention determination module includes: A similarity determination sub-module, configured to determine the similarity between the user posture data of the at least one user and the historical posture data; An interaction intention determination sub-module, configured to use the historical interaction intention corresponding to the historical posture data with the highest similarity as the interaction intention of the at least one user.

[0073] In one embodiment, the priority determination module includes: An intention urgency determination sub-module, configured to determine the intention urgency of the interaction intention of the at least one user; A first priority determination sub-module, configured to determine the intention priority of the interaction intention according to the intention urgency of the interaction intention of the at least one user.

[0074] In one embodiment, the user posture data includes user posture images, and the priority determination module includes: A clarity determination sub-module, configured to acquire the image clarity of the user posture image; A second priority determination sub-module, configured to determine the execution priority of the interaction intention of the at least one user according to the image clarity of the user posture image of the at least one user.

[0075] In one embodiment, the device control module includes: A candidate device determination sub-module, configured to determine at least one candidate device according to the interaction intention of the at least one user; A location data acquisition sub-module, configured to acquire the location data of the target user and the location data of the candidate device; A distance determination sub-module, configured to determine the distance between the at least one candidate device and the target user according to the location data of the target user and the location data of the candidate device; A target device control sub-module, configured to use the candidate device with the shortest distance as the target device, and control the target device according to the interaction intention of the target user.

[0076] Optionally, when a user sets multiple intelligent scenarios, the multiple intelligent scenarios can be parsed into multiple sets of interaction intentions with temporal characteristics or spatial characteristics. At the same time, according to the historical interaction records, the degree of association between the intelligent scenarios is judged. When the degree of association meets the default condition, the interaction intentions of the multiple intelligent scenarios are concatenated according to the temporal characteristics or spatial characteristics. When parsing the user's interaction intention, if the interaction intention is the starting intention of the concatenated multiple interaction intentions, the subsequent intentions will be pre-executed. Exemplarily, the user sets the intelligent scenarios of going home and taking a bath, and multiple historical interaction records all show that after triggering the intelligent scenario of going home, the user will enter the intelligent scenario of taking a bath 20 minutes later. The triggering method of the intelligent scenario of going home is to open the entrance door after 17:00. Then within the time window of 17:00, when the intention of the user to open the entrance door is detected, after responding to this intention, the subsequent intentions will be pre-executed according to the temporal relationship of the interaction intentions.

[0077] In addition, in addition to using the user posture data to determine the user intention, various interaction information such as the user's voice interaction information, intelligent terminal device, interaction information of the App carried by the intelligent terminal device, and touch interaction information can be synchronously received to assist the user posture data to confirm the user interaction intention together, thereby improving the recognition accuracy of the user intention. For example, when the user intends to turn on the air conditioner but there are multiple air conditioner devices at home, the target air conditioner device can be determined and controlled according to the user's voice information "turn on the air conditioner" and the user's location and action information. Compared with only determining the target device controlled by the user intention based on the voice information, combining the user posture data and the voice information can reduce the number of interaction rounds with the user and more quickly locate the target device controlled by the user intention, improving the efficiency of the intention control process.

[0078] Obtain the user posture data and historical interaction information of at least one user, and accurately identify the real needs of individual users according to the user posture data of at least one user and the historical interaction information, effectively avoiding the misrecognition problem of multiple-person instructions. At the same time, by determining the execution priority of the interaction intention of at least one user, the intention conflict between at least one user or between at least one user can be effectively avoided. Then, according to the execution priority of the interaction intention, the target user is judged, realizing the automatic arbitration of the execution of the interaction intention, optimizing the response logic of the smart home device when facing multiple interaction intentions issued by multiple users. Finally, according to the interaction intention of the target user, the target device is determined and controlled. By optimizing the detection method of the interaction intention and determining the execution priority of the interaction intention, the execution accuracy of the smart home device when responding to multiple interaction intentions is effectively improved. Avoiding the risk of lagging execution of important interaction intentions and conflicting interaction intentions significantly improves the device response accuracy and the user's interaction experience.

[0079] For the device embodiments, since they are basically similar to the method embodiments, they are described relatively simply. For the related parts, please refer to the corresponding descriptions in the method embodiments.

[0080] Embodiments of the present invention further provide an electronic device, including: a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, it implements each process of the above-mentioned method embodiment for controlling a home device, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0081] Embodiments of the present invention further provide a computer-readable storage medium with a computer program stored thereon. When the computer program is executed by a processor, it implements each process of the above-mentioned method embodiment for controlling a home device, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0082] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0083] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention can take the form of completely hardware embodiments, completely software embodiments, or embodiments combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0084] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0085] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or the functions specified in a block or multiple blocks.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operational steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or the functions specified in a block or multiple blocks.

[0087] Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0088] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0089] The above has introduced in detail a method, device, equipment and medium for controlling a home device provided by the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, there will be changes in the specific implementation manner and application scope according to the idea of the present invention. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for controlling a home appliance, characterized in that, The method includes: Obtaining user gesture data and historical interaction information of at least one user; Determining the interaction intention of the at least one user according to the user gesture data and the historical interaction information of the at least one user; Determining the execution priority of the interaction intention of the at least one user; Determining a target user among the at least one user according to the execution priority of the interaction intention of the at least one user; Determining and controlling a target device according to the interaction intention of the target user.

2. The method for controlling a home appliance according to claim 1, wherein The obtaining of the user gesture data of at least one user includes: Obtaining the original user gesture data of the at least one user; For each user among the at least one user, when it is determined according to the original user gesture data of the user that the gesture of the user is blocked, generating complementary gesture data; Using the complementary gesture data as the user gesture data of the user.

3. The method for controlling a home appliance according to claim 2, characterized in that, The user gesture data includes user body gesture data. The obtaining of the user gesture data and historical interaction information of at least one user includes: Determining the user identity according to the user body gesture data of the at least one user; Obtaining the historical interaction information corresponding to the user identity.

4. A method for controlling a home appliance according to claim 1, characterized in that, The historical interaction information includes historical gesture data and corresponding historical interaction intentions. The determining of the interaction intention of the at least one user according to the user gesture data and the historical interaction information of the at least one user includes: Determining the similarity between the user gesture data of the at least one user and the historical gesture data; Using the historical interaction intention corresponding to the historical gesture data with the highest similarity as the interaction intention of the at least one user.

5. A method for controlling a home appliance according to claim 1, characterized in that, The determining of the execution priority of the interaction intention of the at least one user includes: Determining the intention urgency of the interaction intention of the at least one user; Determining the intention priority of the interaction intention according to the intention urgency of the interaction intention of the at least one user.

6. A method for controlling a home appliance according to claim 1, wherein, The user gesture data includes user gesture images. The determining of the execution priority of the interaction intention of the at least one user includes: Obtaining the image clarity of the user gesture image; Determining the execution priority of the interaction intention of the at least one user according to the image clarity of the user gesture image of the at least one user.

7. A method for controlling a home device according to claim 1, characterized in that, The determining and controlling of the target device according to the interaction intention of the target user includes: Determining at least one candidate device according to the interaction intention of the at least one user; Obtaining the location data of the target user and the location data of the candidate device; Determining the distance between the at least one candidate device and the target user according to the location data of the target user and the location data of the candidate device; Using the candidate device with the closest distance as the target device and controlling the target device according to the interaction intention of the target user.

8. A home appliance control device, characterized in that, The device includes: A data acquisition module for obtaining user gesture data and historical interaction information of at least one user; An intention determination module for determining the interaction intention of the at least one user according to the user gesture data and the historical interaction information of the at least one user; A priority determination module for determining the execution priority of the interaction intention of the at least one user; A target user determination module for determining a target user among the at least one user according to the execution priority of the interaction intention of the at least one user; A device control module for determining and controlling a target device according to the interaction intention of the target user.

9. An electronic device, characterized in that, Comprising: A processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of a home device control method as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps of a home device control method as described in any one of claims 1-7 are implemented.

Images