A control method, device, storage medium and terminal device for household electrical appliances

Abstract

The present invention discloses a control method, device, storage medium and terminal device for household appliances, including: obtaining depth data to be detected; performing human skeleton key point detection on the depth data to be detected, and obtaining the three-dimensional image coordinate values ​​of the skeleton key points; converting the three-dimensional image coordinate values ​​of the skeleton key points to the world coordinate system, and obtaining the three-dimensional space coordinate values ​​of the skeleton key points; performing directional motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determining the target household appliance according to the directional motion detection results; performing control motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and controlling the target household appliance according to the control motion detection results. The technical solution of the present invention can improve the accuracy of household appliance control by unifying the skeleton key point coordinate data and the environmental data into the same coordinate system, and performing directional motion and control motion detection according to the skeleton key point coordinate data.

Description

Technical Field

[0001] The present invention relates to the field of home appliance control technology, and in particular to a control method, device, computer-readable storage medium and terminal device for home appliance equipment. Background Art

[0002] At present, the action recognition technology based on pictures or videos is relatively mature. According to different sensors, it is mainly divided into action recognition based on RGB pictures / videos and action recognition based on point clouds or depth maps. Action recognition first analyzes the position of the key points of the human skeleton in the image, and then recognizes the action based on the movement trend of the key points.

[0003] The application of motion recognition technology is very extensive. Such applications generally pre-set several motion templates. For example, in skiing motion sensing games, the action is identified by matching the identified key point positions with the set motion templates. If a bent leg is detected, it is considered that skiing is being performed.

[0004] However, this type of application cannot interact more with the environment. The fundamental reason is that it is impossible to unify the skeleton key point data and the environment data into the same coordinate system, and it is impossible to know which interactive devices are in the environment. For example, if there are 3 lights in the room, we can preset gestures to control them. For example, extending one finger means turning on the first light, and so on. However, when there are many home appliances in the room, this method is more troublesome and cannot accurately control each home appliance. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a control method, device, computer-readable storage medium and terminal device for household appliances, which can improve the accuracy of household appliance control by unifying the skeleton key point coordinate data and environmental data into the same coordinate system, and performing pointing and control action detection based on the skeleton key point coordinate data.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a control method for a household appliance, comprising:

[0007] Obtain the depth data to be detected;

[0008] Performing human skeleton key point detection on the depth data to be detected to obtain three-dimensional image coordinate values ​​of the skeleton key points;

[0009] Convert the three-dimensional image coordinate values ​​of the skeleton key points into the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points;

[0010] Performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection result;

[0011] Control-type action detection is performed according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and the target home appliance is controlled according to the control-type action detection results.

[0012] Furthermore, the obtaining of the depth data to be detected specifically includes:

[0013] Acquire the depth data to be detected by collecting through a depth sensor;

[0014] Then, converting the three-dimensional image coordinate values ​​of the skeleton key points into a world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points specifically includes:

[0015] Obtaining the position and posture parameters of the depth sensor in the world coordinate system;

[0016] The three-dimensional image coordinate values ​​of the skeleton key points are converted into coordinates according to the posture parameters of the depth sensor in the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points.

[0017] Furthermore, the obtaining of the posture parameters of the depth sensor in the world coordinate system specifically includes:

[0018] When there is no mobile robot in the field of view of the depth sensor, M background depth images are acquired through the depth sensor, and background modeling is performed according to the M background depth images to obtain a background image; wherein M>0;

[0019] When there is a mobile robot within the field of view of the depth sensor, N depth images corresponding to the mobile robot at N different positions are acquired through the depth sensor; wherein N>1;

[0020] Acquire N mask images according to the background image and the N depth images;

[0021] The coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image are averaged to obtain N corresponding cluster centers; the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N;

[0022] According to the N three-dimensional space coordinate values ​​corresponding to the N different positions of the mobile robot and the N cluster centers, the posture parameters of the depth sensor in the world coordinate system are obtained.

[0023] Further, the acquiring of the pose parameters of the depth sensor in the world coordinate system according to the N three-dimensional space coordinate values ​​corresponding to the N different positions of the mobile robot and the N cluster centers specifically includes:

[0024] According to the formula Solve and obtain the pose parameter H of the depth sensor in the world coordinate system accordingly d ; Among them, P i represents the i-th three-dimensional space coordinate value corresponding to the i-th position of the mobile robot, represents the coordinate value after converting the i-th cluster center into three-dimensional space, K s represents the intrinsic parameter matrix of the depth sensor.

[0025] Further, the performing of directional motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection result, specifically includes:

[0026] Acquire the three-dimensional space coordinate value of the preset starting point key point and the three-dimensional space coordinate value of the preset end point key point according to the three-dimensional space coordinate value of the skeleton key point;

[0027] Obtain the three-dimensional spatial coordinate values ​​and device information of all household appliances in the room;

[0028] According to the three-dimensional space coordinate value of the starting point key point, the three-dimensional space coordinate value of the end point key point and the three-dimensional space coordinate value of all household appliances in the room, respectively obtain the angle between the position of each household appliance and the position of the human body;

[0029] The target home appliance is determined according to the three-dimensional space coordinate value of the home appliance corresponding to the minimum angle and the device information.

[0030] Furthermore, the method obtains the three-dimensional spatial coordinate value and device information of any household appliance in the room through the following steps:

[0031] Capturing a first image at a first position, and acquiring a first device type and a first target area corresponding to a first household electrical appliance in the first image;

[0032] Capturing a second image at a second position, and acquiring a second device type and a second target area corresponding to a second household appliance in the second image;

[0033] When the first device type is the same as the second device type, extracting and matching feature points of the first target area and the second target area to obtain matching feature points;

[0034] Acquire the three-dimensional space coordinate value of the matching feature point according to the matching feature point, the three-dimensional space coordinate value of the first position and the three-dimensional space coordinate value of the second position;

[0035] Querying a preset household appliance information table according to the first device type; wherein the household appliance information table includes a plurality of household appliances and their corresponding device information, and the device information includes at least the device type;

[0036] When there is only one household appliance corresponding to the same device type as the first device type in the household appliance information table, the device information of the first household appliance is determined according to the device information corresponding to the household appliance, and the location information of the first household appliance is determined according to the three-dimensional spatial coordinate value of the matching feature point.

[0037] Further, the controlling action detection is performed according to the three-dimensional space coordinate values ​​of the skeleton key points, and the target home appliance is controlled according to the controlling action detection result, specifically including:

[0038] Matching the three-dimensional spatial coordinate values ​​of the skeleton key points with a preset control action template;

[0039] Determine the control action of the human body according to the successfully matched control action template;

[0040] The target home appliance is controlled according to the determined control action of the human body.

[0041] In order to solve the above technical problems, an embodiment of the present invention further provides a control device for a household appliance, comprising:

[0042] A depth data acquisition module, used to acquire the depth data to be detected;

[0043] A skeleton key point coordinate acquisition module is used to detect the skeleton key points of the human body on the depth data to be detected, and obtain the three-dimensional image coordinate values ​​of the skeleton key points;

[0044] A skeleton key point coordinate conversion module is used to convert the three-dimensional image coordinate value of the skeleton key point into the world coordinate system to obtain the three-dimensional space coordinate value of the skeleton key point;

[0045] A target home appliance determination module, used for performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection results;

[0046] The target home appliance control module is used to perform control action detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and control the target home appliance according to the control action detection results.

[0047] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute any of the above-mentioned methods for controlling a household appliance.

[0048] An embodiment of the present invention further provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any of the above-mentioned methods for controlling a household appliance when executing the computer program.

[0049] Compared with the prior art, the embodiments of the present invention provide a control method, device, computer-readable storage medium and terminal device for household appliances, which first perform human skeleton key point detection on the acquired depth data to be detected, obtain the three-dimensional image coordinate values ​​of the skeleton key points, and convert the three-dimensional image coordinate values ​​of the skeleton key points into the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points; then, perform pointing motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determine the target household appliance according to the pointing motion detection results, thereby performing control motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and control the target household appliance according to the control motion detection results; by unifying the skeleton key point coordinate data and the environmental data into the same coordinate system, and performing pointing motion and control motion detection according to the skeleton key point coordinate data, the accuracy of household appliance control is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a preferred embodiment of a method for controlling household electrical appliances provided by the present invention;

[0051] Figure 2 It is a structural block diagram of a preferred embodiment of a control device for household electrical appliances provided by the present invention;

[0052] Figure 3 It is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this technical field without creative work are within the scope of protection of the present invention.

[0054] The embodiment of the present invention provides a control method for household electrical appliances. Figure 1 FIG. 1 is a flow chart of a preferred embodiment of a method for controlling a household appliance provided by the present invention, wherein the method comprises steps S11 to S15:

[0055] Step S11, obtaining depth data to be detected;

[0056] Step S12, performing human skeleton key point detection on the depth data to be detected to obtain three-dimensional image coordinate values ​​of the skeleton key points;

[0057] Step S13, converting the three-dimensional image coordinate values ​​of the skeleton key points into a world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points;

[0058] Step S14, performing directional motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection result;

[0059] Step S15: performing control action detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and controlling the target home appliance according to the control action detection results.

[0060] Specifically, first, the depth data to be detected corresponding to the human body in the room is obtained, and the key points of the human skeleton are detected on the obtained depth data to be detected, so as to obtain the three-dimensional image coordinate values ​​of the key points of the human skeleton; then, the three-dimensional image coordinate values ​​of the key points of the human skeleton are converted to the world coordinate system, so as to obtain the three-dimensional space coordinate values ​​of the key points of the human skeleton; finally, pointing motion detection is performed according to the three-dimensional space coordinate values ​​of the key points of the human skeleton, so as to determine the target home appliance among all the home appliances in the room according to the results of the pointing motion detection, and control motion detection is performed according to the three-dimensional space coordinate values ​​of the key points of the human skeleton, so as to perform corresponding control on the determined target home appliance according to the results of the control motion detection.

[0061] The embodiment of the present invention divides a complete user action into two types of actions, namely, a pointing action and a control action. The purpose of detecting the pointing action is to determine the direction of the controlled home appliance, so as to determine the controlled target home appliance according to the direction of the controlled home appliance. For example, the direction determined by the two skeletal key points of the user's left elbow and left wrist is used as the pointing action detection result; the purpose of detecting the control action is to determine the control action to be performed on the target home appliance, so as to control the target home appliance accordingly according to the control action. The control action generally includes a special action control gesture. For example, setting a right fist and a "five" gesture to represent the two control actions of opening and closing respectively; a complete user action must include both a pointing action and a control action to ensure accurate control of the home appliance; wherein, the pointing action and the control action can be issued successively or simultaneously. For example, the user can issue a pointing action first and then a control action, or can issue a control action first and then a pointing action, or can issue a pointing action and a control action simultaneously.

[0062] It should be noted that when detecting human skeleton key points of the depth data to be detected, if the three-dimensional image coordinate values ​​of the skeleton key points corresponding to multiple human bodies are detected, the three-dimensional image coordinate values ​​of the skeleton key points corresponding to the multiple human bodies can be first converted to a unified world coordinate system, and the three-dimensional space coordinate values ​​of the skeleton key points corresponding to the multiple human bodies can be obtained accordingly. Then, based on the three-dimensional space coordinate values ​​of the skeleton key points corresponding to the multiple human bodies, it is determined whether the multiple human bodies overlap in the world coordinate system. If they overlap, it means that the multiple human bodies are essentially the same human body, and the three-dimensional space coordinate value of the skeleton key point corresponding to any one of the multiple human bodies is selected for subsequent processing; if they do not overlap, it means that the multiple human bodies are not the same human body, and each user represented by the multiple human bodies and the priority corresponding to each user can be identified through user identification, and subsequent processing is performed according to the three-dimensional space coordinate value of the skeleton key point corresponding to each user and the priority corresponding to each user; wherein, the priority can be set by the user, and if the user does not set it, a default setting is given at will.

[0063] It can be understood that user identification can be based on various image recognition methods provided by the prior art, such as face recognition, or the user's height, arm length and other information can be determined based on the information of skeletal key points, and the user can be identified based on the height, arm length and other information pre-entered by the user.

[0064] In addition, the key points of the human skeleton may include the head, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, chest, pelvis, left hip, right hip, left knee, left ankle, right knee and right ankle of the human body.

[0065] A control method for household appliances provided by an embodiment of the present invention converts the three-dimensional image coordinate values ​​of key points of the human skeleton into a world coordinate system to unify the coordinate data of the key points of the human skeleton and the environmental data into the same coordinate system, and performs pointing action and control action detection based on the coordinate data of the key points of the human skeleton in the world coordinate system, so as to perform corresponding control on the determined target household appliances based on the pointing action detection results and the control action detection results, thereby improving the accuracy of controlling the household appliances.

[0066] In another preferred embodiment, the obtaining of the depth data to be detected specifically includes:

[0067] Acquire the depth data to be detected by collecting through a depth sensor;

[0068] Then, converting the three-dimensional image coordinate values ​​of the skeleton key points into a world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points specifically includes:

[0069] Obtaining the position and posture parameters of the depth sensor in the world coordinate system;

[0070] The three-dimensional image coordinate values ​​of the skeleton key points are converted into coordinates according to the posture parameters of the depth sensor in the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points.

[0071] Specifically, in combination with the above embodiments, when obtaining the depth data to be detected corresponding to the human body indoors, the depth data to be detected corresponding to the human body can be acquired through the depth sensor. Correspondingly, when the three-dimensional image coordinate values ​​of the key points of the human skeleton are converted into world coordinates, since the depth data is obtained based on the depth sensor, it is necessary to use the posture parameters of the depth sensor in the world coordinate system to perform coordinate conversion, that is, first obtain the posture parameters of the depth sensor in the world coordinate system, and the posture parameters are the spatial transformation matrix H={R|t} corresponding to the depth sensor, and then convert the three-dimensional image coordinate values ​​of the key points of the human skeleton into three-dimensional spatial coordinate values ​​in the world coordinate system according to the spatial transformation matrix H={R|t} of the depth sensor.

[0072] For example, if the three-dimensional image coordinate value of a certain bone key point is detected to be P, the three-dimensional image coordinate value P of the bone key point can be converted from the coordinate system corresponding to the depth sensor to the three-dimensional space coordinate value P' in the world coordinate system through the formula P'=H*P.

[0073] It should be noted that in order to obtain the depth information of the key points of the human skeleton, the methods provided by the prior art generally include the following: (1) using a depth sensor to collect a depth image, and using the depth image to extract the depth data of the key points of the skeleton, and then performing the corresponding 3D skeleton key point detection, but this method has low accuracy; (2) using a depth camera, using the RGB image of a camera to identify the key points of the skeleton, and then using the binocular vision method to restore the depth through the pictures of two perspectives, and then converting the key points on the RGB image into three-dimensional coordinates through the depth; (3) using an RGB sensor (i.e. obtaining image data) to perform 2D skeleton key point detection, although the accuracy is high, it is impossible to obtain 3D key points; (4) using an RGBD sensor (i.e. obtaining RGBD data), first using RGB data for 2D detection, and then using D data (depth data) to convert 2D coordinates into 3D coordinates; the embodiment of the present invention is to obtain the 3D coordinates of the skeleton key points, therefore, in addition to the above method (3), any one of the above methods (1), (2) and (4) can be used, the difference is only the problem of accuracy.

[0074] As an improvement of the above solution, the step of obtaining the posture parameters of the depth sensor in the world coordinate system specifically includes:

[0075] When there is no mobile robot in the field of view of the depth sensor, M background depth images are acquired through the depth sensor, and background modeling is performed according to the M background depth images to obtain a background image; wherein M>0;

[0076] When there is a mobile robot within the field of view of the depth sensor, N depth images corresponding to the mobile robot at N different positions are acquired through the depth sensor; wherein N>1;

[0077] Acquire N mask images according to the background image and the N depth images;

[0078] The coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image are averaged to obtain N corresponding cluster centers; the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N;

[0079] According to the N three-dimensional space coordinate values ​​corresponding to the N different positions of the mobile robot and the N cluster centers, the posture parameters of the depth sensor in the world coordinate system are obtained.

[0080] Specifically, in combination with the above embodiment, the scheme for obtaining the posture parameters of the depth sensor in the world coordinate system is as follows:

[0081] First, when there is no mobile robot in the field of view of the depth sensor (i.e., there is no movement), M (M>0) background depth images are acquired by the depth sensor, and background modeling is performed based on the acquired M background depth images, and the background image in the field of view of the depth sensor is obtained accordingly. When there is a mobile robot in the field of view of the depth sensor (i.e., there is movement), N depth images corresponding to N (N>1) different positions of the mobile robot are acquired by the depth sensor, and the three-dimensional spatial coordinate values ​​of the mobile robot in the world coordinate system corresponding to the N different positions are respectively obtained, and the N three-dimensional spatial coordinate values ​​corresponding to the mobile robot are obtained accordingly; wherein, the monitoring area environment of the depth sensor is mapped by using the mobile robot and the SLAM algorithm (Simultaneous Localization and Mapping, synchronous positioning and mapping algorithm), and the environmental grid map corresponding to the monitoring area is obtained accordingly to determine the world coordinate system. The mobile robot can determine its own three-dimensional spatial coordinate value in the world coordinate system according to the constructed environmental grid map and the SLAM positioning function, so that the three-dimensional spatial coordinate values ​​of the mobile robot in the world coordinate system corresponding to the N different positions can be obtained accordingly.

[0082] Next, the obtained background image is respectively subjected to frame difference calculation with each depth image to obtain N frame difference images, and then each frame difference image is processed accordingly according to a preset depth threshold to obtain N mask images; wherein, the processing method of each frame difference image is the same, and here the processing method of the i-th frame difference image is taken as an example, and the depth value of each pixel on the i-th frame difference image is respectively compared with the preset depth threshold, and when it is determined that the depth value of any pixel is greater than the preset depth threshold, the pixel is marked as 1, and when it is determined that the depth value of any pixel is not greater than the preset depth threshold, the pixel is marked as 0, and accordingly, after each pixel on the i-th frame difference image is marked as 1 or 0, the i-th mask image is obtained according to the i-th frame difference image after marking.

[0083] Then, the coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image are averaged, and N cluster centers are obtained accordingly; among them, taking the processing method of the i-th mask image as an example, the uv coordinate values ​​and depth values ​​d of all pixels marked as 1 on the i-th mask image are averaged, and the cluster center corresponding to the i-th mask image is expressed as p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i Represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N.

[0084] Finally, based on the N three-dimensional spatial coordinate values ​​corresponding to the mobile robot, the N cluster centers and the intrinsic parameter matrix of the depth sensor, the posture parameters of the depth sensor can be calculated; among them, the intrinsic parameter matrix of the depth sensor is generally given by the manufacturer or calibrated in advance and is a known parameter.

[0085] It should be noted that the same depth sensor generally does not rotate during the calibration process, that is, the posture remains unchanged. In this case, at least one background depth image collected is similar. Possibly due to the presence of noise, there will be relatively small differences in each background depth image. Therefore, when obtaining the background image within the field of view of the depth sensor based on the M background depth images collected, the background image within the field of view of the depth sensor can be obtained using only one background depth image, or multiple background depth images can be used to obtain the background image within the field of view of the depth sensor. If multiple background depth images are used, the depth values ​​of each pixel in the multiple background depth images can be averaged to obtain the background image within the field of view of the depth sensor.

[0086] As an improvement of the above solution, the method of obtaining the pose parameters of the depth sensor in the world coordinate system according to the N three-dimensional space coordinate values ​​corresponding to the N different positions of the mobile robot and the N cluster centers specifically includes:

[0087] According to the formula Solve and obtain the pose parameter H of the depth sensor in the world coordinate system accordingly d ; Among them, P i represents the i-th three-dimensional space coordinate value corresponding to the i-th position of the mobile robot, represents the coordinate value after converting the i-th cluster center into three-dimensional space, K srepresents the intrinsic parameter matrix of the depth sensor.

[0088] Specifically, in combination with the above embodiment, when calculating the posture parameters of the depth sensor according to the N three-dimensional space coordinate values ​​and N cluster centers corresponding to the obtained mobile robot, the formula Solve and obtain the corresponding posture parameter H of the depth sensor d , P i represents the i-th three-dimensional space coordinate value corresponding to the i-th position of the mobile robot, Indicates that the i-th cluster center p i =(u i , v i , d i ) is converted to the position information (three-dimensional space coordinate value) in the world coordinate system, K s represents the intrinsic parameter matrix of the depth sensor, Represents the Euclidean distance between two 3D vectors.

[0089] It should be noted that the posture parameter H of the depth sensor d That is, the spatial transformation matrix of the depth sensor or the posture of the depth sensor relative to the world coordinate system, through the posture parameter H d A point (u, v) on the depth image collected by the depth sensor can be converted into a three-dimensional coordinate value (x, y, z) in the world coordinate system.

[0090] In another preferred embodiment, the step of performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points and determining the target home appliance according to the directional motion detection result specifically includes:

[0091] Acquire the three-dimensional space coordinate value of the preset starting point key point and the three-dimensional space coordinate value of the preset end point key point according to the three-dimensional space coordinate value of the skeleton key point;

[0092] Obtain the three-dimensional spatial coordinate values ​​and device information of all household appliances in the room;

[0093] According to the three-dimensional space coordinate value of the starting point key point, the three-dimensional space coordinate value of the end point key point and the three-dimensional space coordinate value of all household appliances in the room, respectively obtain the angle between the position of each household appliance and the position of the human body;

[0094] The target home appliance is determined according to the three-dimensional space coordinate value of the home appliance corresponding to the minimum angle and the device information.

[0095] Specifically, in combination with the above embodiments, when performing directional motion detection, any two skeletal key points among the human body's skeletal key points can be pre-selected as the starting point key point and the ending point key point respectively. Then, the three-dimensional spatial coordinate values ​​of the starting point key point and the three-dimensional spatial coordinate values ​​of the ending point key point can be correspondingly obtained based on the obtained three-dimensional spatial coordinate values ​​of all the human body's skeletal key points. Next, based on the pre-obtained three-dimensional spatial coordinate values ​​of all indoor home appliances in the world coordinate system, as well as the three-dimensional spatial coordinate values ​​of the starting point key point and the three-dimensional spatial coordinate values ​​of the ending point key point, the angle between the position of each home appliance and the position of the human body is calculated respectively, and the minimum angle is selected. The home appliance corresponding to the minimum angle is used as the target home appliance, and the three-dimensional spatial coordinate value and device information of the home appliance corresponding to the minimum angle are the three-dimensional spatial coordinate value and device information of the target home appliance.

[0096] For example, let Pstart and Pend represent the starting key point and the ending key point respectively. Assume that there are X home appliances in the room, and the position of each home appliance in the world coordinate system corresponds to P1, P2, ..., PX. For the i-th (i=1, 2, ..., X) home appliance, calculate the angle between the vectors (Pi-Pstart) and (Pend-Pstart) (or the angle between other two vectors can also be calculated). Then the angle between these two three-dimensional vectors can be expressed as the angle between the position of the i-th home appliance and the position of the human body. Similarly, the X corresponding angles between the positions of the X home appliances and the positions of the human body can be obtained, and find the home appliance with the smallest angle as the target home appliance.

[0097] It should be noted that in addition to taking the home appliance corresponding to the minimum angle as the target home appliance, an angle threshold can also be set in advance to determine at least one home appliance whose angle is smaller than the angle threshold, and then select the home appliance closest to the human body from the at least one home appliance as the target home appliance.

[0098] As an improvement of the above solution, the method obtains the three-dimensional space coordinate value and device information of any household appliance in the room through the following steps:

[0099] Capturing a first image at a first position, and acquiring a first device type and a first target area corresponding to a first household electrical appliance in the first image;

[0100] Capturing a second image at a second position, and acquiring a second device type and a second target area corresponding to a second household appliance in the second image;

[0101] When the first device type is the same as the second device type, extracting and matching feature points of the first target area and the second target area to obtain matching feature points;

[0102] Acquire the three-dimensional space coordinate value of the matching feature point according to the matching feature point, the three-dimensional space coordinate value of the first position and the three-dimensional space coordinate value of the second position;

[0103] Querying a preset household appliance information table according to the first device type; wherein the household appliance information table includes a plurality of household appliances and their corresponding device information, and the device information includes at least the device type;

[0104] When there is only one household appliance corresponding to the same device type as the first device type in the household appliance information table, the device information of the first household appliance is determined according to the device information corresponding to the household appliance, and the location information of the first household appliance is determined according to the three-dimensional spatial coordinate value of the matching feature point.

[0105] Specifically, in combination with the above embodiment, before controlling the home appliance, the embodiment of the present invention pre-acquires the three-dimensional space coordinate values ​​and device information of all the controlled home appliances in the room. The acquisition scheme of the three-dimensional space coordinate values ​​and device information of any home appliance is as follows:

[0106] First, a first image including a household appliance is captured at a first position indoors, and a first device type corresponding to the first household appliance in the first image and a first target area corresponding to the first household appliance in the first image are obtained. Similarly, a second image including a household appliance is captured at a second position indoors, and a second device type corresponding to the second household appliance in the second image and a second target area corresponding to the second household appliance in the second image are obtained. The first image can be acquired by a camera on a mobile robot. When the mobile robot moves to the first position indoors, the first image including the first household appliance is captured, and an image recognition method provided by the prior art is used to perform image recognition on the first image, and the first device type corresponding to the first household appliance in the first image is obtained accordingly. Furthermore, a moving target detection method provided by the prior art is used to perform moving target detection on the first image, and the first target area corresponding to the first household appliance in the first image is obtained accordingly. The same is true for the second image.

[0107] Next, determine whether the first device type and the second device type are the same. When it is determined that the first device type is the same as the second device type (that is, the first home appliance and the second home appliance are the same home appliance), perform feature point extraction and feature point matching processing on the first target area and the second target area, and obtain corresponding matching feature points; wherein, a SIFT algorithm (Scale Invariant Feature Transform) or a SURF algorithm (Speeded Up Robust Features) can be used to extract feature points from a local image in the first target area and a local image in the second target area, and perform feature point matching based on the extracted feature points corresponding to the first target area and the feature points corresponding to the second target area, and obtain corresponding matching feature points of the first target area and the second target area.

[0108] Then, according to the three-dimensional space coordinate value corresponding to the first position and the three-dimensional space coordinate value corresponding to the second position, the three-dimensional space coordinate value corresponding to the matching feature point is calculated; wherein, the SLAM algorithm (Simultaneous Localization and Mapping) can be used to obtain the first pose parameter S1={R1|t1} and its corresponding three-dimensional space coordinate value (that is, the three-dimensional space coordinate value corresponding to the first position) of the mobile robot at the first position, and obtain the second pose parameter S2={R2|t2} and its corresponding three-dimensional space coordinate value (that is, the three-dimensional space coordinate value corresponding to the second position) of the mobile robot at the second position, and because the matching feature points are based on the feature points in the image coordinate system, it is necessary to first convert the matching feature points into the three-dimensional camera coordinate system through the camera imaging model corresponding to the camera used when taking the image. The first matching feature point under the image is obtained, and then the first pose parameter S1 (i.e., the first spatial transformation matrix) is used to transform the first matching feature point to the world coordinate system, and a first ray with the first position as the starting point is formed accordingly. Similarly, the first matching feature point is transformed to the world coordinate system through the second pose parameter S2 obtained, and a second ray with the second position as the starting point is formed accordingly. The triangular intersection method is used to obtain the three-dimensional space coordinate value corresponding to the first position and the three-dimensional space coordinate value corresponding to the second position, and the three-dimensional space coordinate value corresponding to the intersection of the first ray and the second ray is obtained, that is, the three-dimensional space coordinate value corresponding to the matching feature point is obtained.

[0109] Finally, a pre-set household appliance information table is queried according to the first device type. The household appliance information table records several household appliances in the room and the device information corresponding to each household appliance (for example, device type, device function and other related device information), so as to find out the household appliance in the household appliance information table whose device type is the same as the first device type. When there is only one household appliance corresponding to the same device type as the first device type found in the household appliance information table, it can be understood that the household appliance found is the first household appliance. The device information corresponding to the found household appliance is used as the device information of the first household appliance, and the three-dimensional spatial coordinate value of the obtained matching feature point is used as the three-dimensional spatial coordinate value of the first household appliance, so that the device information of the household appliance and the three-dimensional spatial coordinate value are associated one-to-one, and the positioning of the household appliance and the identification of the household appliance are realized at the same time.

[0110] In addition, when querying the pre-set household appliance information table according to the first device type, if there are Y household appliances corresponding to the same device type as the first device type found in the household appliance information table (Y>1), that is, there are multiple household appliances of the same device type, then the device information and three-dimensional space coordinate values ​​of the first household appliance can be obtained accordingly in combination with the user's actual indoor location and room information.

[0111] For example, it is possible to communicate with indoor household appliances according to a preset communication protocol, first obtain the current working status of the first household appliance to determine whether the first household appliance is currently running, and when it is determined that the first household appliance is currently running, obtain the three-dimensional spatial coordinate value corresponding to the user, and then based on the three-dimensional spatial coordinate values ​​corresponding to Y household appliances (the three-dimensional spatial coordinate values ​​corresponding to the Y household appliances can also be calculated and obtained respectively using the triangulation intersection method in the above embodiment) and the obtained three-dimensional spatial coordinate value of the user, combined with the room information in the room, determine the household appliance in the same room as the user among the Y household appliances and use it as the first target household appliance. The first target household appliance is the first household appliance, and the device information corresponding to the first target household appliance is used as the device information of the first household appliance, and the three-dimensional spatial coordinate value of the obtained matching feature point is used as the three-dimensional spatial coordinate value of the first household appliance.

[0112] Alternatively, the distance between each of the Y home appliances and the user may be calculated based on the three-dimensional spatial coordinate values ​​corresponding to the Y home appliances and the obtained three-dimensional spatial coordinate values ​​of the user, and the home appliance that is closest to the user among the Y home appliances may be found and used as the second target home appliance. The second target home appliance is the first home appliance, and the device information corresponding to the second target home appliance is used as the device information of the first home appliance, and the three-dimensional spatial coordinate values ​​of the obtained matching feature points are used as the three-dimensional spatial coordinate values ​​of the first home appliance.

[0113] In another preferred embodiment, the control action detection is performed according to the three-dimensional space coordinate values ​​of the skeleton key points, and the target home appliance is controlled according to the control action detection result, specifically including:

[0114] Matching the three-dimensional spatial coordinate values ​​of the skeleton key points with a preset control action template;

[0115] Determine the control action of the human body according to the successfully matched control action template;

[0116] The target home appliance is controlled according to the determined control action of the human body.

[0117] Specifically, in combination with the above embodiments, when performing control-type action detection, several control action templates can be pre-set, and different control action templates respectively represent different control commands issued to household appliances. The obtained three-dimensional spatial coordinate values ​​of the key points of the human skeleton are matched with the pre-set control action templates. The corresponding human control action can be determined based on the successfully matched control action template, that is, the corresponding control command can be determined accordingly, and the target household appliance determined by the directional action detection result and the human control action determined by the control-type action detection result can be combined, that is, the target household appliance is controlled accordingly according to the determined human control action.

[0118] The present invention also provides a control device for household electrical appliances. Figure 2 FIG. 1 is a structural block diagram of a preferred embodiment of a control device for household electrical appliances provided by the present invention, wherein the device comprises:

[0119] A depth data acquisition module 11 is used to acquire the depth data to be detected;

[0120] A skeleton key point coordinate acquisition module 12 is used to detect the skeleton key points of the human body on the depth data to be detected, and obtain the three-dimensional image coordinate values ​​of the skeleton key points;

[0121] A skeleton key point coordinate conversion module 13 is used to convert the three-dimensional image coordinate value of the skeleton key point into a world coordinate system to obtain the three-dimensional space coordinate value of the skeleton key point;

[0122] A target home appliance determination module 14 is used to perform directional motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determine the target home appliance according to the directional motion detection result;

[0123] The target home appliance control module 15 is used to perform control action detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and control the target home appliance according to the control action detection results.

[0124] Preferably, the depth data acquisition module 11 specifically includes:

[0125] A depth data acquisition unit, used for acquiring the depth data to be detected through a depth sensor;

[0126] Then, the skeleton key point coordinate conversion module 13 specifically includes:

[0127] A sensor posture acquisition unit, used to acquire the posture parameters of the depth sensor in the world coordinate system;

[0128] The skeleton key point coordinate conversion unit is used to perform coordinate conversion on the three-dimensional image coordinate value of the skeleton key point according to the posture parameters of the depth sensor in the world coordinate system to obtain the three-dimensional space coordinate value of the skeleton key point.

[0129] Preferably, the sensor posture acquisition unit specifically includes:

[0130] A background image acquisition subunit is used to acquire M background depth images through the depth sensor when there is no mobile robot in the field of view of the depth sensor, and to perform background modeling according to the M background depth images to obtain a background image; wherein M>0;

[0131] A depth image acquisition subunit, used for acquiring N depth images corresponding to N different positions of the mobile robot through the depth sensor when there is a mobile robot in the field of view of the depth sensor; wherein N>1;

[0132] A mask image acquisition subunit, configured to acquire N mask images according to the background image and the N depth images;

[0133] The mask image processing subunit is used to calculate the average coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image, and obtain N corresponding cluster centers; among which, the cluster center corresponding to the i-th mask image is p i=(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N;

[0134] The sensor posture acquisition subunit is used to obtain the posture parameters of the depth sensor in the world coordinate system according to the N three-dimensional space coordinate values ​​corresponding to the N different positions of the mobile robot and the N cluster centers.

[0135] Preferably, the sensor posture acquisition subunit is specifically used for:

[0136] According to the formula Solve and obtain the pose parameter H of the depth sensor in the world coordinate system accordingly d ; Among them, P i represents the i-th three-dimensional space coordinate value corresponding to the i-th position of the mobile robot, represents the coordinate value after converting the i-th cluster center into three-dimensional space, K s represents the intrinsic parameter matrix of the depth sensor.

[0137] Preferably, the target household appliance determination module 14 specifically includes:

[0138] A starting point and end point coordinate acquisition unit, used for acquiring a three-dimensional space coordinate value of a preset starting point key point and a three-dimensional space coordinate value of a preset end point key point according to the three-dimensional space coordinate value of the skeleton key point;

[0139] Home appliance positioning and identification unit, used to obtain the three-dimensional spatial coordinate values ​​and device information of all home appliances in the room;

[0140] An angle calculation unit, used to obtain the angle between the position of each household appliance and the position of the human body according to the three-dimensional space coordinate value of the starting point key point, the three-dimensional space coordinate value of the end point key point and the three-dimensional space coordinate values ​​of all household appliances in the room;

[0141] The target home appliance determining unit is used to determine the target home appliance according to the three-dimensional space coordinate value of the home appliance corresponding to the minimum angle and the device information.

[0142] Preferably, the household appliance positioning and identification unit is specifically used for:

[0143] Capturing a first image at a first position, and acquiring a first device type and a first target area corresponding to a first household electrical appliance in the first image;

[0144] Capturing a second image at a second position, and acquiring a second device type and a second target area corresponding to a second household appliance in the second image;

[0145] When the first device type is the same as the second device type, extracting and matching feature points of the first target area and the second target area to obtain matching feature points;

[0146] Acquire the three-dimensional space coordinate value of the matching feature point according to the matching feature point, the three-dimensional space coordinate value of the first position and the three-dimensional space coordinate value of the second position;

[0147] Querying a preset household appliance information table according to the first device type; wherein the household appliance information table includes a plurality of household appliances and their corresponding device information, and the device information includes at least the device type;

[0148] When there is only one household appliance corresponding to the same device type as the first device type in the household appliance information table, the device information of the first household appliance is determined according to the device information corresponding to the household appliance, and the location information of the first household appliance is determined according to the three-dimensional spatial coordinate value of the matching feature point.

[0149] Preferably, the target household appliance control module 15 specifically includes:

[0150] A control action matching unit, used for matching the three-dimensional space coordinate value of the skeleton key point with a preset control action template;

[0151] A control action determination unit, used to determine the control action of the human body according to the successfully matched control action template;

[0152] The target home appliance control unit is used to control the target home appliance according to the determined control action of the human body.

[0153] It should be noted that a control device for household appliances provided in an embodiment of the present invention can implement all processes of the control method for household appliances described in any of the above embodiments, and the functions of each module, unit and sub-unit in the device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the control method for household appliances described in the above embodiments, and will not be repeated here.

[0154] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the control method of the household appliance described in any of the above embodiments.

[0155] The embodiment of the present invention also provides a terminal device, see Figure 3 As shown, it is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention, wherein the terminal device includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10, and the processor 10 implements the control method of the household appliance described in any of the above embodiments when executing the computer program.

[0156] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0157] The processor 10 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 10 may be any conventional processor. The processor 10 is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0158] The memory 20 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 20 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 20 can also be other volatile solid-state storage devices.

[0159] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 3 The structural block diagram is merely an example of the above-mentioned terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or less components than shown in the figure, or a combination of certain components, or different components.

[0160] In summary, the control method, device, computer-readable storage medium and terminal device of a household appliance provided by the embodiments of the present invention first perform human skeleton key point detection on the acquired depth data to be detected, obtain the three-dimensional image coordinate values ​​of the skeleton key points, and convert the three-dimensional image coordinate values ​​of the skeleton key points into the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points; then perform pointing motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determine the target household appliance according to the pointing motion detection results, thereby performing control motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and controlling the target household appliance according to the control motion detection results; by converting the three-dimensional image coordinate values ​​of the human skeleton key points into the world coordinate system, so as to unify the coordinate data of the human skeleton key points and the environmental data into the same coordinate system, and perform pointing motion and control motion detection according to the coordinate data of the human skeleton key points in the world coordinate system, so as to control the determined target household appliance accordingly according to the pointing motion detection results and the control motion detection results, thereby improving the accuracy of household appliance control.

[0161] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for controlling a household appliance, characterized in that: include: Obtain the depth data to be detected; Performing human skeleton key point detection on the depth data to be detected to obtain three-dimensional image coordinate values ​​of the skeleton key points; Convert the three-dimensional image coordinate values ​​of the skeleton key points into the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points; Performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection result; Performing control action detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and controlling the target home appliance according to the control action detection result; The obtaining of the depth data to be detected specifically includes: Acquire the depth data to be detected by collecting through a depth sensor; Then, converting the three-dimensional image coordinate values ​​of the skeleton key points into a world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points specifically includes: Obtaining the position and posture parameters of the depth sensor in the world coordinate system; Performing coordinate transformation on the three-dimensional image coordinate values ​​of the skeleton key points according to the posture parameters of the depth sensor in the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points; The obtaining of the posture parameters of the depth sensor in the world coordinate system specifically includes: Obtain N mask images according to the background image and N depth images; where N>1; The coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image are averaged to obtain N corresponding cluster centers; the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N; According to the N three-dimensional space coordinate values ​​corresponding to the N different positions of the mobile robot and the N cluster centers, the posture parameters of the depth sensor in the world coordinate system are obtained.

2. The control method of household electrical appliances according to claim 1, characterized in that: The background image and the N depth images are obtained in the following manner: When there is no mobile robot in the field of view of the depth sensor, M background depth images are acquired through the depth sensor, and background modeling is performed according to the M background depth images to obtain a background image; wherein M>0; When there is a mobile robot within the field of view of the depth sensor, N depth images corresponding to the mobile robot at N different positions are acquired through the depth sensor.

3. The control method of household electrical appliances according to claim 1, characterized in that: The step of obtaining the position parameters of the depth sensor in the world coordinate system according to the N three-dimensional space coordinate values ​​corresponding to the N different positions of the mobile robot and the N cluster centers specifically includes: According to the formula Solve and obtain the pose parameter H of the depth sensor in the world coordinate system accordingly d ; Among them, P i represents the i-th three-dimensional space coordinate value corresponding to the i-th position of the mobile robot, represents the coordinate value after converting the i-th cluster center into three-dimensional space, K s represents the intrinsic parameter matrix of the depth sensor.

4. The method for controlling a household appliance according to claim 1, wherein: The step of performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection result, specifically includes: Acquire the three-dimensional space coordinate value of the preset starting point key point and the three-dimensional space coordinate value of the preset end point key point according to the three-dimensional space coordinate value of the skeleton key point; Obtain the three-dimensional spatial coordinate values ​​and device information of all household appliances in the room; According to the three-dimensional space coordinate value of the starting point key point, the three-dimensional space coordinate value of the end point key point and the three-dimensional space coordinate value of all household appliances in the room, respectively obtain the angle between the position of each household appliance and the position of the human body; The target home appliance is determined according to the three-dimensional space coordinate value of the home appliance corresponding to the minimum angle and the device information.

5. The method for controlling a household appliance according to claim 4, characterized in that: The method obtains the three-dimensional space coordinate value and device information of any household appliance in the room through the following steps: Capturing a first image at a first position, and acquiring a first device type and a first target area corresponding to a first household electrical appliance in the first image; Capturing a second image at a second position, and acquiring a second device type and a second target area corresponding to a second household appliance in the second image; When the first device type is the same as the second device type, extracting and matching feature points of the first target area and the second target area to obtain matching feature points; Acquire the three-dimensional space coordinate value of the matching feature point according to the matching feature point, the three-dimensional space coordinate value of the first position and the three-dimensional space coordinate value of the second position; Querying a preset household appliance information table according to the first device type; wherein the household appliance information table includes a plurality of household appliances and their corresponding device information, and the device information includes at least the device type; When there is only one household appliance corresponding to the same device type as the first device type in the household appliance information table, the device information of the first household appliance is determined according to the device information corresponding to the household appliance, and the location information of the first household appliance is determined according to the three-dimensional spatial coordinate value of the matching feature point.

6. The method for controlling a household appliance according to any one of claims 1 to 5, characterized in that: The controlling action detection is performed according to the three-dimensional space coordinate values ​​of the skeleton key points, and the target household appliance is controlled according to the controlling action detection result, specifically including: Matching the three-dimensional spatial coordinate values ​​of the skeleton key points with a preset control action template; Determine the control action of the human body according to the successfully matched control action template; The target home appliance is controlled according to the determined control action of the human body.

7. A control device for household electrical appliances, characterized in that: include: A depth data acquisition module, used to acquire the depth data to be detected; A skeleton key point coordinate acquisition module is used to detect the skeleton key points of the human body on the depth data to be detected, and obtain the three-dimensional image coordinate values ​​of the skeleton key points; A skeleton key point coordinate conversion module is used to convert the three-dimensional image coordinate value of the skeleton key point into the world coordinate system to obtain the three-dimensional space coordinate value of the skeleton key point; A target home appliance determination module, used for performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection results; A target home appliance control module, used to perform control action detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and control the target home appliance according to the control action detection results; The depth data acquisition module specifically includes: A depth data acquisition unit, used for acquiring the depth data to be detected through a depth sensor; Then, the skeleton key point coordinate conversion module specifically includes: A sensor posture acquisition unit, used to acquire the posture parameters of the depth sensor in the world coordinate system; A skeleton key point coordinate conversion unit, used for performing coordinate conversion on the three-dimensional image coordinate value of the skeleton key point according to the posture parameters of the depth sensor in the world coordinate system to obtain the three-dimensional space coordinate value of the skeleton key point; The sensor posture acquisition unit specifically includes: The mask image acquisition subunit is used to acquire N mask images according to the background image and N depth images; wherein N>1; The mask image processing subunit is used to calculate the average coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image, and obtain N corresponding cluster centers; among which, the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N; The sensor posture acquisition subunit is used to obtain the posture parameters of the depth sensor in the world coordinate system according to the N three-dimensional space coordinate values ​​corresponding to the mobile robot at N different positions and the N cluster centers.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the control method of the household appliance according to any one of claims 1 to 6.

9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the control method of the household appliance according to any one of claims 1 to 6 when executing the computer program.

Images