Control method, system and storage medium of household electrical appliances

Abstract

The present application discloses a control method, system and storage medium for household appliances, and relates to the field of smart home / intelligent home technology. The control method for the household appliances includes: obtaining multiple gesture images of a user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple household appliances with image acquisition functions; extracting features from each gesture image to obtain display information features and action features; prioritizing the multiple gesture images according to the display information features, and determining the household appliance corresponding to the gesture image with the highest priority as the target household appliance; determining the operation instruction corresponding to the gesture image with the highest priority according to the action features of the gesture image with the highest priority, and sending the operation instruction to the target household appliance to control the target household appliance. The present application improves the recognition accuracy of gesture-operated household appliances, and improves the user experience.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a control method, system and storage medium for household appliances. Background Art

[0002] In the traditional home appliance control process, the remote control is used to control the home appliance, but the remote control has few control functions and is easy to lose, which is inconvenient for users. The control operation of smart home appliances through mobile phone programs is more complicated, especially for the elderly who cannot use smart phones.

[0003] As the technology of smart home appliances continues to advance, voice operation and gesture operation have emerged to achieve touchless operation of home appliances. In particular, the gesture operation function, which uses a camera to achieve touchless gesture recognition, is not only simple to operate, but also can be used by some special users. For example, deaf-mute users cannot operate the air conditioner by voice, and gesture operation can realize the intelligent operation of the air conditioner for these special users.

[0004] In the current smart home environment, more and more home appliances have gesture operation functions. When controlling home appliances through gestures, there is often a problem of gesture operation recognition error. For example, the gesture operation of controlling a smart TV is recognized as the gesture operation of a smart air conditioner, and the gesture operation belonging to the smart air conditioner cannot be more accurately recognized, resulting in a poor user experience. Therefore, there is an urgent need for a control method, system and storage medium for home appliances to solve the above problems. Summary of the invention

[0005] The present application provides a control method, system and storage medium for household appliances, which are used to solve the defect of low recognition rate of gesture operation of household appliances in a smart home environment in the prior art, and to accurately judge whether the gesture action is used to control the corresponding household appliances from the gesture images collected simultaneously from multiple household appliances.

[0006] The present application provides a control method for a household appliance, comprising:

[0007] Acquire multiple gesture images of the user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple home appliances with image acquisition functions;

[0008] Extracting features from each gesture image to obtain display information features and action features corresponding to each gesture image;

[0009] sorting the plurality of gesture images according to display information features corresponding to each of the gesture images, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance;

[0010] According to the action features of the gesture image with the highest priority, an operation instruction corresponding to the gesture image with the highest priority is determined, and the operation instruction is sent to the first target home appliance to control the first target home appliance.

[0011] According to a control method for a household appliance provided by the present application, the display information feature includes at least one of a deflection angle of a gesture action, a display area ratio, and a palm orientation;

[0012] The step of prioritizing the plurality of gesture images according to display information features corresponding to each gesture image, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance includes:

[0013] The multiple gesture images are prioritized according to at least one of the deflection angle, display area ratio and palm orientation of the gesture action corresponding to each gesture image, and the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

[0014] According to a control method for a home appliance provided by the present application, the priority of the plurality of gesture images is sorted according to at least one of a deflection angle, a display area ratio, and a palm orientation of a gesture action corresponding to each gesture image, and the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance, including:

[0015] According to the display information features corresponding to each of the gesture images, a deflection angle score, a display area ratio score, and a palm orientation score of each of the gesture images are obtained;

[0016] Obtaining a priority ranking result corresponding to each gesture image according to the weights of the deflection angle, the display area ratio and the palm orientation in the priority ranking of the plurality of gesture images, and the deflection angle score, the display area ratio score and the palm orientation score corresponding to each gesture image;

[0017] According to the priority ranking results of the plurality of gesture images, the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

[0018] According to a control method for a home appliance provided by the present application, the method of prioritizing the plurality of gesture images according to at least one of a deflection angle, a display area ratio, and a palm orientation of a gesture action corresponding to each gesture image, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance, further comprising:

[0019] Sorting the plurality of gesture images according to the display area ratios to obtain a sorting result of gesture area ratios of the plurality of gesture images;

[0020] According to the gesture area ratio ranking result, the home appliance corresponding to the gesture image with the highest gesture area ratio is determined as the first target home appliance.

[0021] According to a control method for a household appliance provided by the present application, after sorting the plurality of gesture images by the display area ratio to obtain a sorting result of the gesture area ratio of the plurality of gesture images, the method further includes:

[0022] If there are multiple gesture images with the highest gesture area ratio, sort the multiple gesture images with the highest gesture area ratio by the deflection angle to obtain a deflection angle sorting result of the multiple gesture images with the highest gesture area ratio;

[0023] The step of determining, according to the gesture area percentage ranking result, the home appliance corresponding to the gesture image having the highest gesture area percentage as the first target home appliance comprises:

[0024] According to the deflection angle sorting result, the home appliance corresponding to the gesture image with the highest gesture area ratio and the smallest deflection angle is determined as the first target home appliance.

[0025] According to a control method for a household appliance provided by the present application, before sorting the plurality of gesture images by the display area ratio to obtain a sorting result of the gesture area ratio of the plurality of gesture images, the method further includes:

[0026] Identify the palm direction of the gesture action in each gesture image;

[0027] According to the recognition results, the gesture images in the palm area are stored to construct a gesture image set;

[0028] The sorting of the plurality of gesture images by the display area ratio to obtain the sorting results of the gesture area ratios of the plurality of gesture images includes:

[0029] Each gesture image in the gesture image set is sorted according to the display area ratio to obtain a sorting result of the gesture area ratio of the gesture images in the gesture image set.

[0030] According to a control method for a household appliance provided by the present application, determining an operation instruction corresponding to the gesture image with the highest priority according to the action characteristics of the gesture image with the highest priority includes:

[0031] According to the action feature of the gesture image with the highest priority, matching a gesture function having the highest similarity to the action feature from a gesture function library corresponding to the first target home appliance;

[0032] Based on the gesture function, an operation instruction corresponding to the gesture image with the highest priority is generated.

[0033] According to a control method for household appliances provided by the present application, the method further includes:

[0034] If, according to the action feature of the gesture image with the highest priority, no gesture function having a similarity with the action feature exceeding a preset threshold value can be matched from the gesture function library corresponding to the first target home appliance, the action feature of the gesture image with the highest priority is matched with the gesture function library corresponding to other home appliances;

[0035] If the gesture function of the second target home appliance is matched, an operation instruction corresponding to the gesture image with the highest priority is generated based on the gesture function of the second target home appliance, and the operation instruction is sent to the second target home appliance.

[0036] The present application also provides a control system for a household appliance, comprising:

[0037] A gesture image acquisition module, used to acquire multiple gesture images of a user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple home appliances with image acquisition functions;

[0038] A gesture feature extraction module, used to extract features from each gesture image to obtain display information features and action features corresponding to each gesture image;

[0039] a gesture action analysis module, configured to prioritize the plurality of gesture images according to display information features corresponding to each of the gesture images, and determine the home appliance corresponding to the gesture image with the highest priority as the first target home appliance;

[0040] A control module is used to determine the operation instruction corresponding to the gesture image with the highest priority according to the action characteristics of the gesture image with the highest priority, and send the operation instruction to the first target home appliance to control the first target home appliance.

[0041] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is run, it executes and implements any of the control methods for the household appliance described above.

[0042] The control method, system and storage medium of household appliances provided in the present application simultaneously collect gesture images of the same gesture action through different household appliances, and analyze the display information in these gesture images to determine whether the gesture action is used to control the corresponding household appliance, thereby improving the recognition accuracy of household appliances based on gesture operation and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 is a schematic diagram of a hardware environment of an interaction method of a smart device according to an embodiment of the present application;

[0046] Figure 2 A flowchart of a method for controlling a household appliance provided in this application;

[0047] Figure 3 A schematic diagram of the structure of the control system of the household appliance provided in this application;

[0048] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0050] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0051] According to one aspect of an embodiment of the present application, a method for controlling a home appliance is provided. The method for controlling a home appliance is widely used in smart home, smart home, smart home device ecosystem, smart home ecosystem and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the method for controlling a home appliance can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0052] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0053] In the existing smart home environment, the camera in the home appliance dynamically captures the portrait and gestures, uploads the collected gesture images to the cloud server for big data matching of the corresponding images, confirms the user's gesture intention, and parses the intention through the cloud function to remotely control the home appliance. With the emergence of various home appliances with gesture operations in smart homes, some gestures are not used to control a certain home appliance, but are mistakenly recognized as control instructions for the home appliance, and some recognition instructions of some home appliances are recognized as control actions of other home appliances, resulting in low accuracy of gesture operations of existing home appliances.

[0054] Figure 2 A flow chart of a control method for household appliances provided in this application, such as Figure 2 As shown, the present application provides a control method for a household appliance, comprising:

[0055] Step 201 : acquiring a plurality of gesture images of a user, wherein the plurality of gesture images are obtained by photographing the same gesture action of the user by a plurality of household electrical appliances with an image acquisition function.

[0056] In this application, the gesture operation of the smart air conditioner is used for explanation. First, the smart air conditioner in sleep mode can be awakened by pressing any button on the remote control, voice wake-up or human sensing sensor. Then, face the direction of the smart air conditioner camera with gestures, and make specific gestures without touching the air conditioner (the gestures are the gestures specified in the product manual, or custom gestures can be used). For example, the user gestures in the direction of the air conditioner camera (or the camera of other home appliances) to make different gestures, representing the corresponding instructions. For example, the gesture is in the shape of "OK". After the smart air conditioner recognizes the gesture, it implements the operation instructions of the air conditioner turning on scene.

[0057] In a smart home environment, the gestures generated by the user will be captured by the cameras of various home appliances. Therefore, when the user's operating intention is uncertain, the gesture image collected by the smart air conditioner itself cannot be directly determined as being used to control the smart air conditioner (in one embodiment, there may be only one home appliance in a room, and the home appliance is a smart air conditioner. At this time, based on the room layout data uploaded to the cloud server in the early stage, the gestures captured by the smart air conditioner in the room can be directly used to control the smart air conditioner). It is necessary to obtain the image capture of the same gesture of the user by all home appliances in the room, so as to obtain multiple gesture images generated by the user at the current moment. It should be noted that in this application, other home appliances refer to home appliances other than smart air conditioners, such as smart TVs, smart refrigerators, and smart washing machines.

[0058] Step 202: extract features from each gesture image to obtain display information features and action features corresponding to each gesture image.

[0059] In the present application, since there may be a smart TV and a smart air conditioner that support gesture recognition in the same room, when the smart TV and the smart air conditioner are triggered to wake up in the same time period, the gestures made by the user in the room are captured by the smart TV and the smart air conditioner at the same time. The display information features can be used to prioritize the gesture images collected by different home appliances, so as to determine the user intention corresponding to the gesture, that is, to determine which home appliance the gesture is actually used to control. The action features are used to match the operation instructions corresponding to the gesture after determining the home appliance corresponding to the gesture, so as to complete the control of the home appliance.

[0060] Step 203, according to the display information feature corresponding to each of the gesture images, the plurality of gesture images are prioritized, and the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance;

[0061] Step 204 , determining an operation instruction corresponding to the gesture image with the highest priority according to the action features of the gesture image with the highest priority, and sending the operation instruction to the first target home appliance to control the first target home appliance.

[0062] In the present application, the direction of the gesture is determined by far-field distributed technology. When the smart air conditioner is identified as the home appliance facing the gesture, the smart air conditioner implements corresponding control according to the gesture, and other home appliances are not controlled by the gesture, so as to realize gesture recognition and control of the nearest direction. Preferably, the present application uploads the collected image data to the cloud server and is located in the same cloud decision distributed server. The server analyzes the contents of the two pictures through the priority evaluation algorithm, thereby scoring the pictures, so as to determine whether the gesture obtained at the current moment is used to control the smart air conditioner according to the scoring result. Specifically, in one embodiment, priority evaluation is mainly performed based on the display information features in each gesture image, including: performing comprehensive logical operations based on the deflection angle of the gesture, the display area ratio of the gesture on the picture, and the palm direction in the gesture, and scoring and evaluating the priority according to the operation results. After the priority is confirmed, the cloud server returns the judgment result to the corresponding home appliance, and starts the control instruction for the home appliance with the highest priority (i.e., the first target home appliance), and the remaining home appliances are not controlled.

[0063] Furthermore, in the present application, after determining that the gesture action in the user's current gesture image is an air-conditioning control gesture, the corresponding operation instructions can be obtained through action feature matching based on the preset air-conditioning gesture instruction database, such as turning the air-conditioning on, off, and controlling the air-conditioning mode, so that the smart air-conditioning can quickly and accurately execute the corresponding gesture actions.

[0064] The control method of household appliances provided in the present application simultaneously collects gesture images of the same gesture action through different household appliances, and analyzes the display information in these gesture images to determine whether the gesture action is used to control the corresponding household appliance, thereby improving the recognition accuracy of household appliances based on gesture operation and improving the user experience.

[0065] Based on the above embodiment, the display information feature includes at least one of a deflection angle of the gesture action, a display area ratio, and a palm direction;

[0066] The step of prioritizing the plurality of gesture images according to display information features corresponding to each gesture image, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance includes:

[0067] The multiple gesture images are prioritized according to at least one of the deflection angle, display area ratio and palm orientation of the gesture action corresponding to each gesture image, and the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

[0068] In the present application, the user's gesture image is collected by the camera of each home appliance. Since the placement of each home appliance in the room is fixed, when the same gesture action is collected, the gesture image collected by each home appliance has a different proportion in the gesture image due to different shooting angles, that is, the display area ratio (Area Ratio) of the entire palm area corresponding to each gesture action in the gesture image. In another embodiment, the deflection angle between the palm surface and the home appliance is obtained according to the deflection angle (Deflection). When the entire palm surface is facing the camera of a home appliance, its deflection angle is 0 degrees. In this embodiment, the maximum deflection angle between the palm surface and the camera of a home appliance is 90 degrees; in another embodiment, the palm orientation in the gesture image can also be obtained to determine whether the gesture image contains a palm area. The gesture images are prioritized by one or more of the above-mentioned deflection angles, display area ratios, and palm orientations, so that the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

[0069] In the present application, priority sorting can be performed by any of the above-mentioned display information features. For example, the home appliance corresponding to the gesture image with a larger display area ratio has a greater intention to be controlled by the user; similarly, the home appliance corresponding to the gesture image with a smaller deflection angle has a greater intention to be controlled by the user; and when prioritizing the palm direction, it is only necessary to determine whether there is a palm area in the gesture image. For example, three gesture images are obtained at the same time, which are taken by a smart refrigerator, a smart air conditioner and a smart TV, and only the gesture image taken by the smart air conditioner contains a palm area, then the gesture image corresponding to the smart air conditioner has the highest priority. It should be noted that in the present application, the priority sorting of palm direction is mainly used in scenes where the placement of home appliances is relatively spacious, such as a smart TV is installed on the north wall of the living room, a smart air conditioner is installed on the south wall, and a smart refrigerator is placed on the east wall. At this time, when the user performs gesture operations on the home appliances, generally only one of the gesture images collected by the home appliances will contain a palm area.

[0070] On the basis of the above embodiment, the method of prioritizing the plurality of gesture images according to at least one of a deflection angle, a display area ratio, and a palm orientation of a gesture action corresponding to each of the gesture images, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance includes:

[0071] According to the display information features corresponding to each of the gesture images, a deflection angle score, a display area ratio score, and a palm orientation score of each of the gesture images are obtained;

[0072] Obtaining a priority ranking result corresponding to each gesture image according to the weights of the deflection angle, the display area ratio and the palm orientation in the priority ranking of the plurality of gesture images, and the deflection angle score, the display area ratio score and the palm orientation score corresponding to each gesture image;

[0073] According to the priority ranking results of the plurality of gesture images, the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

[0074] In the present application, the display information features of each gesture image can be normalized to obtain the score corresponding to each feature, and then the priority ranking result of the gesture image can be obtained according to the preset weight of each feature and the scores of multiple display information features. Specifically, in one embodiment, by normalizing the display information features, the deflection angle score obtained is a maximum of 1 (i.e., when the deflection angle is 0 degrees) and a minimum of 0 (i.e., when the deflection angle is 90 degrees); the maximum value of the display area ratio data is 1, i.e., the entire gesture image is covered by the gesture action (it can be adjusted according to the actual situation, for example, when the proportion reaches 90%, it can be determined to be completely covered by the gesture action), and the minimum value is 0, i.e., there is no gesture action in the image; and the palm orientation score only includes 2 arrays, i.e., 1 and 0. Furthermore, a bubble sort algorithm is used to sort the gesture images according to the weighted sum of the scores in each gesture image. When the gesture image captured by the camera of the smart air conditioner has the highest score, it can be determined that it has the highest priority. At this time, the smart air conditioner is used as the first target home appliance, and according to the gesture actions corresponding to the smart air conditioner, corresponding operation instructions are generated to control the air conditioner.

[0075] According to a control method for a home appliance provided by the present application, the method of prioritizing the plurality of gesture images according to at least one of a deflection angle, a display area ratio, and a palm orientation of a gesture action corresponding to each gesture image, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance, further comprising:

[0076] Sorting the plurality of gesture images according to the display area ratio to obtain a sorting result of the gesture area ratio of the plurality of gesture images;

[0077] According to the gesture area ratio ranking result, the home appliance corresponding to the gesture image with the highest gesture area ratio is determined as the first target home appliance.

[0078] In the present application, in order to improve the efficiency of gesture recognition, sorting can be performed based on a single display information feature. Preferably, in one embodiment, the display area ratio is used, and the size of the area occupied by the gesture action in the gesture image is used as the basis for sorting, so that the home appliance corresponding to the gesture image with the largest gesture area ratio is used as the first target home appliance, and it is determined that the gesture action is used to control the first target home appliance.

[0079] According to a control method for a household appliance provided by the present application, after sorting the plurality of gesture images by the display area ratio to obtain a sorting result of the gesture area ratio of the plurality of gesture images, the method further includes:

[0080] If there are multiple gesture images with the highest gesture area ratio, sort the multiple gesture images with the highest gesture area ratio by the deflection angle to obtain a deflection angle sorting result of the multiple gesture images with the highest gesture area ratio;

[0081] The step of determining, according to the gesture area percentage ranking result, the home appliance corresponding to the gesture image having the highest gesture area percentage as the first target home appliance comprises:

[0082] According to the deflection angle sorting result, the home appliance corresponding to the gesture image with the highest gesture area ratio and the smallest deflection angle is determined as the first target home appliance.

[0083] In the present application, when the display area ratio data is used to prioritize multiple gesture images, there may be a situation where the display area ratio data is the same. Therefore, in one embodiment, it is also necessary to further analyze the gesture images with the same display area ratio data according to the deflection angle. Preferably, in order to more accurately obtain the deflection angle between the palm surface and the home appliance, the image acquisition end of the home appliance can be set to a binocular camera to obtain the spatial position relationship between the palm surface and the smart terminal. In the present application, according to the layout of the home appliance in the room, there are generally no multiple identical and positionally identical home appliances in the same room. Therefore, even if the display area ratio data is the same, the deflection angles of gesture shooting by different home appliances are different. When the entire palm surface is facing the camera of a home appliance, its deflection angle is 0 degrees. In the present application, the maximum deflection angle between the palm surface and the camera of a home appliance is 90 degrees.

[0084] Furthermore, for the case where the display area ratio data is the same, the user's actual operation intention is determined by the deflection angle between the palm of the user and the image acquisition end. That is, for the display area ratio data of the gesture image corresponding to a certain home appliance, when there are gesture images of other home appliances that are the same as it, the gesture image with a smaller deflection angle needs to be determined as the operation that the user's intention is more inclined to.

[0085] On the basis of the above embodiment, before sorting the plurality of gesture images by the display area ratio to obtain the sorting result of the gesture area ratio of the plurality of gesture images, the method further includes:

[0086] Identify the palm direction of the gesture action in each gesture image;

[0087] According to the recognition results, the gesture images in the palm area are stored to construct a gesture image set;

[0088] The sorting of the plurality of gesture images by the display area ratio to obtain the sorting results of the gesture area ratios of the plurality of gesture images includes:

[0089] Each gesture image in the gesture image set is sorted according to the display area ratio to obtain a sorting result of the gesture area ratio of the gesture images in the gesture image set.

[0090] In the present application, when prioritizing gesture images, the validity of the gesture image can be determined based on the palm orientation (Palmorientation), that is, whether the palm is displayed in the gesture image is determined, and two types of results, true and false, are output, and the gesture image with the palm is retained. This is because only if there is a palm area in the gesture image can the gesture image reflect the user's operation intention. For gesture images without a palm area, it may be an image of the back of the hand taken by some home appliances. It can be directly judged that the gesture image is not used to control this home appliance.

[0091] On the basis of the above embodiment, determining the operation instruction corresponding to the gesture image with the highest priority according to the action feature of the gesture image with the highest priority includes:

[0092] According to the action feature of the gesture image with the highest priority, matching a gesture function having the highest similarity to the action feature from a gesture function library corresponding to the first target home appliance;

[0093] Based on the gesture function, an operation instruction corresponding to the gesture image with the highest priority is generated.

[0094] In the present application, the target gesture image (i.e., the gesture image with the highest priority) can be matched with the gesture images stored in advance in the preset gesture function library (such as the gesture function library corresponding to the smart air conditioner), so as to obtain the most similar sample gesture image, and determine the operation instruction corresponding to the target gesture image based on the mapping relationship between the sample gesture image and the operation instruction in the database. Preferably, in the present application, the convolutional neural network can also be trained by using sample gesture images marked with air conditioner operation gesture instruction type labels, so that the trained model can predict the corresponding air conditioner operation gesture recognition results based on the input gesture images, and directly generate the corresponding air conditioner control instructions based on the recognition results.

[0095] According to a control method for household appliances provided by the present application, the method further includes:

[0096] If, according to the action feature of the gesture image with the highest priority, no gesture function having a similarity with the action feature exceeding a preset threshold value can be matched from the gesture function library corresponding to the first target home appliance, the action feature of the gesture image with the highest priority is matched with the gesture function library corresponding to other home appliances;

[0097] If the gesture function of the second target home appliance is matched, an operation instruction corresponding to the gesture image with the highest priority is generated based on the gesture function of the second target home appliance, and the operation instruction is sent to the second target home appliance.

[0098] In the present application, when it is determined that the priority of the gesture image captured by the smart air conditioner is not the highest priority, the current gesture image can be determined to be a gesture action of other home appliances. The gesture action can be analyzed based on the cloud server, and the operation instructions obtained from the analysis can be sent to the corresponding home appliances.

[0099] Preferably, in one embodiment, when the gesture image captured by the smart air conditioner has the highest priority, but the gesture action in the gesture image does not match the corresponding function in the smart air conditioner, including the custom gesture function, if the gesture action is a gesture action of other home appliances, gesture recognition operation can be performed on the smart air conditioner based on the gesture action of other home appliances, and other home appliances can also be controlled. For example, when the user controls the camera of the smart air conditioner with an "OK" gesture, the cloud server will identify the home appliances and their locations under the same network of the current home when receiving the gesture instruction. If there is no "OK" gesture in the custom instruction of the air conditioner, and the sweeping robot at home supports the "OK" gesture, the sweeping robot will automatically turn on and clean the floor.

[0100] The control system of the household appliance provided in the present application is described below. The control system of the household appliance described below and the control method of the household appliance described above can be referenced to each other.

[0101] Figure 3 The schematic diagram of the control system of the household appliance provided in this application is as follows: Figure 3As shown, the present application provides a control system for household appliances, including a gesture image acquisition module 301, a gesture feature extraction module 302, a gesture action analysis module 303 and a control module 304, wherein the gesture image acquisition module 301 is used to obtain multiple gesture images of a user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple household appliances with image acquisition functions; the gesture feature extraction module 302 is used to extract features from each of the gesture images to obtain display information features and action features corresponding to each of the gesture images; the gesture action analysis module 303 is used to prioritize the multiple gesture images according to the display information features corresponding to each of the gesture images, and determine the household appliance corresponding to the gesture image with the highest priority as the first target household appliance; the control module 304 is used to determine the operation instruction corresponding to the gesture image with the highest priority according to the action features of the gesture image with the highest priority, and send the operation instruction to the first target household appliance to control the first target household appliance.

[0102] The control system for household appliances provided in the present application simultaneously collects gesture images of the same gesture action through different household appliances, and analyzes the display information in these gesture images to determine whether the gesture action is used to control the corresponding household appliance, thereby improving the recognition accuracy of household appliances based on gesture operation and improving the user experience.

[0103] Based on the above embodiment, the display information feature includes at least one of a deflection angle of the gesture action, a display area ratio, and a palm direction;

[0104] The gesture action analysis module includes a priority sorting unit, which is used to prioritize the multiple gesture images according to at least one of the deflection angle, display area ratio and palm orientation of the gesture action corresponding to each gesture image, and determine the home appliance corresponding to the gesture image with the highest priority as the first target home appliance.

[0105] Based on the above embodiment, the priority sorting unit is specifically used for:

[0106] According to the display information features corresponding to each of the gesture images, a deflection angle score, a display area ratio score, and a palm orientation score of each of the gesture images are obtained;

[0107] Obtaining a priority ranking result corresponding to each gesture image according to the weights of the deflection angle, the display area ratio and the palm orientation in the priority ranking of the plurality of gesture images, and the deflection angle score, the display area ratio score and the palm orientation score corresponding to each gesture image;

[0108] According to the priority ranking results of the plurality of gesture images, the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

[0109] Based on the above embodiment, the priority sorting unit is further used for:

[0110] Sorting the plurality of gesture images according to the display area ratio to obtain a sorting result of the gesture area ratio of the plurality of gesture images;

[0111] According to the gesture area ratio ranking result, the home appliance corresponding to the gesture image with the highest gesture area ratio is determined as the first target home appliance.

[0112] Based on the above embodiment, the priority sorting unit is further used for:

[0113] If there are multiple gesture images with the highest gesture area ratio, sort the multiple gesture images with the highest gesture area ratio by the deflection angle to obtain a deflection angle sorting result of the multiple gesture images with the highest gesture area ratio;

[0114] According to the deflection angle sorting result, the home appliance corresponding to the gesture image with the highest gesture area ratio and the smallest deflection angle is determined as the first target home appliance.

[0115] Based on the above embodiment, the gesture action analysis module is further used for:

[0116] Identify the palm direction of the gesture action in each gesture image;

[0117] According to the recognition results, the gesture images in the palm area are stored to construct a gesture image set;

[0118] The priority sorting unit is specifically used for:

[0119] Each gesture image in the gesture image set is sorted according to the display area ratio to obtain a sorting result of the gesture area ratio of the gesture images in the gesture image set.

[0120] Based on the above embodiment, the control module includes a gesture action matching unit and an instruction generating unit, wherein the gesture action matching unit is used to match the gesture function with the highest similarity to the action feature from the gesture function library corresponding to the first target home appliance according to the action feature of the gesture image with the highest priority; the instruction generating unit is used to generate an operation instruction corresponding to the gesture image with the highest priority based on the gesture function.

[0121] Based on the above embodiment, the control module is further used for:

[0122] If, according to the action feature of the gesture image with the highest priority, no gesture function having a similarity with the action feature exceeding a preset threshold value can be matched from the gesture function library corresponding to the first target home appliance, the action feature of the gesture image with the highest priority is matched with the gesture function library corresponding to other home appliances;

[0123] If the gesture function of the second target home appliance is matched, an operation instruction corresponding to the gesture image with the highest priority is generated based on the gesture function of the second target home appliance, and the operation instruction is sent to the second target home appliance.

[0124] Figure 4 A schematic diagram of the structure of the electronic device provided in this application, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the control method of the home appliance, the method comprising: acquiring multiple gesture images of the user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple home appliances with image acquisition functions; extracting features from each gesture image to obtain display information features and action features corresponding to each gesture image; prioritizing the multiple gesture images according to the display information features corresponding to each gesture image, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance; determining the operation instruction corresponding to the gesture image with the highest priority according to the action features of the gesture image with the highest priority, and sending the operation instruction to the first target home appliance to control the first target home appliance.

[0125] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0126] On the other hand, the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the control method of the home appliance provided by the above-mentioned methods, and the method includes: obtaining multiple gesture images of the user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple home appliances with image acquisition functions; performing feature extraction on each of the gesture images to obtain display information features and action features corresponding to each of the gesture images; prioritizing the multiple gesture images according to the display information features corresponding to each of the gesture images, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance; determining the operation instruction corresponding to the gesture image with the highest priority according to the action features of the gesture image with the highest priority, and sending the operation instruction to the first target home appliance to control the first target home appliance.

[0127] On the other hand, the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein the program, when running, executes the control method of the home appliance provided by the above-mentioned methods, the method comprising: obtaining multiple gesture images of the user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple home appliances with image acquisition functions; performing feature extraction on each of the gesture images to obtain display information features and action features corresponding to each of the gesture images; prioritizing the multiple gesture images according to the display information features corresponding to each of the gesture images, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance; determining the operation instruction corresponding to the gesture image with the highest priority according to the action features of the gesture image with the highest priority, and sending the operation instruction to the first target home appliance to control the first target home appliance.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for controlling a household appliance, characterized in that: include: Acquire multiple gesture images of the user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple home appliances with image acquisition functions; Extracting features from each gesture image to obtain display information features and action features corresponding to each gesture image; sorting the plurality of gesture images according to display information features corresponding to each of the gesture images, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance; Determine, according to the action features of the gesture image with the highest priority, an operation instruction corresponding to the gesture image with the highest priority, and send the operation instruction to the first target home appliance to control the first target home appliance; The display information feature includes at least one of a deflection angle of the gesture action, a display area ratio, and a palm orientation; The step of prioritizing the plurality of gesture images according to display information features corresponding to each gesture image, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance includes: sorting the plurality of gesture images according to at least one of a deflection angle, a display area ratio, and a palm orientation of a gesture action corresponding to each of the gesture images, and determining a home appliance corresponding to a gesture image with a highest priority as a first target home appliance; The method of prioritizing the plurality of gesture images according to at least one of a deflection angle, a display area ratio, and a palm orientation of a gesture action corresponding to each of the gesture images, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance includes: According to the display information features corresponding to each of the gesture images, a deflection angle score, a display area ratio score and a palm orientation score of each of the gesture images are obtained, wherein when the entire palm is facing the camera of the home appliance, the deflection angle is 0 degrees, and the maximum deflection angle between the palm and the camera of the home appliance is 90 degrees; by normalizing the display information features, it is obtained that the maximum deflection angle score when the deflection angle is 0 degrees is 1, and the minimum deflection angle score when the deflection angle is 90 degrees is 0; when the value of the display area ratio data is 1 at most, it indicates that the entire gesture image is covered by the gesture action; when the value of the display area ratio data is 0 at least, it indicates that there is no gesture action in the gesture image; the palm orientation score includes 1 and 0; Obtaining a priority ranking result corresponding to each gesture image according to the weights of the deflection angle, the display area ratio and the palm orientation in the priority ranking of the plurality of gesture images, and the deflection angle score, the display area ratio score and the palm orientation score corresponding to each gesture image; According to the priority ranking results of the plurality of gesture images, the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

2. The control method of household electrical appliances according to claim 1, characterized in that: The step of prioritizing the plurality of gesture images according to at least one of a deflection angle, a display area ratio, and a palm orientation of a gesture action corresponding to each of the gesture images, and determining the home appliance corresponding to the gesture image with the highest priority as the first target home appliance, further includes: Sorting the plurality of gesture images according to the display area ratios to obtain a sorting result of gesture area ratios of the plurality of gesture images; According to the gesture area ratio ranking result, the home appliance corresponding to the gesture image with the highest gesture area ratio is determined as the first target home appliance.

3. The control method of household electrical appliances according to claim 2, characterized in that: After sorting the plurality of gesture images by the display area ratio to obtain a sorting result of the gesture area ratio of the plurality of gesture images, the method further includes: If there are multiple gesture images with the highest gesture area ratio, sort the multiple gesture images with the highest gesture area ratio by the deflection angle to obtain a deflection angle sorting result of the multiple gesture images with the highest gesture area ratio; The step of determining, according to the gesture area percentage ranking result, the home appliance corresponding to the gesture image having the highest gesture area percentage as the first target home appliance comprises: According to the deflection angle sorting result, the home appliance corresponding to the gesture image with the highest gesture area ratio and the smallest deflection angle is determined as the first target home appliance.

4. The control method of household electrical appliances according to claim 2, characterized in that: Before sorting the plurality of gesture images by the display area ratio to obtain a sorting result of the gesture area ratio of the plurality of gesture images, the method further includes: Identify the palm direction of the gesture action in each gesture image; According to the recognition results, the gesture images in the palm area are stored to construct a gesture image set; The sorting of the plurality of gesture images by the display area ratio to obtain the sorting results of the gesture area ratios of the plurality of gesture images includes: Each gesture image in the gesture image set is sorted according to the display area ratio to obtain a sorting result of the gesture area ratio of the gesture images in the gesture image set.

5. The control method of a household appliance according to any one of claims 1 to 4, characterized in that: Determining, according to the action feature of the gesture image with the highest priority, the operation instruction corresponding to the gesture image with the highest priority, includes: According to the action feature of the gesture image with the highest priority, matching a gesture function having the highest similarity to the action feature from a gesture function library corresponding to the first target home appliance; Based on the gesture function, an operation instruction corresponding to the gesture image with the highest priority is generated.

6. The control method of household electrical appliances according to claim 5, characterized in that: The method further comprises: If, according to the action feature of the gesture image with the highest priority, no gesture function having a similarity with the action feature exceeding a preset threshold value can be matched from the gesture function library corresponding to the first target home appliance, the action feature of the gesture image with the highest priority is matched with the gesture function library corresponding to other home appliances; If the gesture function of the second target home appliance is matched, an operation instruction corresponding to the gesture image with the highest priority is generated based on the gesture function of the second target home appliance, and the operation instruction is sent to the second target home appliance.

7. A control system for household electrical appliances, characterized in that: include: A gesture image acquisition module, used to acquire multiple gesture images of a user, wherein the multiple gesture images are obtained by photographing the same gesture action of the user by multiple home appliances with image acquisition functions; A gesture feature extraction module, used to extract features from each gesture image to obtain display information features and action features corresponding to each gesture image; a gesture action analysis module, configured to prioritize the plurality of gesture images according to display information features corresponding to each of the gesture images, and determine the home appliance corresponding to the gesture image with the highest priority as the first target home appliance; a control module, configured to determine an operation instruction corresponding to the gesture image with the highest priority according to an action feature of the gesture image with the highest priority, and send the operation instruction to the first target home appliance to control the first target home appliance; The display information feature includes at least one of a deflection angle of the gesture action, a display area ratio, and a palm orientation; The gesture action analysis module includes a priority sorting unit, which is used to prioritize the multiple gesture images according to at least one of the deflection angle, display area ratio and palm orientation of the gesture action corresponding to each gesture image, and determine the home appliance corresponding to the gesture image with the highest priority as the first target home appliance; The priority sorting unit is specifically used for: According to the display information features corresponding to each of the gesture images, a deflection angle score, a display area ratio score and a palm orientation score of each of the gesture images are obtained, wherein when the entire palm is facing the camera of the home appliance, the deflection angle is 0 degrees, and the maximum deflection angle between the palm and the camera of the home appliance is 90 degrees; by normalizing the display information features, it is obtained that the maximum deflection angle score when the deflection angle is 0 degrees is 1, and the minimum deflection angle score when the deflection angle is 90 degrees is 0; when the value of the display area ratio data is 1 at most, it indicates that the entire gesture image is covered by the gesture action; when the value of the display area ratio data is 0 at least, it indicates that there is no gesture action in the gesture image; the palm orientation score includes 1 and 0; Obtaining a priority ranking result corresponding to each gesture image according to the weights of the deflection angle, the display area ratio and the palm orientation in the priority ranking of the plurality of gesture images, and the deflection angle score, the display area ratio score and the palm orientation score corresponding to each gesture image; According to the priority ranking results of the plurality of gesture images, the home appliance corresponding to the gesture image with the highest priority is determined as the first target home appliance.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

Images