Electric appliance, control method and device thereof, storage medium and computer program product

Abstract

The invention provides an electric appliance, a control method and device thereof, a storage medium and a computer program product. The method comprises the following steps: acquiring first motion data acquired in advance; performing model training by using the first motion data to obtain an identification model; collecting second motion data of a target user, and identifying the second motion data by using the identification model to obtain an identification result of the second motion data; displaying an identification result of the second motion data to a target user, and determining whether the identification result is accurate or not by the target user so as to update the identification model; collecting third motion data of the target user, and identifying the third motion data by using the updated identification model to obtain an identification result of the third motion data; and generating a control instruction of the corresponding electric appliance according to an identification result of the third motion data, and sending the control instruction to the corresponding electric appliance to control the corresponding electric appliance to execute the control instruction. According to the scheme provided by the invention, the accuracy of the multi-modal interaction system can be improved.

Description

Technical Field

[0001] The present invention relates to the field of control, and in particular to an electrical appliance and a control method, device, storage medium and computer program product thereof. Background Art

[0002] Multimodal interaction systems in related technologies usually combine multiple interaction methods such as vision, sound and touch to improve the naturalness and efficiency of human-computer interaction. For example, some smart home systems operate devices through voice recognition and gesture control, while advanced virtual reality systems may combine full-body motion capture and facial expression recognition to enhance the user's immersion.

[0003] The motion capture, expression recognition and gesture recognition technologies in the related technologies still have room for improvement in accuracy, especially in complex environments or when user movements change rapidly, the recognition error rate is high. Summary of the invention

[0004] The main purpose of the present invention is to overcome the defects of the above-mentioned related technologies and to provide an electrical appliance and a control method, device, storage medium and computer program product thereof to solve the problem of low accuracy of motion capture, expression recognition and gesture recognition technologies in the related technologies.

[0005] On the one hand, the present invention provides a control method for an electrical appliance, comprising: acquiring first motion data collected in advance; the first motion data comprises: at least one of action data, expression data and gesture data; using the first motion data to perform model training to obtain a recognition model; the recognition model comprises: at least one of an action recognition model, an expression recognition model and a gesture recognition model; collecting second motion data of a target user, and using the trained recognition model to recognize the second motion data to obtain a recognition result of the second motion data; displaying the recognition result of the recognized second motion data to the target user, and having the target user confirm whether the recognition result is accurate to update the recognition model; collecting third motion data of the target user, and using the updated recognition model to recognize the third motion data to obtain a recognition result of the third motion data; generating a control instruction for the corresponding electrical appliance based on the recognition result of the third motion data, and sending it to the corresponding electrical appliance to control the corresponding electrical appliance to execute the control instruction.

[0006] Optionally, using the first motion data to perform model training to obtain a recognition model includes: preprocessing the first motion data to obtain a training data set, wherein the preprocessing includes: at least one of data cleaning, data standardization, and data labeling; and using the obtained training data set to perform model training to obtain the recognition model.

[0007] Optionally, the data cleaning includes: processing the first motion data using a filtering algorithm; using the first motion data to perform model training to obtain a recognition model, and also includes: processing the first motion data by polynomial fitting before using the filtering algorithm to process the first motion data.

[0008] Optionally, collecting the first motion data, the second motion data and / or the third motion data includes: capturing human motion data through a Kinect and / or Vicon motion capture system; and / or, capturing facial expression data through a facial recognition camera equipped with OpenCV and / or Dlib library; and / or, capturing gesture data through a depth camera.

[0009] Optionally, the recognition result of the second motion data is displayed to the target user, and the target user confirms whether the recognition result is accurate, including: displaying the recognition result of the second motion data and / or voice broadcasting the recognition result of the second motion data; receiving confirmation information from the target user confirming whether the recognition result of the second motion data is accurate.

[0010] Optionally, it also includes: if the confirmation information received from the target user is that the recognition result of the second motion data is inaccurate, saving the second motion data for iterative training of the recognition model; and / or, if the confirmation information received from the target user is that the recognition result of the second motion data is inaccurate, saving the second motion data as the motion recognition data of the current target user.

[0011] On the other hand, the present invention provides a control device for an electrical appliance, comprising: an acquisition unit, used to acquire first motion data collected in advance; the first motion data includes: at least one of action data, expression data and gesture data; a training unit, used to perform model training using the first motion data to obtain a recognition model; the recognition model includes: at least one of an action recognition model, an expression recognition model and a gesture recognition model; a first recognition unit, used to collect second motion data of a target user, and use the trained recognition model to recognize the second motion data to obtain a recognition result of the second motion data; a confirmation unit, used to display the recognition result of the second motion data obtained by recognition to the target user, and the target user confirms whether the recognition result is accurate to update the recognition model; a second recognition unit, used to collect third motion data of the target user, and use the updated recognition model to recognize the third motion data to obtain a recognition result of the third motion data; a control unit, used to generate a control instruction for the corresponding electrical appliance according to the recognition result of the third motion data, and send it to the corresponding electrical appliance to control the corresponding electrical appliance to execute the control instruction.

[0012] Optionally, the training unit uses the first motion data to perform model training to obtain a recognition model, including: preprocessing the first motion data to obtain a training data set, the preprocessing including: at least one of data cleaning, data standardization and data labeling; using the obtained training data set to perform model training to obtain the recognition model.

[0013] Optionally, the data cleaning includes: using a filtering algorithm to process the first motion data; the training unit uses the first motion data to perform model training to obtain a recognition model, and also includes: before using the filtering algorithm to process the first motion data, processing the first motion data through polynomial fitting.

[0014] Optionally, collecting the first motion data, the second motion data and / or the third motion data includes: capturing human motion data through a Kinect and / or Vicon motion capture system; and / or, capturing facial expression data through a facial recognition camera equipped with OpenCV and / or Dlib library; and / or, capturing gesture data through a depth camera.

[0015] Optionally, the confirmation unit displays the recognition result of the second motion data to the target user, and the target user confirms whether the recognition result is accurate, including: displaying the recognition result of the second motion data and / or voice broadcasting the recognition result of the second motion data; receiving confirmation information from the target user confirming whether the recognition result of the second motion data is accurate.

[0016] Optionally, it also includes: a saving unit, which is used to save the second motion data for iterative training of the recognition model if the confirmation information received from the target user is that the recognition result of the second motion data is inaccurate; and / or, if the confirmation information received from the target user is that the recognition result of the second motion data is inaccurate, save the second motion data as the motion recognition data of the current target user.

[0017] Another aspect of the present invention provides a storage medium having a computer program stored thereon, wherein the program implements the steps of any of the aforementioned methods when executed by a processor.

[0018] In another aspect, the present invention provides an electrical appliance, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned methods when executing the program.

[0019] Another aspect of the present invention provides an electrical appliance, comprising any of the control devices described above.

[0020] In yet another aspect, the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0021] According to the technical solution of the present invention, the user confirmation mechanism is used to improve the accuracy of recognition, and the precision of the multimodal interactive system can be improved. The curve fitting technology is used to ensure the recognition accuracy of actions, expressions and gestures, and reduce the recognition error rate in complex environments or when the user's actions change rapidly.

[0022] According to the technical solution of the present invention, a unified action, expression and gesture recognition model is used to ensure that users can use a consistent interaction method between different devices and systems, reduce learning costs, and improve the consistency of user experience.

[0023] According to the technical solution of the present invention, by optimizing the data preprocessing and model training process, the system's real-time response capability to multimodal data processing is improved, the natural fluency of the interaction is ensured, and the real-time performance of data processing is enhanced.

[0024] According to the technical solution of the present invention, intelligent adjustments can be made according to the habits and preferences of users, providing a more personalized appliance control experience and achieving personalized adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a method schematic diagram of an embodiment of a method for controlling an electrical appliance provided by the present invention;

[0027] Figure 2 is a method schematic diagram of a specific embodiment of the control method of an electrical appliance provided by the present invention;

[0028] Figure 3 It is a structural block diagram of an embodiment of the control device of an electrical appliance provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The deficiencies of the technical solutions in the relevant technologies mainly include: Accuracy issues: There is still room for improvement in the accuracy of motion capture and expression recognition technologies, especially in complex environments or when the user's movements change rapidly, the recognition error rate is high. Consistency of user experience: The interaction methods between different devices and systems are not unified, which requires users to learn multiple interaction modes, increasing the complexity of use. Real-time data processing: When processing large amounts of multimodal data, the system's real-time response capabilities are insufficient, affecting the natural fluency of the interaction. Personalized adaptability: Most systems lack the ability to adapt to users' personalized needs and cannot make intelligent adjustments based on users' habits and preferences. Integration and compatibility: Integration and compatibility issues between different technologies and devices limit the scalability and flexibility of the system.

[0032] The present invention provides a control method for an electrical appliance, wherein the electrical appliance may be at least one of an air conditioner, a television, a refrigerator, and a washing machine.

[0033] Figure 1 It is a method schematic diagram of an embodiment of the method for controlling an electrical appliance provided by the present invention.

[0034] like Figure 1 As shown, according to an embodiment of the present invention, the control method of the electrical appliance at least includes step S110, step S120, step S130, step S140, step S150 and step S160.

[0035] Step S110, obtaining pre-collected first motion data.

[0036] The first motion data may specifically include: at least one of action data, expression data and gesture data.

[0037] The motion data may specifically be multi-angle human motion data. For example, cameras are set at different positions in the room to collect multi-angle human motion data. For example, in order to achieve high-precision full-body motion capture, human motion data can be captured by Kinect and / or Vicon motion capture systems, and the user's full-body motion can be captured from multiple angles by cameras set in different corners of the room. The Kinect system (body tracking system) uses infrared depth sensors and RGB cameras, while Vicon (optical motion capture system) uses high-precision optical cameras and marker technology. Both systems can record the user's walking, waving, shaking, sneezing and other actions in real time to ensure the comprehensiveness and accuracy of the data.

[0038] For facial expression data, facial expression data can be captured through facial recognition cameras equipped with OpenCV and / or Dlib libraries. These cameras can be specifically used to capture the user's facial expressions, such as smile, frown, blush, etc. OpenCV (cross-platform computer vision library) and Dlib (machine learning algorithm library) libraries provide powerful image processing and facial feature point detection functions, which can accurately identify and record the changes in the user's facial expressions.

[0039] For gesture data collection, in order to accurately capture the user's gestures, a depth camera can be used to capture the user's gestures. For example, the Leap Motion depth camera can capture the user's gestures, such as pointing and grabbing, through its high-precision sensors. This depth camera can provide detailed hand movement data and provide high-quality input for subsequent gesture recognition models.

[0040] Step S120: Perform model training using the first motion data to obtain a recognition model.

[0041] The collected first motion data may include: at least one of action data, expression data and gesture data. Correspondingly, the recognition model includes at least one of an action recognition model, an expression recognition model and a gesture recognition model.

[0042] In a specific implementation, training the recognition model mainly includes the following steps S1 to S2;

[0043] Step S1, preprocessing the first motion data to obtain a training data set.

[0044] The data preprocessing may specifically include: data cleaning, data standardization and data labeling.

[0045] Data cleaning may specifically include: using a filtering algorithm to process the acquired data to remove noise and outliers; the filtering algorithm preferably uses a median filtering algorithm. For example, the original signal: [10, 12, 11, 100, 13, 14, 15], in this signal 100 is obviously an outlier because it is much larger than other data values, then use the median filter, first assume that the window size is 3, then slide the window to calculate the median, the first window [10, 12, 11], first sort from small to large [10, 11, 12], then take the middle value 11, then move the window [12, 11, 100], repeat the above steps and so on, and finally put all the medians into a window [11, 12, 13, 14, 14], this is the filtered signal, it can be seen that the outlier 100 in the original signal is effectively removed, and the filtered signal is smoother and closer to the true value. For facial expression data and gesture data, image processing technology (such as edge detection) is used to improve the quality of the data.

[0046] Preferably, before using the filtering algorithm to process the first motion data, the first motion data is processed by polynomial fitting; in a specific implementation, the least squares method can be used to perform polynomial fitting, and the obtained polynomial is used to process the first motion data, that is, the least squares method is used to calculate the polynomial coefficients. Assume that there are the following time steps and corresponding joint angle data: time step: x (1 2 3 4 5 6 7) joint angle y (1012 11 100 13 14 15), assuming that the order of the polynomial is m = 2m = 2, that is, the quadratic polynomial P (x) = a 0 +a 1 x+a 2 x 2 , use the least squares method to calculate the coefficients a of the polynomial 0 、a 1 and a 2 The goal of the least squares method is to minimize the sum of squared errors. Assuming that a specific coefficient a is obtained 0 ,a 1 ,a 2 For example: 0 =10,a 1 =0.5,a 2 =-0.01, and use the calculated polynomial coefficients to generate the fitting curve P(x): P(x)=10+0.5x-0.01x 2 , substitute the time steps (1 2 3 4 5 6 7) respectively and get the smoothed data [10, 11, 11.5, 12, 12.5, 13, 13.5].

[0047] The purpose of using polynomial fitting is to smooth the original user data and remove outliers. Although there is a median filter algorithm to remove outliers during model training, it is very important for image models to process data outliers to ensure data smoothness. Therefore, polynomial fitting technology is used to process data before median filtering the model. Because, if there are two consecutive outliers, such as (10, 100, 102), the median filter will take 100, and the outlier will not be removed in the end. After fitting, median filtering is performed to make the median filtering process more accurate, and the model training begins after filtering.

[0048] Data standardization is the process of converting the collected data into a uniform format and scale for easy subsequent processing. For example, motion capture data is standardized to a range of 0-1, and expression data is normalized.

[0049] Data annotation is to assign labels to each action, expression and / or gesture. These labels will be used as part of the training data set to guide the model's learning process. For example, if a smiling expression is used as a training set input, the output result is the label "smile".

[0050] Step S2, using the obtained training data set to perform model training to obtain the recognition model.

[0051] The recognition model may specifically be a neural network model. For example, the gesture recognition model may be a deep neural network model DNN, the expression recognition model may be a convolutional neural network model CNN, and the action recognition model may be a long short-term memory network model LSTM or a convolutional neural network model CNN.

[0052] In a specific embodiment, a long short-term memory network (LSTM) is used to train the recognition model. The long short-term memory network model can process sequence data, such as time series data, that is, the collected motion data is motion data with time characteristics. For example, when collecting data, every preset time interval can be used as a time step to collect the motion data corresponding to each time step. For example, taking the action model as an example, it is assumed that there is a set of motion capture data that records a person's walking action. The data includes joint angles and speed information for each time step. The joint angles and speeds of each time step are collected (through Kinect or Vicon motion capture system). During data preprocessing, median filtering is used to remove outliers, and then the data is segmented into multiple time series segments by data segmentation, each segment corresponding to an action. For example, the processed result data is as follows: time step 1: [10, 12, 11]; time step 2: [12, 11, 13]; time step 3: [11, 13, 14]; time step 4: [13, 14, 15]; time step 5: [14, 15, 16]).

[0053] The collected time series data is used as the input of the LSTM network, and the corresponding labels (action, expression and / or gesture labels) are used as the output to obtain the corresponding recognition model. The labels output by the recognition model are compared with the true labels to calculate the cross entropy loss, and the gradient is calculated through the back propagation algorithm to update the parameters of the LSTM layer. Then, iterative training is performed to ensure that the model can accurately recognize actions, labels and / or gestures.

[0054] Specifically, the collected time series data is used as the input of the LSTM network input layer, and then enters the LSTM layer. The LSTM layer contains multiple LSTM units, each of which processes a time step of data, calculates the hidden state and output, and the output layer obtains the labels of action categories, expression categories and / or gesture categories through the classification algorithm. Then the model output is compared with the true label to calculate the cross entropy loss. If the difference is large, the gradient is calculated through the back propagation algorithm, the parameters of the LSTM layer are updated, and then iterative training is performed to ensure that the model can accurately recognize the walking action.

[0055] For example, a set of walking action data is used as the input of the LSTM network input layer, and then enters the LSTM layer, which contains multiple LSTM units, each unit processes a time step of data, for example, time step 1: input [10,12,11], calculate the hidden state and output. Time step 2: input [12,11,13], combined with the hidden state of the previous time step, calculate the new hidden state and output. Time step 3: and so on..., the output layer obtains a label of the action category through the classification algorithm, and then compares the model output with the true label to calculate the cross entropy loss. If the difference is large, the gradient is calculated through the back propagation algorithm, the parameters of the LSTM layer are updated, and then iterative training is performed to ensure that the model can accurately recognize the walking action. For the expression recognition model and the gesture recognition model, the LSTM (long short-term memory network) model can also be used. The input is the preprocessed expression or gesture image, and the output is the expression or gesture label, which is then compared with the true label to calculate the cross entropy loss. Then the parameters are updated and iterative training is repeated to ensure the accuracy of the model.

[0056] Step S130: collect the second motion data of the target user, and use the trained recognition model to recognize the second motion data to obtain a recognition result of the second motion data.

[0057] Specifically, the second motion data of the target user is collected in the same collection method as the first motion data. The second motion data is the same as the first motion data, and may specifically include: at least one of motion data, expression data, and gesture data. For example, human motion data is captured by Kinect and / or Vicon motion capture system; and / or, facial expression data is captured by a facial recognition camera equipped with OpenCV and / or Dlib library; and / or, gesture data is captured by a depth camera.

[0058] The second motion data is recognized by using the recognition model obtained through training to obtain a recognition result of the second motion data, i.e., a simulation mapping result. The recognition result is specifically a label obtained by recognizing the second motion data. When the model is trained, the input of the training set is an image with image features (such as expression, action), and the output result is a result label of the image.

[0059] Specifically, the collected second motion data (collected pictures) is used to generate multiple time series segments, each of which is a mapping of a motion segment (action, expression, gesture). That is, the action features in the picture are converted into numbers, such as [10,12,11], which records the image features of three dimensions. Combining multiple such segments during this period can identify the user's action trajectory during this period. The training set has enabled the model to recognize the user's actions in these images through iterative training.

[0060] Step S140 , displaying the recognition result of the second motion data to the target user, and having the target user confirm whether the recognition result is accurate, so as to update the recognition model.

[0061] In a specific embodiment, the recognition result of the second motion data is displayed and / or the recognition result of the second motion data is broadcasted by voice; and confirmation information from the target user is received to confirm whether the recognition result of the second motion data is accurate. That is, the recognition result is displayed on a display screen and / or the recognition result is converted into voice and broadcasted by the voice module of the device, and confirmation information from the user is received. The user can provide feedback through a preset button (such as a button on the interactive interface) or through voice to confirm the accuracy of the recognition result.

[0062] For example, the user stands in the middle of the room, and the air conditioner's camera begins to capture his whole body movements. He first walks slowly for a few steps, then waves his hand, and finally makes a jumping motion. The air conditioner's display screen immediately displays the labels "walking", "waving", and "jumping", indicating that the system has accurately recognized his movements. Then the user makes several expressions to the air conditioner's facial recognition camera: smile, frown, and surprise. The air conditioner's display screen displays the labels "smile", "frown", and "surprise" in turn, indicating that the system has successfully captured and recognized his facial expressions. The user reaches out and makes several gestures to the air conditioner's camera: point to the air conditioner, grab, and rotate. The air conditioner's display screen displays the labels "point", "grab", and "rotate", indicating that the system has accurately recognized his gestures.

[0063] Optionally, the method further comprises: if the confirmation information fed back by the target user is received that the recognition result of the second motion data is inaccurate, saving the second motion data for iterative training of the recognition model.

[0064] That is, if the target user feedbacks that the recognition result is inaccurate, the second motion data is saved for iterative training of the model, updating the model parameters, and improving the accuracy of model recognition.

[0065] Optionally, the method further comprises: if the confirmation information fed back by the target user indicates that the recognition result of the second motion data is inaccurate, saving the second motion data as the motion recognition data of the target user. That is, when the user makes the same action, expression and / or gesture again, recognition can be performed directly according to the motion recognition data.

[0066] In an actual scenario, the display screen gives a text description of the user's action, expression or gesture. If it matches the action, expression or gesture made by the user, the user can choose yes. If it does not match, it means that the appliance cannot recognize the action, expression or gesture. This function can be used to test people whose actions are relatively inconvenient (such as the elderly). For example, the actions of the elderly may be inconvenient. If the action made by the elderly cannot be recognized by the appliance, and it is also the elderly's habitual action, such as the elderly want to use this action to tell the air conditioner to increase the temperature, then the image and label of this action (such as increasing the temperature) can be input, and this action can be stored. The next time the elderly do this action, the air conditioner can recognize it. This is an additional method to make up for the incorrect recognition. After accurate recognition, the corresponding appliance can be controlled by action, such as controlling the air conditioner. Shivering means cold, and the air conditioner will increase the temperature. The same is true for gestures and expressions.

[0067] Step S150: collect the third motion data of the target user, and use the updated recognition model to recognize the third motion data to obtain a recognition result of the third motion data.

[0068] Specifically, after the recognition model is updated, the updated recognition model can be used to recognize the third motion data of the target user collected in real time, and the recognition result of the third motion data can be obtained, that is, the label of the third motion data of the target user is output. The third motion data is the same as the first motion data and the second motion data, and includes at least one of action data, expression data and gesture data. The third motion data of the target user can be collected in the same way as the first and second motion data.

[0069] Step S160: Generate a control instruction for the corresponding electrical appliance according to the recognition result of the third motion data, and send the control instruction to the corresponding electrical appliance to control the corresponding electrical appliance to execute the control instruction.

[0070] Specifically, the control instruction is sent to the corresponding electrical appliance (e.g., smart air conditioner) by communicating with the corresponding electrical appliance (e.g., Zigbee or Wi-Fi protocol) through a short-range wireless communication technology (e.g., Zigbee or Wi-Fi protocol). The trained model and curve fitting algorithm are integrated into a central processing unit of an air conditioner (e.g., Raspberry Pi or Arduino), and communicate with the smart air conditioner through the Zigbee or Wi-Fi protocol. This integration method has the advantage of reducing chip costs over directly integrating the trained model and curve fitting algorithm into the air conditioner controller, and can achieve efficient system control and data transmission.

[0071] Based on the user's motion, expression and gesture recognition results, the central processing unit generates corresponding control instructions, which are sent to the electrical appliances through Zigbee or Wi-Fi protocols and executed by the electrical appliances. For example, if the appliance is a smart air conditioner, the motion, expression and gesture recognition results are sent to the air conditioner to adjust parameters such as temperature and wind speed. This intelligent control logic can provide a personalized air conditioning control experience based on the actual needs of the user.

[0072] In order to clearly illustrate the technical solution of the present invention, the execution process of the control method of the electrical appliance provided by the present invention is described below with reference to a specific embodiment.

[0073] Figure 2 FIG. 1 is a schematic diagram of a specific embodiment of the control method of an electrical appliance provided by the present invention. Figure 2As shown, first, data collection is performed, and human motion data is captured by Kinect and / or Vicon motion capture system, facial expression data is captured by facial recognition camera, and user gestures are captured by Leap Motion depth camera. After that, data preprocessing is performed, including data cleaning, data standardization and data labeling. After that, recognition models are trained, including gesture recognition model, expression recognition model and action recognition model; then simulation mapping and user confirmation are performed, and then system integration and control are performed, that is, integration into the central processing unit of the appliance to control the appliance (air conditioner).

[0074] The present invention provides a control device for an electrical appliance, wherein the electrical appliance may be at least one of an air conditioner, a television, a refrigerator, and a washing machine.

[0075] Figure 3 FIG. 1 is a structural block diagram of an embodiment of the control device of an electrical appliance provided by the present invention. Figure 3 As shown, the control device 100 of the electrical appliance includes: an acquisition unit 110 , a training unit 120 , a first recognition unit 130 , a confirmation unit 140 , a second recognition unit 150 and a control unit 160 .

[0076] The acquisition unit 110 is used to acquire pre-collected first motion data.

[0077] The first motion data may specifically include: at least one of action data, expression data and gesture data.

[0078] The motion data may specifically be multi-angle human motion data. For example, cameras are set at different positions in the room to collect multi-angle human motion data. For example, in order to achieve high-precision full-body motion capture, human motion data can be captured by Kinect and / or Vicon motion capture systems, and the user's full-body motion can be captured from multiple angles by cameras set in different corners of the room. The Kinect system (body tracking system) uses infrared depth sensors and RGB cameras, while Vicon (optical motion capture system) uses high-precision optical cameras and marker technology. Both systems can record the user's walking, waving, shaking, sneezing and other actions in real time to ensure the comprehensiveness and accuracy of the data.

[0079] For facial expression data, facial expression data can be captured through facial recognition cameras equipped with OpenCV and / or Dlib libraries. These cameras can be specifically used to capture the user's facial expressions, such as smile, frown, blush, etc. OpenCV (cross-platform computer vision library) and Dlib (machine learning algorithm library) libraries provide powerful image processing and facial feature point detection functions, which can accurately identify and record the changes in the user's facial expressions.

[0080] For gesture data collection, in order to accurately capture the user's gestures, a depth camera can be used to capture the user's gestures. For example, the Leap Motion depth camera can capture the user's gestures, such as pointing and grabbing, through its high-precision sensors. This depth camera can provide detailed hand movement data and provide high-quality input for subsequent gesture recognition models.

[0081] The training unit 120 is used to perform model training using the first motion data to obtain a recognition model.

[0082] The collected first motion data may include: at least one of action data, expression data and gesture data. Correspondingly, the recognition model includes at least one of an action recognition model, an expression recognition model and a gesture recognition model.

[0083] In a specific implementation, training the recognition model mainly includes the following steps S1 to S2;

[0084] Step S1, preprocessing the first motion data to obtain a training data set.

[0085] The data preprocessing may specifically include: data cleaning, data standardization and data labeling.

[0086] Data cleaning may specifically include: using a filtering algorithm to process the acquired data to remove noise and outliers; the filtering algorithm preferably uses a median filtering algorithm. For example, the original signal: [10, 12, 11, 100, 13, 14, 15], in this signal 100 is obviously an outlier because it is much larger than other data values, then use the median filter, first assume that the window size is 3, then slide the window to calculate the median, the first window [10, 12, 11], first sort from small to large [10, 11, 12], then take the middle value 11, then move the window [12, 11, 100], repeat the above steps and so on, and finally put all the medians into a window [11, 12, 13, 14, 14], this is the filtered signal, it can be seen that the outlier 100 in the original signal is effectively removed, and the filtered signal is smoother and closer to the true value. For facial expression data and gesture data, image processing technology (such as edge detection) is used to improve the quality of the data.

[0087] Preferably, before using the filtering algorithm to process the first motion data, the first motion data is processed by polynomial fitting; in a specific implementation, the least squares method can be used to perform polynomial fitting, and the obtained polynomial is used to process the first motion data, that is, the least squares method is used to calculate the polynomial coefficients. Assume that there are the following time steps and corresponding joint angle data: time step: x (1 2 3 4 5 6 7) joint angle y (1012 11 100 13 14 15), assuming that the order of the polynomial is m = 2m = 2, that is, the quadratic polynomial P (x) = a 0 +a 1 x+a 2 x 2 , use the least squares method to calculate the coefficients a of the polynomial 0 、a 1 and a 2 The goal of the least squares method is to minimize the sum of squared errors. Assuming that a specific coefficient a is obtained 0 ,a 1 ,a 2 For example: 0 =10,a 1 =0.5,a 2 =-0.01, and use the calculated polynomial coefficients to generate the fitting curve P(x): P(x)=10+0.5x-0.01x 2 , substitute the time steps (1 2 3 4 5 6 7) respectively and get the smoothed data [10, 11, 11.5, 12, 12.5, 13, 13.5].

[0088] The purpose of using polynomial fitting is to smooth the original user data and remove outliers. Although there is a median filter algorithm to remove outliers during model training, it is very important for image models to process data outliers to ensure data smoothness. Therefore, polynomial fitting technology is used to process data before median filtering the model. Because, if there are two consecutive outliers, such as (10, 100, 102), the median filter will take 100, and the outlier will not be removed in the end. After fitting, median filtering is performed to make the median filtering process more accurate, and the model training begins after filtering.

[0089] Data standardization is the process of converting the collected data into a uniform format and scale for easy subsequent processing. For example, motion capture data is standardized to a range of 0-1, and expression data is normalized.

[0090] Data annotation is to assign labels to each action, expression and / or gesture. These labels will be used as part of the training data set to guide the model's learning process. For example, if a smiling expression is used as a training set input, the output result is the label "smile".

[0091] Step S2, using the obtained training data set to perform model training to obtain the recognition model.

[0092] The recognition model may specifically be a neural network model. For example, the gesture recognition model may be a deep neural network model DNN, the expression recognition model may be a convolutional neural network model CNN, and the action recognition model may be a long short-term memory network model LSTM or a convolutional neural network model CNN.

[0093] In a specific embodiment, a long short-term memory network (LSTM) is used to train the recognition model. The long short-term memory network model can process sequence data, such as time series data, that is, the collected motion data is motion data with time characteristics. For example, when collecting data, every preset time interval can be used as a time step to collect the motion data corresponding to each time step. For example, taking the action model as an example, it is assumed that there is a set of motion capture data that records a person's walking action. The data includes joint angles and speed information for each time step. The joint angles and speeds of each time step are collected (through Kinect or Vicon motion capture system). During data preprocessing, median filtering is used to remove outliers, and then the data is segmented into multiple time series segments by data segmentation, each segment corresponding to an action. For example, the processed result data is as follows: time step 1: [10, 12, 11]; time step 2: [12, 11, 13]; time step 3: [11, 13, 14]; time step 4: [13, 14, 15]; time step 5: [14, 15, 16]).

[0094] The collected time series data is used as the input of the LSTM network, and the corresponding labels (action, expression and / or gesture labels) are used as the output to obtain the corresponding recognition model. The labels output by the recognition model are compared with the true labels to calculate the cross entropy loss, and the gradient is calculated through the back propagation algorithm to update the parameters of the LSTM layer. Then, iterative training is performed to ensure that the model can accurately recognize actions, labels and / or gestures.

[0095] Specifically, the collected time series data is used as the input of the LSTM network input layer, and then enters the LSTM layer. The LSTM layer contains multiple LSTM units, each of which processes a time step of data, calculates the hidden state and output, and the output layer obtains the labels of action categories, expression categories and / or gesture categories through the classification algorithm. Then the model output is compared with the true label to calculate the cross entropy loss. If the difference is large, the gradient is calculated through the back propagation algorithm, the parameters of the LSTM layer are updated, and then iterative training is performed to ensure that the model can accurately recognize the walking action.

[0096] For example, a set of walking action data is used as the input of the LSTM network input layer, and then enters the LSTM layer, which contains multiple LSTM units, each unit processes a time step of data, for example, time step 1: input [10,12,11], calculate the hidden state and output. Time step 2: input [12,11,13], combined with the hidden state of the previous time step, calculate the new hidden state and output. Time step 3: and so on..., the output layer obtains a label of the action category through the classification algorithm, and then compares the model output with the true label to calculate the cross entropy loss. If the difference is large, the gradient is calculated through the back propagation algorithm, the parameters of the LSTM layer are updated, and then iterative training is performed to ensure that the model can accurately recognize the walking action. For the expression recognition model and the gesture recognition model, the LSTM (long short-term memory network) model can also be used. The input is the preprocessed expression or gesture image, and the output is the expression or gesture label, which is then compared with the true label to calculate the cross entropy loss. Then the parameters are updated and iterative training is repeated to ensure the accuracy of the model.

[0097] The first recognition unit 130 is used to collect the second motion data of the target user, and recognize the second motion data by using the trained recognition model to obtain a recognition result of the second motion data.

[0098] Specifically, the second motion data of the target user is collected in the same collection method as the first motion data. The second motion data is the same as the first motion data, and may specifically include: at least one of motion data, expression data, and gesture data. For example, human motion data is captured by Kinect and / or Vicon motion capture system; and / or, facial expression data is captured by a facial recognition camera equipped with OpenCV and / or Dlib library; and / or, gesture data is captured by a depth camera.

[0099] The second motion data is recognized by using the recognition model obtained through training to obtain a recognition result of the second motion data, i.e., a simulation mapping result. The recognition result is specifically a label obtained by recognizing the second motion data. When the model is trained, the input of the training set is an image with image features (such as expression, action), and the output result is a result label of the image.

[0100] Specifically, the collected second motion data (collected pictures) is used to generate multiple time series segments, each of which is a mapping of a motion segment (action, expression, gesture). That is, the action features in the picture are converted into numbers, such as [10,12,11], which records the image features of three dimensions. Combining multiple such segments during this period can identify the user's action trajectory during this period. The training set has enabled the model to recognize the user's actions in these images through iterative training.

[0101] The confirmation unit 140 is used to display the recognition result of the second motion data to the target user, and the target user confirms whether the recognition result is accurate to update the recognition model.

[0102] In a specific embodiment, the recognition result of the second motion data is displayed and / or the recognition result of the second motion data is broadcasted by voice; and confirmation information from the target user is received to confirm whether the recognition result of the second motion data is accurate. That is, the recognition result is displayed on a display screen and / or the recognition result is converted into voice and broadcasted by the voice module of the device, and confirmation information from the user is received. The user can provide feedback through a preset button (such as a button on the interactive interface) or through voice to confirm the accuracy of the recognition result.

[0103] For example, the user stands in the middle of the room, and the air conditioner's camera begins to capture his whole body movements. He first walks slowly for a few steps, then waves his hand, and finally makes a jumping motion. The air conditioner's display screen immediately displays the labels "walking", "waving", and "jumping", indicating that the system has accurately recognized his movements. Then the user makes several expressions to the air conditioner's facial recognition camera: smile, frown, and surprise. The air conditioner's display screen displays the labels "smile", "frown", and "surprise" in turn, indicating that the system has successfully captured and recognized his facial expressions. The user reaches out and makes several gestures to the air conditioner's camera: point to the air conditioner, grab, and rotate. The air conditioner's display screen displays the labels "point", "grab", and "rotate", indicating that the system has accurately recognized his gestures.

[0104] Optionally, the device 100 further includes a storage unit (not shown).

[0105] In a specific implementation, the storage unit is used to store the second motion data for iterative training of the recognition model if the target user receives confirmation information that the recognition result of the second motion data is inaccurate. That is, if the target user feedbacks that the recognition result is inaccurate, the second motion data is stored for iterative training of the model, updating the model parameters, and improving the accuracy of model recognition.

[0106] In another specific implementation, the storage unit is used to store the second motion data as the motion recognition data of the target user if the confirmation information fed back by the target user indicates that the recognition result of the second motion data is inaccurate. That is, when the user makes the same action, expression and / or gesture again, recognition can be performed directly according to the motion recognition data.

[0107] In an actual scenario, the display screen gives a text description of the user's action, expression or gesture. If it matches the action, expression or gesture made by the user, the user can choose yes. If it does not match, it means that the appliance cannot recognize the action, expression or gesture. This function can be used to test people whose actions are relatively inconvenient (such as the elderly). For example, the actions of the elderly may be inconvenient. If the action made by the elderly cannot be recognized by the appliance, and it is also the elderly's habitual action, such as the elderly want to use this action to tell the air conditioner to increase the temperature, then the image and label of this action (such as increasing the temperature) can be input, and this action can be stored. The next time the elderly do this action, the air conditioner can recognize it. This is an additional method to make up for the incorrect recognition. After accurate recognition, the corresponding appliance can be controlled by action, such as controlling the air conditioner. Shivering means cold, and the air conditioner will increase the temperature. The same is true for gestures and expressions.

[0108] The second recognition unit 150 is used to collect the third motion data of the target user, and recognize the third motion data by using the updated recognition model to obtain a recognition result of the third motion data.

[0109] Specifically, after the recognition model is updated, the updated recognition model can be used to recognize the third motion data of the target user collected in real time, and the recognition result of the third motion data can be obtained, that is, the label of the third motion data of the target user is output. The third motion data is the same as the first motion data and the second motion data, and includes at least one of action data, expression data and gesture data. The third motion data of the target user can be collected in the same way as the first and second motion data.

[0110] The control unit 160 is used to generate a control instruction of the corresponding electrical appliance according to the recognition result of the third motion data, and send the control instruction to the corresponding electrical appliance to control the corresponding electrical appliance to execute the control instruction.

[0111] Specifically, the control instruction is sent to the corresponding electrical appliance (e.g., smart air conditioner) by communicating with the corresponding electrical appliance (e.g., Zigbee or Wi-Fi protocol) through a short-range wireless communication technology (e.g., Zigbee or Wi-Fi protocol). The trained model and curve fitting algorithm are integrated into a central processing unit of an air conditioner (e.g., Raspberry Pi or Arduino), and communicate with the smart air conditioner through the Zigbee or Wi-Fi protocol. This integration method has the advantage of reducing chip costs over directly integrating the trained model and curve fitting algorithm into the air conditioner controller, and can achieve efficient system control and data transmission.

[0112] Based on the user's motion, expression and gesture recognition results, the central processing unit generates corresponding control instructions, which are sent to the electrical appliances through Zigbee or Wi-Fi protocols and executed by the electrical appliances. For example, if the appliance is a smart air conditioner, the motion, expression and gesture recognition results are sent to the air conditioner to adjust parameters such as temperature and wind speed. This intelligent control logic can provide a personalized air conditioning control experience based on the actual needs of the user.

[0113] The present invention also provides a storage medium corresponding to the control method of the electrical appliance, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0114] The present invention also provides an electrical appliance corresponding to the control method of the electrical appliance, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned methods when executing the computer program.

[0115] The present invention also provides an electrical appliance corresponding to the control device of the electrical appliance, comprising any of the control devices described above.

[0116] The present invention also provides a computer program product corresponding to the control method of the electrical appliance, comprising a computer program, and when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0117] Accordingly, the solution provided by the present invention improves the accuracy of recognition through the user confirmation mechanism, and can improve the accuracy of the multimodal interactive system. The recognition accuracy of actions, expressions and gestures is ensured through curve fitting technology, and the recognition error rate is reduced in complex environments or when the user's actions change rapidly.

[0118] The solution provided by the present invention ensures that users can use a consistent interaction method between different devices and systems through a unified action, expression and gesture recognition model, thereby reducing learning costs and improving the consistency of user experience.

[0119] The solution provided by the present invention improves the real-time response capability of the system to multimodal data processing by optimizing data preprocessing and model training processes, ensures the natural fluency of interaction, and enhances the real-time performance of data processing.

[0120] The solution provided by the present invention can be intelligently adjusted according to the user's habits and preferences, providing a more personalized appliance control experience and achieving personalized adaptability.

[0121] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0125] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A method for controlling an electrical appliance, characterized in that: include: Acquire pre-collected first motion data; The first motion data includes: at least one of action data, expression data and gesture data; Using the first motion data to perform model training to obtain a recognition model; the recognition model includes: at least one of an action recognition model, an expression recognition model, and a gesture recognition model; Collecting second motion data of the target user, and using the trained recognition model to recognize the second motion data to obtain a recognition result of the second motion data; showing the recognition result of the second motion data to the target user, and having the target user confirm whether the recognition result is accurate, so as to update the recognition model; Collecting the third motion data of the target user, and identifying the third motion data using the updated recognition model to obtain a recognition result of the third motion data; A control instruction of the corresponding electrical appliance is generated according to the recognition result of the third motion data, and is sent to the corresponding electrical appliance to control the corresponding electrical appliance to execute the control instruction.

2. The method according to claim 1, characterized in that Using the first motion data to perform model training to obtain a recognition model includes: Preprocessing the first motion data to obtain a training data set, wherein the preprocessing includes: at least one of data cleaning, data standardization, and data labeling; The obtained training data set is used to perform model training to obtain the recognition model.

3. The method according to claim 2, characterized in that The data cleaning includes: processing the first motion data using a filtering algorithm; Using the first motion data to perform model training to obtain a recognition model also includes: Before using a filtering algorithm to process the first motion data, the first motion data is processed by polynomial fitting.

4. The method according to any one of claims 1 to 3, characterized in that: Collecting the first motion data, the second motion data and / or the third motion data includes: Capture human motion data via Kinect and / or Vicon motion capture systems; and / or, Capture facial expression data via a facial recognition camera using OpenCV and / or Dlib libraries; and / or, Capture gesture data through a depth camera.

5. The method according to any one of claims 1 to 3, characterized in that: Displaying the recognition result of the second motion data to the target user, and asking the target user to confirm whether the recognition result is accurate, includes: Displaying the recognition result of the second motion data and / or announcing the recognition result of the second motion data by voice; Receive confirmation information fed back by the target user to confirm whether the recognition result of the second motion data is accurate.

6. The method according to any one of claims 1 to 3, characterized in that: Also includes: If the confirmation information fed back by the target user is that the recognition result of the second motion data is inaccurate, saving the second motion data for iterative training of the recognition model; and / or, If the confirmation information fed back by the target user is received that the recognition result of the second motion data is inaccurate, the second motion data is saved as the motion recognition data of the current target user.

7. A control device for an electrical appliance, characterized in that: include: An acquisition unit, used for acquiring pre-collected first motion data; The first motion data includes: at least one of action data, expression data and gesture data; A training unit, configured to perform model training using the first motion data to obtain a recognition model; the recognition model includes: at least one of an action recognition model, an expression recognition model, and a gesture recognition model; A first recognition unit is used to collect second motion data of a target user, and recognize the second motion data using the trained recognition model to obtain a recognition result of the second motion data; a confirmation unit, configured to display a recognition result of the second motion data obtained by recognition to the target user, and have the target user confirm whether the recognition result is accurate, so as to update the recognition model; A second recognition unit is used to collect third motion data of the target user, and recognize the third motion data using the updated recognition model to obtain a recognition result of the third motion data; The control unit is used to generate a control instruction of the corresponding electrical appliance according to the recognition result of the third motion data, and send the control instruction to the corresponding electrical appliance to control the corresponding electrical appliance to execute the control instruction.

8. A storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of any method described in claims 1-6 are implemented.

9. An electrical appliance, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any method described in claims 1 to 6 when executing the program, or comprises the control device described in claim 7.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of any one of the methods of claims 1 to 6 when executed by a processor.

Images