Gesture recognition method, control device, storage medium and household appliance

By setting preset gesture judgment on the opening conditions in the camera system, and judging user intentions using thermal imaging sensors and touch sensors, the problem of privacy leakage in the prior art is solved, and a safer air-to-air operation experience is achieved.

CN118945461APending Publication Date: 2024-11-12QINGDAO HAIER WASHING MASCH CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310522318.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art is difficult to effectively protect user privacy when using cameras to operate in air distances, resulting in an increase in the risk of privacy leakage.

Method used

By setting preset gesture judgment on the turn-on condition, using the thermal imaging sensor and touch sensor to determine the user's intention, activate the camera only when specific conditions are met for gesture recognition, ensuring that the camera is only activated when the user is willing to use the air-distance operation function.

Benefits of technology

It reduces the risk of privacy leakage, improves user's operating experience, and reduces the possibility of privacy leakage while ensuring the practicality of functions, making users more at ease when using the device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945461A_ABST
    Figure CN118945461A_ABST
Patent Text Reader

Abstract

The invention relates to a gesture recognition method, a control device, a storage medium and a household appliance. The method comprises the following steps: judging whether a preset gesture judgment starting condition is met or not; if the preset gesture judgment starting condition is met, obtaining a first image; recognizing a user gesture according to the acquired first image; and correspondingly adjusting the equipment mode based on the user gesture and the gesture corresponding rule. According to the technical scheme, the starting condition is judged through the preset gesture, it is ensured that the camera is started only when the user intends to use the air operation function, and the privacy leakage risk is reduced. The operation experience of the user is improved, and meanwhile user privacy protection is fully considered. And on the basis of guaranteeing the function practicability, the possibility of privacy leakage is reduced, so that the user can use the equipment more securely.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of electrical system control, and specifically provides a gesture recognition method, a control device, a storage medium and a household appliance. Background Art

[0002] With the development of technology, the application of cameras on home appliances is becoming more and more popular, bringing many conveniences to people's lives. For example, the remote operation of the device improves the user experience.

[0003] However, the widespread use of cameras, while providing convenience, has also raised concerns about privacy protection.

[0004] Accordingly, a new gesture recognition solution is needed in the art to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present invention is proposed to provide a solution or at least partially solve the demand for protecting user privacy in the prior art.

[0006] In a first aspect, the present invention provides a gesture recognition method, the method comprising: determining whether a preset gesture judgment start-up condition is met; if the preset gesture judgment start-up condition is met, acquiring a first image; recognizing a user gesture based on the acquired first image; and making corresponding adjustments to the device mode based on the user gesture and the gesture correspondence rule.

[0007] As an alternative or supplement to the above scheme, in a method according to an embodiment of the present invention, the judgment of whether a preset gesture judgment activation condition is met includes: acquiring a thermal imaging image; judging whether a preset gesture judgment activation condition is met based on the thermal imaging image; and the corresponding adjustment of the device mode includes adjusting the display font size.

[0008] As an alternative or supplement to the above scheme, in a method according to an embodiment of the present invention, the determining whether a preset gesture determination start condition is satisfied based on the thermal imaging image includes: determining whether the temperature of an object in a first area based on the thermal imaging image is within a preset temperature range, wherein the first area is a recognition area of ​​a thermal imaging sensor; if the temperature of the object is within the preset temperature range, identifying whether there is a user's hand in the thermal imaging image; if there is a user's hand in the thermal imaging image, the preset gesture determination start condition is satisfied.

[0009] As an alternative or supplement to the above scheme, in a method according to an embodiment of the present invention, it is determined whether the length of time the user touches the touch sensor is greater than a preset first threshold; if it is greater than the preset first threshold, it is determined that the preset gesture judgment start condition is met; if it is less than or equal to the preset first threshold, it is determined that the preset gesture judgment start condition is met.

[0010] As an alternative or supplement to the above scheme, in a method according to an embodiment of the present invention, recognizing a user gesture based on an acquired first image includes: preprocessing the first image to obtain a second image; and obtaining the user's gesture action based on the second image and a trained gesture recognition model.

[0011] As an alternative or supplement to the above scheme, in a method according to an embodiment of the present invention, obtaining the user's gesture action based on the second image and a trained gesture recognition model includes: extracting features from the second image to obtain a second feature vector; and inputting the second feature vector into the trained gesture recognition model to obtain the user's gesture action.

[0012] As an alternative or supplement to the above scheme, in a method according to an embodiment of the present invention, the gesture recognition model includes at least: a fully connected layer, a discard layer and a Softmax layer, and the inputting the second feature vector into the trained gesture recognition model to obtain the user's gesture action includes: inputting the second feature vector into the fully connected layer to obtain a third feature vector; inputting the third feature vector into the Softmax layer to classify the gesture action; and outputting the user's gesture action.

[0013] In a second aspect, a control device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple computer programs, and the computer programs are suitable for being loaded and run by the processor to execute the gesture recognition method described in any one of the technical solutions of the above-mentioned gesture recognition method.

[0014] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of computer programs are stored in the computer-readable storage medium, wherein the computer programs are suitable for being loaded and run by a processor to execute the gesture recognition method described in any one of the technical solutions of the above-mentioned gesture recognition method.

[0015] In a fourth aspect, a household appliance is provided, comprising a control device and a camera, wherein the camera is used to capture a first image; and the control device is used to execute the gesture recognition method described in any one of the technical solutions of the above-mentioned gesture recognition method.

[0016] The above one or more technical solutions of the present invention have at least one or more of the following Beneficial effects:

[0017] In the technical solution of the present invention, by setting a preset gesture to determine the start condition, it is ensured that the camera is started only when the user intends to use the air operation function, thereby reducing the risk of privacy leakage. The user's operating experience is improved, while fully considering the user's privacy protection. On the basis of ensuring the practicality of the function, the possibility of privacy leakage is reduced, making the user feel more at ease when using the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the drawings represent similar components, wherein:

[0019] Figure 1 is a schematic flow chart of main steps of a gesture recognition method according to an embodiment of the present invention;

[0020] Figure 2 is a flowchart of the minor steps of a gesture recognition method according to an embodiment of the present invention;

[0021] Figure 3 is a flowchart of the minor steps of a gesture recognition method according to an embodiment of the present invention;

[0022] Figure 4 is a flowchart of the minor steps of a gesture recognition method according to an embodiment of the present invention;

[0023] Figure 5 is a flowchart of the minor steps of a gesture recognition method according to an embodiment of the present invention;

[0024] Figure 6 1 is a flowchart of the main steps of a gesture recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0026] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as computer programs, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store computer programs, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0027] Embodiment 1:

[0028] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a gesture recognition method according to an embodiment of the present invention. Figure 1 As shown, the gesture recognition method in the embodiment of the present invention mainly includes the following steps S10 to S40.

[0029] Step S10: Determine whether a preset gesture determination start condition is met.

[0030] In this embodiment, the preset gesture determination activation condition is a set of pre-set rules for determining when to activate the gesture recognition function.

[0031] In one embodiment, the system will determine whether the preset conditions are met to determine whether the camera should be turned on for gesture recognition. These preset conditions usually include sensor data and other relevant information to ensure that the user's intention is accurately recognized while also protecting the user's privacy.

[0032] The preset gesture judgment activation conditions may include the object temperature, object duration, object shape, etc. detected by the thermal imaging sensor, and may also involve data from other sensors (such as touch sensors, ultrasonic sensors, etc.). When all these conditions are met, the system will assume that the user intends to perform gesture operations and activate the gesture recognition function. In this embodiment, for the sake of understanding the present technology, a specific gesture judgment activation condition is given by taking a washing machine as an example, which specifically includes steps S101-S102, such as Figure 2 As shown, the details are as follows.

[0033] Step S101: Acquire a thermal imaging image.

[0034] In this embodiment, an image containing thermal radiation information is captured and generated by a thermal imaging sensor.

[0035] In one embodiment, the thermal imaging sensor can detect infrared radiation from the surface of an object, thereby obtaining the temperature distribution of the surface of the object. This temperature distribution information is represented in different colors in the image for analysis and processing. In this embodiment, when the device is started or enters the standby state, the thermal imaging sensor is automatically activated.

[0036] The thermal imaging sensor captures infrared radiation emitted by the surface of surrounding objects through its internal detector. The intensity of infrared radiation is closely related to the temperature of the surface of the object, so it can be used to reflect the temperature distribution of the surface of the object. Based on the captured infrared radiation data, the thermal imaging sensor generates a thermal imaging image. Each pixel in the image represents a temperature value, and different colors represent different temperature ranges. Usually, areas with higher temperatures are displayed in red or orange, and areas with lower temperatures are displayed in blue or green. The thermal imaging image can be further processed and analyzed by the device's processor to extract temperature information of objects in a specific area. This information can be used to determine whether the preset gesture judgment activation conditions are met, thereby achieving corresponding adjustments to the device mode.

[0037] Step S102: judging whether a preset gesture judgment start condition is met based on the thermal imaging image.

[0038] In this embodiment, steps S1021-S1023 are used to determine whether the preset gesture determination start condition is met, such as Figure 3 As shown, the details are as follows:

[0039] Step S1021: determining whether the temperature of an object in a first area is within a preset temperature range based on the thermal imaging image recognition.

[0040] In this embodiment, the first area is the recognition area of ​​the thermal imaging sensor. In this embodiment, according to the actual application scenario and the installation location of the device, a specific area is selected as the recognition area of ​​the thermal imaging sensor.

[0041] In one embodiment, the first area should include as much of the spatial range as possible where the gesture operation occurs to ensure that the user's hand movements can be effectively captured. In this embodiment, the temperature value of each pixel in the first area is extracted from the thermal imaging image. These temperature values ​​can be obtained by analyzing the color of the pixel. The extracted temperature information is analyzed to determine the temperature distribution characteristics of the object in the first area. For example, statistical indicators such as the mean value and variance of each temperature value in the area can be calculated to evaluate the distribution of temperature.

[0042] According to the analysis results, it is determined whether the temperature of the object in the first area is within a preset temperature range. The preset temperature range is usually set based on the normal body temperature range of the human body, such as 36.5-37.5 degrees Celsius. If the temperature of the object in the first area is within the preset temperature range, it means that someone may be operating the device, and the subsequent hand recognition steps can be continued.

[0043] Step S1022: If the temperature of the object is within a preset temperature range, identify whether there is a user's hand in the thermal imaging image.

[0044] In one implementation, first, features of the hand are extracted from the image. These features may include the shape, size, temperature distribution, etc. of the hand. By analyzing these features, the hand can be effectively distinguished from other objects.

[0045] Then, according to the actual application scenario and user needs, a recognition threshold is set to determine whether the user's hand exists in the image. This threshold can be a fixed value or can be dynamically adjusted according to the actual situation. For example, a threshold for the temperature distribution of the hand can be set. When the temperature distribution in the image meets this threshold, it is considered that the user's hand exists.

[0046] Finally, the extracted hand features are compared with the set recognition threshold to determine whether the user's hand exists in the thermal imaging image. If the threshold condition is met, it means that the user's hand exists in the image, and the subsequent gesture recognition steps can be continued.

[0047] This step ensures that the gesture recognition function is activated only when the user intentionally performs a gesture operation by identifying whether the user's hand is present in the thermal imaging image, thereby improving recognition accuracy and protecting user privacy.

[0048] Step S1023: If there is a user's hand in the thermal imaging image, the preset gesture judgment start condition is met.

[0049] In this embodiment, after confirming that the user's hand exists in the thermal imaging image, the system will believe that the user may have the intention to perform a gesture operation, thereby satisfying the preset gesture judgment activation condition.

[0050] In one implementation, the steps are described here using the application scenario of a washing machine as an example. In the application scenario of a washing machine, a user may want to control the start, stop, mode switching and other functions of the washing machine through gestures. Since washing machines are usually placed in bathrooms or other areas where people may enter and exit, directly using a camera for gesture recognition may capture irrelevant gestures, resulting in misrecognition. At the same time, continuous monitoring may infringe on user privacy.

[0051] In this case, the thermal imaging sensor can first determine whether the object temperature is within the preset temperature range (such as the normal body temperature range of the human body), which can effectively screen out targets that may be the user's hand. When the presence of the hand is confirmed, the system will activate the camera for gesture recognition. In this way, the false recognition rate can be reduced and the accuracy of gesture recognition can be improved.

[0052] In addition, this step also has the significance of protecting user privacy. By pre-screening the target with the thermal imaging sensor, it can be ensured that the camera is only turned on when the user intentionally performs a gesture operation, rather than continuously monitoring. In this way, the gesture recognition function can be guaranteed while protecting the user's privacy.

[0053] In summary, in this embodiment, by first determining the temperature of the object and then identifying the hand in the thermal imaging image, the gesture recognition function is activated only when the user intentionally performs a gesture operation, thereby improving recognition accuracy and protecting user privacy.

[0054] Step S20: If the preset gesture judgment start condition is met, a first image is acquired.

[0055] In this embodiment, when the system confirms that the preset gesture judgment start condition is met, the camera will be activated to start capturing the real-time image of the user, wherein the first image is the image of the user captured by the camera.

[0056] In one embodiment, the first image is an image of the user captured by the camera after confirming that the preset gesture judgment start condition is met. It should be noted that the first image includes the user's hand and other objects or backgrounds that may appear in the camera's field of view.

[0057] Naturally, if the preset gesture judgment start condition is not met, the first image is not acquired. That is, if the preset gesture judgment start condition is not met, the camera remains closed, thereby protecting the user's privacy.

[0058] Step S30: recognizing a user gesture according to the acquired first image.

[0059] In this embodiment, user gestures refer to a way in which a user interacts and communicates with a computer, smart device or other person through body movements (especially finger and hand movements).

[0060] In one embodiment, gesture recognition is performed on the captured first image through an image recognition algorithm (such as a convolutional neural network, etc.) to determine the user's gesture action. In this embodiment, the user's gesture is recognized through steps S301-S302, such as Figure 4 shown.

[0061] Step S301: preprocessing the first image to obtain a second image.

[0062] In this embodiment, preprocessing refers to performing operations such as cropping, scaling, and rotating the collected image data to increase the diversity and robustness of the data. In addition, image enhancement, such as contrast adjustment and noise removal, can also be performed to improve the generalization ability of the model.

[0063] In one embodiment, the first image is preprocessed. In this embodiment, the image is preprocessed by image cropping, scaling and normalization.

[0064] Image cropping: Crop the acquired user image to keep only the hand area. Use some image processing techniques to locate the hand in the image, such as background subtraction, edge detection, or object detection. Once the hand area is determined, crop this part of the image to remove background information that is not related to the gesture. Image cropping reduces computational complexity and improves recognition speed and accuracy.

[0065] Image scaling: Scale the cropped hand image to the size required by the trained model. In this embodiment, since the trained model usually has certain requirements on the size of the input image, the cropped hand image needs to be scaled to an appropriate size. The scaling operation can be achieved through an interpolation algorithm (such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.). For example, if the input size is 224x224 pixels when training the model, the cropped hand image needs to be scaled to this size to ensure that the image can be correctly input into the model.

[0066] Normalization: Normalize the scaled image to map the pixel values ​​to a range of 0-1. In this embodiment, normalization maps the pixel values ​​of the image to a smaller range (such as 0-1 or -1 to 1), which helps to improve the convergence speed and performance of the model. Taking the pixel value range of 0-255 as an example, normalization is to divide the pixel value of the image by 255 so that it falls between 0-1.

[0067] After the above preprocessing steps, a user image adapted to the gesture recognition model is obtained, namely the second image. Next, the preprocessed image is input into the model for gesture recognition. According to the recognition result, the device mode of the washing machine is adjusted.

[0068] Step S302: obtaining the user's gesture action based on the second image and the trained gesture recognition model.

[0069] In this embodiment, the trained gesture recognition model can recognize features in the second image, thereby obtaining the user's gesture action.

[0070] In one implementation, the user's gesture action is obtained through steps S3021-S3022, such as Figure 5 As shown, the details are as follows:

[0071] Step S3021: extract features from the second image to obtain a second feature vector.

[0072] In this embodiment, feature extraction is the process of extracting key information that can describe the characteristics and attributes of the data from the original data. These features are usually easier to process and understand, and can greatly reduce the computational complexity.

[0073] In one embodiment, the feature extraction of the second image is mainly performed by a convolutional neural network. Feature extraction is the process of extracting key information that can describe the characteristics and attributes of the data from the original data. These key information are subjected to a sliding window operation on the image through multiple convolution kernels to extract local features. For each convolution kernel, the inner product between it and the local area of ​​the image is calculated. The calculation result is mapped to a new two-dimensional feature map. Usually, there are multiple convolution kernels in the network, so multiple feature maps are obtained.

[0074] The output feature map of the convolutional neural network is processed by applying a nonlinear activation function to obtain an activated feature map. The output feature map of the convolutional layer is obtained by sliding multiple convolution kernels on the input image and performing convolution operations. The convolution operation is linear. If multiple convolutional layers are simply stacked without using nonlinear activation functions, these convolutional layers can only capture linear features and cannot express complex nonlinear relationships. Introducing nonlinear activation functions can increase the expressive power of the model, enabling it to capture higher-level and more complex features.

[0075] The ReLU activation function is introduced in this embodiment. ReLU (Rectified Linear Unit) is a commonly used activation function, and its mathematical expression is: f(x) = max(0, x). In other words, when the input value is negative, the output of the ReLU function is 0; when the input value is positive, the output of the ReLU function is equal to the input value. The feature map output by the convolution layer is used as the input of the ReLU function. The ReLU function is applied pixel by pixel, and negative values ​​are set to 0, while positive values ​​are retained. In this way, nonlinear features are introduced into the convolution map, and an activated feature map is obtained. The activated feature map contains the nonlinear information in the original feature map, which helps to capture the complex features in the image. Afterwards, these feature maps processed by the activation function will be used as the input of the next convolution layer or pooling layer.

[0076] The activation feature map is input to the pooling layer to obtain the second feature vector. The pooling layer is mainly used to downsample the activation feature map output by the convolution layer to reduce the feature dimension. This helps to reduce computational complexity, memory requirements, and the number of parameters, thereby increasing the speed of model training and prediction. At the same time, the pooling layer also has a certain degree of translation invariance, which can improve the robustness of the model to small changes in the input image.

[0077] In this embodiment, a pooling method and a window size are selected. According to the task requirements and the model structure, the maximum pooling or average pooling and the appropriate window size are selected. Then the pooling operation is applied to the activated feature map. The selected pooling method is applied to each channel of the activated feature map, sliding on the feature map according to the set window size, and performing a pooling operation in each window. In this way, a downsampled feature map is obtained. Then the output of the pooling layer is flattened to obtain a one-dimensional feature vector, i.e., the second feature vector.

[0078] Step S3022: input the second feature vector into the trained gesture recognition model to obtain the user's gesture action.

[0079] In this embodiment, the second feature vector is used as the input of a trained gesture recognition model. This model usually includes several fully connected layers for further combining and processing the features extracted by the previous convolutional neural network to recognize the user's gesture actions.

[0080] In one embodiment, the trained gesture recognition model includes at least: a fully connected layer, a dropout layer, and a Softmax layer. In this embodiment, the second feature vector is input into the trained gesture recognition model through the following steps S3023-S3024 to obtain the user's gesture action:

[0081] Step S3023: Input the second eigenvector to the fully connected layer to obtain a third eigenvector.

[0082] In this embodiment, the function of the fully connected layer is to perform nonlinear transformation and dimension reduction on the second eigenvector so that it can better represent the gesture information of the input image. The fully connected layer multiplies the second eigenvector by the weight matrix, then adds the bias vector, and processes the result through a nonlinear activation function to obtain a third eigenvector.

[0083] In this embodiment, in order to increase the expressive power and learning ability of the model, multiple fully connected layers are usually used. These fully connected layers are usually arranged in a certain order, and each layer takes the output of the previous layer as input. Usually, the output dimension of the fully connected layer will decrease layer by layer, thereby achieving the purpose of dimensionality reduction. This helps to reduce the computational complexity of the model and improve the training and prediction speed.

[0084] Through these fully connected layers, the model can further process the features extracted by the convolutional neural network so that it can better represent the gesture information of the input image. In practical applications, the number and structure of fully connected layers can be adjusted according to specific tasks and data sets to achieve optimal performance.

[0085] Step S3024: Input the third feature vector to the Softmax layer to classify the gesture action.

[0086] In this embodiment, in order to obtain the final gesture recognition result, the output of the fully connected layer is classified. Here, the Softmax layer is used for classification, which can convert the output of the fully connected layer into a probability distribution, thereby obtaining the predicted probability of each gesture action.

[0087] In this embodiment, the basic principle of the Softmax layer is to perform an exponential operation on the output of the fully connected layer (the third eigenvector) and then normalize it so that the sum of the predicted probabilities of all categories is 1. The specific operation is as follows:

[0088] Perform an exponential operation on each element of the third eigenvector: Take the exponent of each element of the fully connected layer output to obtain a new vector. The purpose of this is to map the original output value to the positive number domain for normalization.

[0089] Normalization: Divide each element of the new vector after the exponential operation by the sum of all elements. In this way, the result is a probability distribution in which each element represents the predicted probability of the corresponding gesture action.

[0090] The output dimension of the Softmax layer is equal to the number of gesture categories. By calculating the predicted probability of each category, the gesture with the highest probability can be found as the recognition result. Specifically, by comparing the elements in the probability distribution output by the Softmax layer, the category corresponding to the element with the maximum value can be found and used as the result of gesture recognition.

[0091] The use of the Softmax layer enables the gesture recognition model to output the predicted probability of each gesture action, which helps to provide more information in practical applications. For example, a threshold can be set according to the predicted probability to filter out low-probability erroneous recognition results, or to fuse multiple models.

[0092] Step S3025: Output the user's gesture action.

[0093] In this embodiment, by comparing the elements in the output probability distribution, the category corresponding to the element with the maximum value is found. This category represents the recognized gesture action.

[0094] In the process of training the gesture recognition model, a dropout layer is also used for regularization, namely step S3026. The details are as follows:

[0095] Step S3026: Apply a dropout layer for regularization

[0096] In this embodiment, in order to prevent overfitting, a dropout layer can be added between the fully connected layers. The function of the dropout layer is to randomly discard the output of some neurons with a certain probability during the training process, thereby reducing the model's dependence on a single data sample and improving the model's generalization ability.

[0097] In this embodiment, the discard rate can be set according to actual needs. Usually, the discard rate is set at about 0.5.

[0098] Step S40: adjusting the device mode accordingly based on the user gesture and the gesture correspondence rule.

[0099] In this embodiment, the gesture corresponding rules are pre-set rules, which can be rules formulated by the manufacturer according to product features and user needs, or rules customized by the user according to his or her own preferences and usage scenarios. Such a design not only ensures that the device can meet the expected operation mode of most users, but also allows users to make personalized settings according to their own habits and needs. The corresponding adjustment of the device mode includes the adjustment of the display font size.

[0100] In one embodiment, the recognized gesture action is mapped to the corresponding device mode according to a preset rule of correspondence between gesture actions and device modes. For example, a finger ratio of 1 corresponds to a small font, a finger ratio of 2 corresponds to a medium font, and a finger ratio of 3 corresponds to a large font. According to the user's gesture action result output in step SS30, the font size or other related settings of the device are automatically adjusted to meet the user's needs.

[0101] Preferably, the prediction results of multiple models can also be fused.

[0102] For example, suppose there are three gesture recognition models A, B, and C, each of which outputs a prediction probability for a certain gesture action. Suppose the user makes a finger-to-1 gesture in front of the washing machine panel, model A recognizes that the probability of finger-to-1 is 0.8, model B recognizes that the probability of finger-to-1 is 0.9, and model C recognizes that the probability of finger-to-1 is 0.7. In order to improve the recognition accuracy, the prediction results of the three models are fused.

[0103] There are many fusion methods, such as weighted average, voting, etc. Taking weighted average as an example, assume that each model is assigned a weight of 0.4, 0.3, and 0.3 respectively. The prediction probability after fusion is: (0.8*0.4)+(0.9*0.3)+(0.7*0.3)=0.8. This fused prediction probability can provide a comprehensive evaluation result, thereby improving the reliability of recognition.

[0104] By fusion of multiple models, the advantages of each model can be effectively utilized and the impact of the weaknesses of a single model can be reduced, thereby improving the accuracy and robustness of gesture recognition in practical applications. This helps to achieve more reliable device mode adjustment and provide users with a better interactive experience.

[0105] This can improve the accuracy and robustness of gesture recognition, thereby improving the reliability of device mode adjustments.

[0106] Through the above steps, the device mode is adjusted accordingly based on the user gesture and the gesture correspondence rule, achieving the goal of conveniently and accurately meeting user needs. Combined with the output prediction probability of the Softmax layer, the reliability and stability of the device mode adjustment can be further improved.

[0107] In this embodiment, the thermal imaging image is not selected as the basis for gesture recognition mainly because of the recognition accuracy. The thermal imaging image has relatively less information and has a low accuracy rate for gesture recognition.

[0108] Embodiment 2:

[0109] Most of the technical contents in this embodiment are the same as those in Embodiment 1, except that another gesture judgment enabling condition is proposed in this embodiment. In this embodiment, the user's operation on the touch sensor is used as the gesture judgment enabling condition, such as Figure 6 As shown, the details are as follows.

[0110] Step S103: determining whether the time duration for which the user touches the touch sensor is greater than a preset first threshold.

[0111] In this embodiment, the touch sensor is a device that converts a user's touch action into an electronic signal, and can identify the user's operation on the touch surface.

[0112] In one embodiment, the touch sensor can be a capacitive, resistive or other type of touch sensor, and the specific selection can be determined based on cost and performance requirements. In this embodiment, the system detects the length of time the user touches the sensor and compares it with a preset first threshold. The preset first threshold is to ensure that the user's touch behavior is a conscious operation, not an accidental touch. For example, the first threshold can be set to 2 seconds, which means that only when the user touches the sensor for more than 2 seconds will it be considered that the preset gesture judgment start condition is met.

[0113] Step S104: If it is greater than a preset first threshold, it is determined that a preset gesture judgment start condition is met.

[0114] In one embodiment, the system determines whether the gesture judgment activation condition is met based on the result of step S102. If the user touches the sensor for a time period greater than a preset first threshold, the system will assume that the user consciously wants to activate the gesture recognition function, thereby meeting the preset gesture judgment activation condition.

[0115] Naturally, if it is less than or equal to the preset first threshold, it is determined that the preset gesture judgment start condition is met.

[0116] Similarly, the steps in this embodiment can also protect privacy and prevent the camera from shooting when the gesture recognition function is not needed.

[0117] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0118] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above method embodiments when executed by the processor. Among them, the computer program includes a computer program, and the computer program can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry a computer program. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electric carrier signals and telecommunication signals.

[0119] Furthermore, the present invention also provides a control device. In an embodiment of a control device according to the present invention, the control device includes a processor and a storage device, the storage device can be configured to store a program for executing the gesture recognition method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the gesture recognition method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The control device can be a control device device formed by various electronic devices.

[0120] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the gesture recognition method of the above method embodiment, and the program can be loaded and run by the processor to implement the above gesture recognition method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0121] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0122] Furthermore, the present invention also provides a household appliance, comprising a control device and a camera, wherein the camera is used to capture a first image; the control device is used to execute the gesture recognition method described in any one of the technical solutions of the above-mentioned gesture recognition method.

[0123] Those skilled in the art will appreciate that the modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present invention.

[0124] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A gesture recognition method, characterized in that: include: Determine whether the preset gesture judgment start condition is met; If the preset gesture judgment start condition is met, acquiring the first image; recognizing a user gesture based on the acquired first image; Based on the user gesture and the gesture correspondence rule, the device mode is adjusted accordingly.

2. The gesture recognition method according to claim 1, characterized in that: The determining whether a preset gesture determination enabling condition is met includes: Acquire thermal imaging images; Determining whether a preset gesture determination start condition is met based on the thermal imaging image; The corresponding adjustment of the device mode includes adjusting the display font size.

3. The gesture recognition method according to claim 2, characterized in that: The determining whether a preset gesture determination start condition is met based on the thermal imaging image includes: Based on the thermal imaging image, the temperature of the object in the first area is identified, and it is determined that the temperature of the object is within a preset temperature range, wherein the first area is an identification area of ​​the thermal imaging sensor; If the temperature of the object is within a preset temperature range, identifying whether there is a user's hand in the thermal imaging image; If the user's hand is in the thermal imaging image, the preset gesture judgment start condition is met.

4. The gesture recognition method according to claim 1, characterized in that: The determining whether a preset gesture determination enabling condition is met includes: Determine whether the time duration of the user touching the touch sensor is greater than a preset first threshold; If it is greater than a preset first threshold, it is determined that the preset gesture judgment start condition is met; If it is less than or equal to the preset first threshold, it is determined that the preset gesture judgment start condition is met.

5. The gesture recognition method according to any one of claims 1 to 4, characterized in that: The step of recognizing a user gesture according to the acquired first image includes: Preprocessing the first image to obtain a second image; Based on the second image and the trained gesture recognition model, the user's gesture action is obtained.

6. The gesture recognition method according to claim 5, characterized in that: The obtaining of the user's gesture action based on the second image and the trained gesture recognition model includes: Extract features from the second image to obtain a second feature vector; The second feature vector is input into the trained gesture recognition model to obtain the user's gesture action.

7. The gesture recognition method according to claim 6, characterized in that: The gesture recognition model at least includes: a fully connected layer, a dropout layer, and a Softmax layer. Inputting the second feature vector into the trained gesture recognition model to obtain the user's gesture action includes: Input the second eigenvector into the fully connected layer to obtain the third eigenvector; The third feature vector is input into the Softmax layer to classify the gesture action; Output the user's gesture actions.

8. A control device, comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of computer programs, characterized in that: The computer program is suitable for being loaded and run by the processor to execute the gesture recognition method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a plurality of computer programs, characterized in that: The computer program is suitable for being loaded and run by a processor to execute the gesture recognition method according to any one of claims 1 to 7.

10. A household appliance, comprising a control device and a camera, characterized in that: The camera is used to capture a first image; The control device is used to execute the gesture recognition method according to any one of claims 1 to 7.

Citation Information

Cited By

  • A door opening control method and system based on human perception and active interaction, and a washing machine

    CN122728069A