Air conditioner interaction method, device and equipment based on gestures and storage medium

Abstract

The invention relates to the technical field of gesture recognition, in particular to an air conditioner interaction method, device and equipment based on gestures and a storage medium, according to the method, through the double-sample-set design of a normal sample set and an abnormal sample set, a gesture recognition model has the recognition capacity for abnormal gestures, the strict requirement of a traditional scheme for action standardization is broken through, and the recognition efficiency is improved. The adaptive blind area of the handicapped user is filled; a three-dimensional convolutional network is combined with an attention mechanism to construct a to-be-trained model, the three-dimensional convolutional network can effectively extract space and time dynamic features of gestures, the attention mechanism can focus on key areas of the gestures, the influence of non-key interference features is reduced, and even if a handicapped user cannot complete standard gestures, the user experience is improved. The gesture recognition model can still realize accurate recognition, and the problem of high requirements on joint flexibility and action coordination in the prior art is solved; through lightweight processing of the gesture recognition model, it is ensured that the lightweight model can efficiently run on the air conditioner terminal equipment, and the real-time gesture interaction requirement is met.

Description

Technical Field

[0001] This invention relates to the field of gesture recognition technology, and in particular to a gesture-based air conditioning interaction method, apparatus, device, and storage medium. Background Technology

[0002] With the development of smart home technology, gesture recognition-based smart control has gradually become an important direction for improving user interaction experience. Among them, smart air conditioners (air conditioners) are frequently used home appliances, and their gesture control function has seen a surge in demand in recent years in scenarios such as health and epidemic prevention and elderly care due to its ability to achieve non-contact operation and shorten the human-computer interaction path.

[0003] Current mainstream air conditioner gesture control solutions generally adopt a standard ergonomic model design framework. The technical process typically includes: acquiring dynamic image sequences through cameras or infrared sensors, extracting the coordinates of 21 key hand points using open-source frameworks such as MediaPipe, classifying gestures using convolutional neural networks or Transformer models, and finally establishing fixed mapping relationships such as a single fist corresponding to power on / off, vertical index finger sliding to adjust temperature, and palm rotation corresponding to fan speed switching, which can meet the basic operation needs of ordinary users.

[0004] However, this technology system has high requirements for the standardization of movement range and joint flexibility, meaning that there are significant blind spots in the adaptation of the disabled user group. Disabled users are often unable to complete the preset standard gestures due to limited joint movement (such as finger curling caused by rheumatoid arthritis), insufficient muscle strength (such as patients with spinal cord injury), or lack of motor coordination (such as patients with cerebral palsy). Existing solutions lack adaptive adjustment mechanisms for special limb characteristics. For example, they do not consider alternative solutions for single-handed operation, simplify the complexity of gestures, or provide error-tolerant compensation for movements, so disabled users still need to rely on remote control or voice control, failing to achieve the accessibility design goal of technology for all.

[0005] It is evident that existing technologies still need improvement and enhancement. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a gesture-based air conditioning interaction method, which solves the adaptation defects of the prior art for disabled users and significantly improves the inclusiveness and universality of gesture interaction.

[0007] The first aspect of this invention provides a gesture-based air conditioner interaction method, comprising: constructing a training sample set and preprocessing it to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set; labeling the preprocessed sample set to obtain a labeled sample set, and constructing a label mapping table based on the labeled sample set; constructing a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the model to be trained based on the labeled sample set to obtain a gesture recognition model; performing lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploying it; acquiring real-time gesture images, preprocessing the real-time gesture images, and labeling the preprocessed real-time gesture images based on the label mapping table to obtain labeled real-time images; inputting the labeled real-time images into the lightweight recognition model to obtain gesture recognition results, and adjusting the working state of the air conditioner according to the gesture recognition results.

[0008] Optionally, in a first implementation of the first aspect of the present invention, the step of constructing a training sample set and performing preprocessing to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set, includes: constructing a training sample set, wherein the training sample set includes a normal sample set composed of multiple normal gesture images and an abnormal sample set composed of multiple abnormal gesture images; performing grayscale conversion processing on the training sample set using a weighted grayscale algorithm, and performing binarization processing on the grayscale converted training sample set based on the image grayscale histogram using an adaptive threshold segmentation algorithm to obtain a binarized sample set; performing denoising processing on the binarized sample set using a Gaussian filtering method, and performing restoration processing on the denoised binarized samples using a morphological restoration algorithm to obtain a restored sample set; and performing feature extraction processing on the restored sample set using a three-dimensional convolutional neural network to obtain a preprocessed sample set including feature vectors.

[0009] Optionally, in a second implementation of the first aspect of the present invention, the step of labeling the preprocessed sample set to obtain a labeled sample set, and constructing a label mapping table based on the labeled sample set, includes: labeling the preprocessed sample set to obtain a labeled sample set, wherein the labeled tags include gesture category, abnormality type, and / or abnormal location; and constructing a label mapping table by combining the feature vector of the preprocessed sample set and the tags of the labeled sample set, wherein the label mapping table is stored in a key-value pair structure, wherein the key is the hash value of the feature vector, and the value is a set of tags including gesture category tags, abnormality type tags, and / or abnormal location tags.

[0010] Optionally, in a third implementation of the first aspect of the present invention, the step of constructing a training model using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the training model based on the labeled sample set to obtain a gesture recognition model, includes: constructing a training model, the training model comprising an input layer, a three-dimensional convolutional layer, a CBAM attention layer, an anomaly feature enhancement layer, and an output layer connected in sequence, the CBAM attention layer comprising a channel attention sublayer and a spatial attention sublayer, the anomaly feature enhancement layer employing a residual connection structure to enhance the gradient propagation of anomaly gesture features; dividing the labeled sample set according to a preset division ratio to obtain a training set and a test set; and iteratively training the training model based on the training set and the test set using an Adam optimizer and a cosine annealing algorithm to obtain a gesture recognition model.

[0011] Optionally, in the fourth implementation of the first aspect of the present invention, the step of lightweighting the gesture recognition model to obtain a lightweight recognition model and deploying it includes: pruning the gesture recognition model using an L1 regularized channel pruning algorithm to obtain a compressed recognition model; quantizing the compressed recognition model by using an asymmetric quantization method to quantize 32-bit floating-point parameters into 8-bit integer parameters to obtain a quantized recognition model; distilling the quantized recognition model using inter-layer parameters based on knowledge distillation to obtain a lightweight recognition model; and converting the lightweight recognition model into the TFLite Micro format supported by the air conditioner main control unit and deploying it.

[0012] Optionally, in a fifth implementation of the first aspect of the present invention, the step of acquiring a real-time gesture image, preprocessing the real-time gesture image, and annotating the preprocessed real-time gesture image based on the annotation mapping table to obtain an annotated real-time image includes: acquiring a real-time gesture image, sequentially performing grayscale conversion, binarization, denoising, repair, and feature extraction processing on the real-time gesture image to obtain a preprocessed gesture image including a real-time feature vector; and annotating the preprocessed real-time gesture image according to the annotation mapping table and the real-time feature vector to obtain an annotated real-time image.

[0013] Optionally, in a sixth implementation of the first aspect of the present invention, the step of inputting the labeled real-time image into the lightweight recognition model to obtain a gesture recognition result, and adjusting the working state of the air conditioner according to the gesture recognition result, includes: inputting the labeled real-time image into the lightweight recognition model to obtain a gesture recognition result, wherein the gesture recognition result includes a gesture category, a degree of fit, and / or anomaly type; obtaining a preset similarity threshold, and comparing the degree of fit with the preset similarity threshold; if the degree of fit is greater than or equal to the preset similarity threshold, then adjusting the working state of the air conditioner based on the gesture recognition result; if the degree of fit is less than the preset similarity threshold, then generating a gesture adjustment command.

[0014] A second aspect of the present invention provides a gesture-based air conditioner interaction device, comprising: a construction module for constructing and preprocessing a training sample set to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set; a labeling module for labeling the preprocessed sample set to obtain a labeled sample set, and constructing a labeling mapping table based on the labeled sample set; a training module for constructing a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the model to be trained based on the labeled sample set to obtain a gesture recognition model; a lightweighting module for lightweighting the gesture recognition model to obtain a lightweight recognition model and deploying it; a processing module for acquiring real-time gesture images, preprocessing the real-time gesture images, and labeling the preprocessed real-time gesture images based on the labeling mapping table to obtain labeled real-time images; and a recognition module for inputting the labeled real-time images into the lightweight recognition model to obtain gesture recognition results, and adjusting the working state of the air conditioner according to the gesture recognition results.

[0015] A third aspect of the present invention provides a gesture-based air conditioning interaction device, the gesture-based air conditioning interaction device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the gesture-based air conditioning interaction device to perform various steps of the gesture-based air conditioning interaction method described above.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the gesture-based air conditioning interaction method described in any of the preceding claims.

[0017] In the technical solution of this invention, the dual-sample set design of normal and abnormal sample sets enables the gesture recognition model to recognize abnormal gestures, breaking through the stringent requirements of traditional solutions for action standardization and filling the adaptation blind spot for disabled users. A three-dimensional convolutional network combined with an attention mechanism is used to construct the model to be trained. The three-dimensional convolutional network can effectively extract the spatial and temporal dynamic features of gestures, while the attention mechanism can focus on key areas of the gesture and reduce the influence of non-critical interference features. Even if disabled users cannot complete standard gestures, the gesture recognition model can still achieve accurate recognition, solving the problem of high requirements for joint flexibility and action coordination in existing technologies. By lightweighting the gesture recognition model, it is ensured that the lightweight model can run efficiently on air conditioning terminal devices, meeting the needs of real-time gesture interaction. Attached Figure Description

[0018] Figure 1 A logic flowchart of a gesture-based air conditioning interaction method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a gesture-based air conditioning interaction device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a gesture-based air conditioning interaction device provided in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a gesture-based air conditioning interaction method, apparatus, device, and storage medium. In this invention, the terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the gesture-based air conditioning interaction method in this invention includes: 101. Construct a training sample set and preprocess it to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set; In this embodiment, the normal sample set corresponds to standard gestures that ordinary users can complete, such as a complete fist or five fingers spread. The abnormal sample set corresponds to special forms of gestures that disabled users cannot complete due to missing fingers, skeletal deformities, etc., such as a fist with a missing finger or a deformed index finger. The total size of the training sample set is ≥18,000 frames, of which the abnormal sample set covers three core anomalies: missing fingers (1 / 3-1 / 2 of the length missing), missing fingers (1-2 fingers completely missing), and skeletal deformities (joint extension angle ≤120°). The proportion of real disabled user samples in the abnormal samples is ≥75%, and the proportion of synthetic samples is ≤25%, ensuring the authenticity and diversity of the data.

[0021] 102. The preprocessed sample set is labeled to obtain a labeled sample set, and a label mapping table is constructed based on the labeled sample set; In this embodiment, a combination of manual and automated annotation is used to add identity tags to the preprocessed feature vectors and establish a fast mapping relationship between feature vectors and tags. This allows the gesture recognition model to distinguish between normal / abnormal gestures and different abnormal types, while improving the tag matching efficiency during real-time recognition. The annotation mapping table enables quick lookup of feature vectors and corresponding tags, avoiding repeated annotation during real-time recognition, improving interactive response speed, and meeting the immediacy requirements of air conditioning control.

[0022] 103. A three-dimensional convolutional network combined with an attention mechanism is used to construct a model to be trained, and the model to be trained is iteratively trained based on the labeled sample set to obtain a gesture recognition model; In this embodiment, to meet the requirements of gesture dynamism and saliency of abnormal features, a dedicated network structure is designed. Through iterative training on a labeled sample set, the gesture recognition model is equipped with the ability to recognize normal gestures and accurately match abnormal gestures. Compared with existing solutions that use 2D convolutional networks to extract only static features and lack an attention mechanism, this embodiment uses 3D convolution to capture dynamic gestures and combines a CBAM attention layer to focus on abnormal features, thereby solving the problem of poor performance of existing models in recognizing dynamic abnormal gestures and improving the accuracy of gesture recognition for disabled users.

[0023] 104. Perform lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploy it; In this embodiment, to address the hardware limitations of air conditioner main control chips, such as ARM Cortex-M7, a lightweight recognition model is obtained by compressing the size of the gesture recognition model and reducing computing power consumption. This lightweight model can run stably in the air conditioner embedded system while retaining the core recognition capabilities. This solves the problem that existing models are large in size and have high computing power, making them unsuitable for deployment in air conditioner embedded systems. Furthermore, the lightweight model does not affect the core functions of the air conditioner, such as cooling / heating, during operation.

[0024] 105. Acquire real-time gesture images, preprocess the real-time gesture images, and annotate the preprocessed real-time gesture images based on the annotation mapping table to obtain an annotated real-time image. In this embodiment, the user's real-time gestures are captured by the air conditioner infrared sensor, and the preprocessing logic of the training sample set is reproduced to ensure the consistency between the real-time feature vector and the feature vector during training. Then, the label mapping table is combined to quickly match the labels, providing standardized input for the subsequent lightweight model recognition. After the real-time gesture image is obtained by the air conditioner infrared sensor, the hand area is first filtered by skin color ratio and contour ratio to eliminate non-gesture interference.

[0025] 106. Input the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjust the working state of the air conditioner according to the gesture recognition result; In this embodiment, the labeled real-time image is input into the lightweight recognition model to obtain the gesture recognition result. The degree of fit is used to determine whether to execute the command, and finally, the non-contact control of the air conditioner by gesture is realized, forming a complete closed loop from user gesture to air conditioner response. This technical solution effectively solves the problems of disabled users being unable to hold the remote control and voice control being easily interfered with by environmental noise, truly realizing barrier-free interaction. When the air conditioner main controller receives the command, it controls the electronic expansion valve (temperature regulation), indoor fan (wind speed regulation) and other modules to perform operations. The execution result based on the command is fed back in real time through the air conditioner APP.

[0026] This invention discloses a gesture-based air conditioning interaction method. Through a dual-sample set design (normal and abnormal sample sets), the gesture recognition model is able to recognize abnormal gestures, overcoming the stringent requirements of traditional methods for standardized movements and filling the adaptation blind spot for disabled users. A three-dimensional convolutional network combined with an attention mechanism is used to construct the training model. The three-dimensional convolutional network can effectively extract the spatial and temporal dynamic features of gestures, while the attention mechanism can focus on key areas of the gesture and reduce the influence of non-critical interference features. Even if disabled users cannot perform standard gestures, the gesture recognition model can still achieve accurate recognition, solving the problem of high requirements for joint flexibility and movement coordination in existing technologies. By lightweighting the gesture recognition model, it is ensured that the lightweight model can run efficiently on air conditioning terminal devices, meeting the needs of real-time gesture interaction.

[0027] In this embodiment of the invention, the construction and preprocessing of the training sample set to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set, including: 201. Construct a training sample set, which includes a normal sample set consisting of multiple normal gesture images and an abnormal sample set consisting of multiple abnormal gesture images. In this embodiment, the collected normal and abnormal gesture images are deduplicated to remove duplicate frames, and then filtered to remove blurred and occluded frames, thereby improving the effectiveness of the training sample set.

[0028] 202. The training sample set is subjected to grayscale conversion using a weighted grayscale algorithm, and the training sample set after grayscale conversion is binarized using an adaptive threshold segmentation algorithm based on the image grayscale histogram to obtain a binarized sample set. In this embodiment, a weighted grayscale algorithm is used to perform grayscale conversion on the training sample set, which can improve the contrast between the finger edge and the background and avoid the gap being misjudged as the background during subsequent binarization processing. Based on the image grayscale histogram, the OTSU algorithm is used to calculate the optimal threshold: traversing all possible thresholds T, calculating the inter-class variance corresponding to T, and the T with the largest inter-class variance is the optimal threshold. For abnormal gesture images, the OTSU algorithm will automatically reduce the threshold by 5-10 units to ensure that the gap area is retained as the foreground.

[0029] 203. The binarized sample set is denoised using Gaussian filtering, and the morphological restoration algorithm is used to restore the denoised binarized samples to obtain the restored sample set. In this embodiment, a 5×5 Gaussian convolution kernel is used to smooth isolated noise points in the binarized image, improving the signal-to-noise ratio of the binarized image. This preserves key details of the gesture and solves the problem of blurred edges caused by existing mean filtering, ensuring that subsequent feature extraction can capture the core features of the abnormal gesture. A 3×3 rectangular structuring element is used for closing operations to fill the gaps in the severed fingers and the subtle depressions in the deformed joints. The closing operation includes a dilation step and an erosion step. The dilation step expands the foreground region with the structuring element to fill the gaps, while the erosion step shrinks the foreground region to restore the smoothness of the gesture edges. After repair, the integrity of the abnormal gesture contour is improved, solving the problem of incomplete abnormal gesture contours and avoiding feature vector distortion caused by missing contours, thus laying the foundation for accurate feature extraction in the future.

[0030] 204. A three-dimensional convolutional neural network is used to perform feature extraction processing on the repair sample set to obtain a preprocessed sample set including feature vectors; In this embodiment, the three-dimensional convolutional neural network includes three 3D convolutional layers. The first layer has a 3×3×3 kernel and 32 output channels, extracting basic spatial and temporal features. The second layer has a 3×3×3 kernel and 64 output channels, extracting mid-level features, such as finger joint positions. The third layer has a 3×3×3 kernel and 128 output channels, extracting high-level features, such as the combination of a severed finger and a clenched fist. Finally, global average pooling is used to convert the 128-channel features into a 128-dimensional feature vector, with each channel corresponding to one feature value. The 128-dimensional feature vector enables the conversion between images and structured data, preserving key gesture information while significantly reducing data dimensionality, providing an efficient data format for subsequent annotation, storage, and training of the gesture recognition model.

[0031] In this embodiment of the invention, the step of labeling the preprocessed sample set to obtain a labeled sample set, and constructing a label mapping table based on the labeled sample set, includes: 301. The preprocessed sample set is labeled to obtain a labeled sample set, wherein the labeled tags include gesture category, abnormality type and / or abnormal location; In this embodiment, explicit identity labels are added to the preprocessed 128-dimensional feature vectors. These labels must cover what the gesture is (category), whether it is abnormal (type), and where the abnormality occurs (location). That is, the labels are three-dimensional, allowing the gesture recognition model to understand the actual meaning of the gesture corresponding to the feature vector, thus preventing the gesture recognition model from learning only meaningless feature associations. The specific encoding rules for these labels can be as follows: 1. Gesture categories: 01 = Clenched fist (power on / off), 02 = Index finger up (temperature +1℃), 03 = Index finger down (temperature -1℃), 04 = Fingers spread (mode switching); 2. Abnormal type: 00 = Normal, 01 = Amputated finger, 02 = Missing finger, 03 = Skeletal deformity; 3. Abnormal areas: 00 = None (normal), 01 = Index finger, 02 = Middle finger, 03 = Ring finger, 04 = Little finger, 05 = Palm; By employing 3D labels, the problem of existing solutions only labeling gesture categories without distinguishing between abnormal gestures is solved. This allows the gesture recognition model to accurately identify the differences between normal and abnormal gestures, such as the difference between a normal fist clench and a fist clenched with a missing finger, providing a semantic basis for adaptation for users with disabilities.

[0032] 302. Combining the feature vectors of the preprocessed sample set and the labels of the labeled sample set, a label mapping table is constructed. The label mapping table is stored in a key-value pair structure, where the key is the hash value of the feature vector and the value is a set of labels including gesture category labels, abnormality type labels and / or abnormal part labels. In this embodiment, a hash algorithm is used to convert feature vectors into short-byte keys, and a set of tags is used as values ​​to construct a key-value pair label mapping table. This enables the function of quickly querying tags by inputting feature vectors, avoiding the repeated execution of complex labeling logic during real-time recognition and improving the interactive response speed. Specifically, a simplified version of the SHA-256 hash algorithm is used to generate keys, and ASCII encoding is used to store the generated values. After the label mapping table is constructed, it is synchronously stored in the cloud and the label data area of ​​the air conditioner flash. When the gesture recognition model is iteratively updated, the label mapping table is updated synchronously.

[0033] In this embodiment of the invention, the step of constructing a training model using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the training model based on the labeled sample set to obtain a gesture recognition model, includes: 401. Construct a model to be trained, wherein the model to be trained includes an input layer, a three-dimensional convolutional layer, a CBAM attention layer, an anomaly feature enhancement layer and an output layer connected in sequence. The CBAM attention layer includes a channel attention sub-layer and a spatial attention sub-layer. The anomaly feature enhancement layer adopts a residual connection structure to enhance the gradient propagation of abnormal gesture features. In this embodiment, the input layer is used to receive a sequence of dynamic images, specifically three consecutive 128×128×1 (grayscale) gesture images; the three-dimensional convolutional layer includes three convolutional sub-layers: the first convolutional sub-layer uses a 3×3×3 convolutional kernel, a stride of 1×1×1, 32 output channels, a ReLU activation function, and the same padding method (maintaining feature map size); the second convolutional sub-layer uses a 3×3×3 convolutional kernel, a stride of 1×1×1, 64 output channels, a ReLU activation function, and the same padding method. The third convolutional sublayer uses a 3×3×3 convolutional kernel with a stride of 1×1×1, 128 output channels, ReLU activation function, and same padding. No pooling operation is applied to this 3D convolutional layer to avoid loss of anomalous features. The CBAM attention layer consists of a channel attention sublayer and a spatial attention sublayer: the channel attention sublayer performs global average pooling on the 128-channel features, reduces the dimensionality to 64 channels via 1×1 convolution, activates them using ReLU, then increases the dimensionality to 128 channels via another 1×1 convolution, and finally activates them using Sigma. The Sigmoid function is used to calculate and generate channel weights to enhance key channel features such as joints and severed fingers. The spatial attention sublayer performs global average pooling on the channel-weighted features, reduces the dimensionality to a single channel via 1×1 convolution, and then calculates spatial weights using the Sigmoid function to focus on abnormal areas such as severed finger gaps. The abnormal feature enhancement layer employs a residual connection structure, adding the features output by the CBAM attention layer to the output features of the third convolutional sublayer of the 3D convolutional layer, followed by 1×1 convolution and ReLU activation to enhance abnormal features. The gradient propagation can avoid the gradient vanishing problem during training and improve the gradient magnitude of abnormal features. The output layer contains two fully connected layers: the first fully connected layer transforms the 128-channel × 128 × 128 feature map into a 1024-dimensional feature vector; the second fully connected layer maps the 1024-dimensional feature vector to an 8-dimensional output (corresponding to 4 types of gestures × 2 states: normal / abnormal). After processing by the Softmax activation function, the probability distribution of the 8 types of results is output. For example, the probability of "clenched fist - normal" is 0.95, and the probability of "clenched fist - broken finger" is 0.03.

[0034] 402. The labeled sample set is divided according to a preset division ratio to obtain a training set and a test set; In this embodiment, the labeled sample set is divided into a training set and a test set using a 7:3 ratio. Stratified sampling is used during the division to ensure that the proportion of each type of sample is consistent in the training set and the test set, and to avoid excessive concentration of a certain type of sample in the training set / test set.

[0035] 403. Based on the training set and the test set, the model to be trained is iteratively trained using the Adam optimizer and cosine annealing algorithm to obtain the gesture recognition model; In this embodiment, the Adam optimizer efficiently updates the parameters of the model to be trained, effectively solving the problem of slow convergence speed of traditional stochastic gradient descent (SGD). At the same time, the cosine annealing algorithm is combined to dynamically adjust the learning rate, avoiding parameter oscillations caused by excessively high learning rates in the later stages of training. After multiple iterations, the gesture recognition model converges to the optimal state, ensuring recognition accuracy and generalization ability. In addition, during the training process, a joint optimization strategy of cross-entropy loss and contrastive loss is adopted, which not only ensures accurate classification of gesture categories, but also effectively reduces the feature distance of similar abnormal gestures, improving the generalization ability of real-time abnormal gestures and significantly reducing the risk of recognition failure due to individual user differences.

[0036] In this embodiment of the invention, the step of performing lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploying it includes: 501. The L1 regularized channel pruning algorithm is used to prune the gesture recognition model to obtain a compressed recognition model; In this embodiment, the importance of each channel weight is calculated by L1 regularization, and redundant channels with small absolute weight values ​​are removed. While ensuring minimal loss of accuracy, the size and computing power consumption of the gesture recognition model are greatly reduced, laying the foundation for subsequent quantization processing.

[0037] 502. The compressed recognition model is quantized by using an asymmetric quantization method to quantize the 32-bit floating-point parameters into 8-bit integer parameters to obtain the quantized recognition model. In this embodiment, the 32-bit floating-point parameters (FP32), which account for the majority of storage and computing power in the compressed recognition model, are converted into 8-bit integer parameters (INT8) by using an asymmetric quantization method. This significantly reduces storage overhead and computing power consumption. At the same time, the asymmetric range adapts the parameter distribution, reducing the loss of quantization accuracy. Through quantization processing, the core problem that existing models cannot be deployed on embedded devices is solved.

[0038] 503. The quantized recognition model is distilled using interlayer parameters based on knowledge distillation to obtain a lightweight recognition model; In this embodiment, the original high-precision model (teacher model) that has not been lightweighted is used as a guide to fine-tune the lightweight recognition model (student model) after lightweighting. This allows the student model to learn the knowledge of the teacher model (such as feature distribution and probability output), making up for the accuracy loss caused by pruning and quantization. This ensures that the lightweight recognition model can still accurately recognize the gestures of disabled users without affecting the core interactive functions.

[0039] 504. Convert the lightweight recognition model to the TFLite Micro format supported by the air conditioning main controller and deploy it; In this embodiment, the main control chip of the air conditioner can be an ARM Cortex-M7. The lightweight recognition model is converted into the TFLite Micro format supported by the main control chip of the air conditioner, and deployed to the air conditioner system through block loading, hardware adaptation and other methods to ensure that the lightweight recognition model can run stably and respond to real-time gesture recognition requests. After the main control chip of the air conditioner is powered on, the lightweight recognition model is loaded into RAM in blocks. During inference, the model is loaded and calculated block by block in the order of input layer, 3D convolutional layer, CBAM attention layer, anomaly enhancement layer and output layer to avoid RAM overflow.

[0040] In this embodiment of the invention, the steps of acquiring a real-time gesture image, preprocessing the real-time gesture image, and annotating the preprocessed real-time gesture image based on the annotation mapping table to obtain an annotated real-time image include: 601. Acquire a real-time gesture image, and sequentially perform grayscale conversion, binarization, denoising, repair, and feature extraction on the real-time gesture image to obtain a preprocessed gesture image including real-time feature vectors; In this embodiment, the preprocessing of real-time gesture images is consistent with the preprocessing of the training sample set, solving the problem of large differences between existing real-time processing and training processing, resulting in recognition bias, and improving the recognition accuracy of the lightweight recognition model. Specifically: 1. Grayscale conversion processing: A weighted grayscale algorithm (RGB weights 0.3 / 0.59 / 0.11) is used, consistent with the processing of the training sample set; 2. Binarization: The adaptive OTSU algorithm is used to calculate the threshold. If the threshold is <110 or >190 (extreme lighting), lighting compensation is automatically triggered, the image is recaptured and the threshold is calculated to ensure that the threshold is within the effective range. 3. Denoising and Repair Processing: The Gaussian filtering algorithm, combined with 3×3 rectangular closing operation, ensures that the parameters are completely consistent with the training sample set. 4. Feature extraction processing: The same 3D convolutional network as the training sample set is used to extract 128-dimensional real-time feature vectors, and the dimension of the feature vectors is completely matched with that of the training sample set.

[0041] 602. Based on the annotation mapping table and the real-time feature vector, the preprocessed real-time gesture image is annotated to obtain an annotated real-time image; In this embodiment, by calculating the hash value of the real-time feature vector, a pre-built label mapping table is queried to quickly match the corresponding gesture category, anomaly type, and anomaly part label, generating a labeled real-time image. This provides standardized semantic information for the input of the lightweight recognition model, avoiding repeated learning of label mappings and improving recognition speed. If the match fails, such as if the key is not in the mapping table, temporary labeling is triggered, using the label initially predicted by the lightweight recognition model as a temporary label, which is subsequently synchronized to the cloud to update the label mapping table, avoiding the waste of computing power caused by the lightweight recognition model processing invalid data.

[0042] In this embodiment of the invention, the step of inputting the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjusting the working state of the air conditioner according to the gesture recognition result, includes: 701. Input the labeled real-time image into the lightweight recognition model to obtain gesture recognition results, wherein the gesture recognition results include gesture category, fit degree and / or anomaly type; In this embodiment, a 128-dimensional real-time feature vector from annotated real-time images is input into a lightweight recognition model. The lightweight recognition model infers in the order of 3D convolutional layer, CBAM attention layer, anomaly feature enhancement layer, and output layer, outputting the probability distribution of 8 types of results. For example, the probability of "clenched fist - normal" is 0.02, the probability of "clenched fist - severed finger" is 0.92, and the probabilities of others are <0.05. The category with the highest probability is taken as the gesture category and anomaly type. For example, the probability of 0.92 corresponds to "clenched fist - severed finger". This probability value is the goodness of fit, which reflects the reliability of recognition. The lightweight recognition model infers and outputs multi-dimensional gesture recognition results, solving the problem that existing recognition only outputs categories without reliability judgment. The goodness of fit provides a quantitative basis for subsequent instruction execution, avoiding blindly executing incorrect instructions.

[0043] 702. Obtain a preset similarity threshold, and compare the fitting degree with the preset similarity threshold; In this embodiment, the preset similarity threshold can be 0.85.

[0044] 703. If the fitting degree is greater than or equal to a preset similarity threshold, the working state of the air conditioner is adjusted based on the gesture recognition result; In this embodiment, four sets of gesture mapping relationships are preset, including: clenching a fist (normal / missing finger / deformed) corresponds to powering on or off; raising the index finger upward (normal / missing finger / deformed) corresponds to a temperature increase of 1℃; raising the index finger downward (normal / missing finger / deformed) corresponds to a temperature decrease of 1℃; and spreading all five fingers (normal / multiple missing fingers / deformed) corresponds to switching operating modes. Furthermore, during temperature adjustment, the preset adjustable temperature range is 16-30℃. If the temperature rises to 30℃, it will not be increased further; if the temperature drops to 16℃, it will not be decreased further. During operating mode switching, the system cycles through cooling mode, heating mode, fan mode, and dehumidification mode.

[0045] 704. If the fitting degree is less than a preset similarity threshold, then a gesture adjustment command is generated; In this embodiment, when the recognition result is unreliable, i.e., the fitting degree is less than the preset similarity threshold, the air conditioner control command is not executed. Instead, a gesture adjustment prompt is generated to guide the user to adjust the gesture posture, such as adjusting the angle or adjusting the distance between the user and the air conditioner, thereby improving the success rate of subsequent gesture recognition and preventing the user from giving up due to recognition failure. Text prompts can be pushed through the air conditioner APP, and the air conditioner display screen can be controlled to display simplified icons, such as the "gesture closer" icon, to suit users with poor eyesight.

[0046] The gesture-based air conditioner interaction method in the embodiments of the present invention has been described above. The gesture-based air conditioner interaction device in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 2 One embodiment of the gesture-based air conditioning interaction device in this invention includes: The construction module 801 is used to construct a training sample set and perform preprocessing to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set. The annotation module 802 is used to perform annotation processing on the preprocessed sample set to obtain an annotated sample set, and to construct an annotation mapping table based on the annotated sample set; The training module 803 is used to construct a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and to iteratively train the model to be trained based on the labeled sample set to obtain a gesture recognition model. The lightweight module 804 is used to perform lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploy it. The processing module 805 is used to acquire real-time gesture images, preprocess the real-time gesture images, and annotate the preprocessed real-time gesture images based on the annotation mapping table to obtain an annotated real-time image. The recognition module 806 is used to input the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjust the working state of the air conditioner according to the gesture recognition result.

[0047] Based on the same ideas as the methods in the above embodiments, the apparatus provided by the present invention can implement the methods in the above embodiments.

[0048] The above combination Figure 2 The gesture-based air conditioning interaction device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The gesture-based air conditioning interaction device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0049] Figure 3 This is a schematic diagram of the structure of a gesture-based air conditioning interaction device 900 provided in an embodiment of the present invention. The gesture-based air conditioning interaction device 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (e.g., one or more processors) and a memory 920, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 933 or data 932. The memory 920 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the gesture-based air conditioning interaction device 900. Furthermore, the processor 910 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the gesture-based air conditioning interaction device 900 to implement the steps of the gesture-based air conditioning interaction method provided in the above-described method embodiments.

[0050] The gesture-based air conditioning interaction device 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input / output interfaces 960, and / or one or more operating systems 931, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The illustrated structure of the gesture-based air conditioning interaction device does not constitute a limitation on gesture-based air conditioning interaction devices. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0051] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a gesture-based air conditioning interaction method.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0053] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A gesture-based air conditioner interaction method, characterized in that, include: A training sample set is constructed and preprocessed to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set; The preprocessed sample set is labeled to obtain a labeled sample set, and a label mapping table is constructed based on the labeled sample set; A three-dimensional convolutional network combined with an attention mechanism is used to construct a model to be trained, and the model to be trained is iteratively trained based on the labeled sample set to obtain a gesture recognition model; The gesture recognition model is lightweighted to obtain a lightweight recognition model, which is then deployed. Acquire real-time gesture images, preprocess the real-time gesture images, and annotate the preprocessed real-time gesture images based on the annotation mapping table to obtain an annotated real-time image; The labeled real-time image is input into the lightweight recognition model to obtain the gesture recognition result, and the working state of the air conditioner is adjusted according to the gesture recognition result.

2. The gesture-based air conditioning interaction method according to claim 1, characterized in that, The process of constructing and preprocessing a training sample set yields a preprocessed sample set. The training sample set includes a normal sample set and an abnormal sample set, comprising: Construct a training sample set, which includes a normal sample set consisting of multiple normal gesture images and an abnormal sample set consisting of multiple abnormal gesture images. The training sample set is subjected to grayscale conversion using a weighted grayscale algorithm, and the grayscale converted training sample set is binarized using an adaptive threshold segmentation algorithm based on the image grayscale histogram to obtain a binarized sample set. The binarized sample set is denoised using Gaussian filtering, and the morphological restoration algorithm is used to restore the denoised binarized samples to obtain a restored sample set. A three-dimensional convolutional neural network is used to perform feature extraction on the repair sample set to obtain a preprocessed sample set including feature vectors.

3. The gesture-based air conditioning interaction method according to claim 2, characterized in that, The step of labeling the preprocessed sample set to obtain a labeled sample set, and constructing a label mapping table based on the labeled sample set, includes: The preprocessed sample set is labeled to obtain a labeled sample set, and the labeled labels include gesture category, anomaly type and / or anomaly location; By combining the feature vectors of the preprocessed sample set and the labels of the labeled sample set, a label mapping table is constructed. The label mapping table is stored in a key-value pair structure, where the key is the hash value of the feature vector and the value is a set of labels including gesture category labels, abnormality type labels and / or abnormal part labels.

4. The gesture-based air conditioning interaction method according to claim 1, characterized in that, The method involves constructing a training model using a 3D convolutional network combined with an attention mechanism, and iteratively training the training model based on the labeled sample set to obtain a gesture recognition model, including: A training model is constructed, which includes an input layer, a three-dimensional convolutional layer, a CBAM attention layer, an anomaly feature enhancement layer, and an output layer connected in sequence. The CBAM attention layer includes a channel attention sub-layer and a spatial attention sub-layer. The anomaly feature enhancement layer adopts a residual connection structure to enhance the gradient propagation of anomaly gesture features. The labeled sample set is divided according to a preset division ratio to obtain a training set and a test set; Based on the training and test sets, the model to be trained is iteratively trained using the Adam optimizer and cosine annealing algorithm to obtain the gesture recognition model.

5. The gesture-based air conditioning interaction method according to claim 1, characterized in that, The step of lightweighting the gesture recognition model to obtain a lightweight recognition model and deploying it includes: The gesture recognition model is pruned using the L1 regularized channel pruning algorithm to obtain a compressed recognition model. The compressed recognition model is quantized by using an asymmetric quantization method to quantize the 32-bit floating-point parameters into 8-bit integer parameters, thus obtaining the quantized recognition model. The quantized recognition model is distilled using interlayer parameters based on knowledge distillation to obtain a lightweight recognition model; Convert the lightweight recognition model to the TFLite Micro format supported by the air conditioning main controller and deploy it.

6. The gesture-based air conditioning interaction method according to claim 2, characterized in that, The process of acquiring a real-time gesture image, preprocessing the real-time gesture image, and annotating the preprocessed real-time gesture image based on the annotation mapping table to obtain an annotated real-time image includes: A real-time gesture image is acquired, and the real-time gesture image is sequentially processed by grayscale conversion, binarization, denoising, restoration and feature extraction to obtain a preprocessed gesture image including real-time feature vectors. The preprocessed real-time gesture image is annotated according to the annotation mapping table and the real-time feature vector to obtain an annotated real-time image.

7. The gesture-based air conditioning interaction method according to claim 1, characterized in that, The step of inputting the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjusting the working state of the air conditioner according to the gesture recognition result, includes: The labeled real-time image is input into the lightweight recognition model to obtain gesture recognition results, which include gesture category, fit degree and / or anomaly type. Obtain a preset similarity threshold, and compare the fit with the preset similarity threshold; If the fitting degree is greater than or equal to the preset similarity threshold, the working state of the air conditioner is adjusted based on the gesture recognition result; If the fit is less than a preset similarity threshold, a gesture adjustment command is generated.

8. A gesture-based air conditioning interaction device, characterized in that, include: The construction module is used to construct a training sample set and perform preprocessing to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set. The annotation module is used to annotate the preprocessed sample set to obtain an annotated sample set, and to construct an annotation mapping table based on the annotated sample set; The training module is used to construct a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and to iteratively train the model to be trained based on the labeled sample set to obtain a gesture recognition model. The lightweight module is used to perform lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploy it. The processing module is used to acquire real-time gesture images, preprocess the real-time gesture images, and annotate the preprocessed real-time gesture images based on the annotation mapping table to obtain an annotated real-time image. The recognition module is used to input the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjust the working state of the air conditioner according to the gesture recognition result.

9. A gesture-based air conditioning interaction device, characterized in that, The gesture-based air conditioning interaction device includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the gesture-based air conditioning interaction device to perform the steps of the gesture-based air conditioning interaction method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the gesture-based air conditioning interaction method as described in any one of claims 1-7.

Images