Refrigerator interaction method, device and equipment based on gestures and storage medium

Abstract

The invention relates to the technical field of gesture recognition, in particular to a refrigerator interaction method, device and equipment based on gestures and a storage medium, according to the method, through the double-sample-set design of a normal sample set and an abnormal sample set, a gesture recognition model has the recognition capacity for abnormal gestures, the strict requirement of a traditional scheme for action standardization is broken through, and the gesture recognition efficiency is improved. The adaptive blind area of the handicapped user is filled; a three-dimensional convolutional network is combined with an attention mechanism to construct a to-be-trained model, the three-dimensional convolutional network can effectively extract space and time dynamic features of gestures, the attention mechanism can focus on key areas of the gestures, the influence of non-key interference features is reduced, and even if a handicapped user cannot complete standard gestures, the user experience is improved. The gesture recognition model can still realize accurate recognition, and the problem of high requirements on joint flexibility and action coordination in the prior art is solved; through lightweight processing of the gesture recognition model, it is ensured that the lightweight model can efficiently run on the refrigerator terminal equipment, and the real-time gesture interaction requirement is met.

Description

Technical Field

[0001] This invention relates to the field of gesture recognition technology, and in particular to a gesture-based refrigerator interaction method, apparatus, device, and storage medium. Background Technology

[0002] With the development of smart home technology, gesture recognition-based smart control has gradually become an important direction for improving user interaction experience. Among them, refrigerators, as frequently used home appliances, have seen a surge in demand for gesture control functions in recent years due to their ability to achieve contactless operation and shorten the human-computer interaction path. This is especially true in scenarios such as health and epidemic prevention and elderly care.

[0003] Current mainstream refrigerator gesture control solutions generally adopt a standard ergonomic model design framework. The technical process typically includes: acquiring dynamic image sequences through cameras or infrared sensors, extracting the coordinates of 21 key hand points using open-source frameworks such as MediaPipe, classifying gestures using convolutional neural networks or Transformer models, and finally establishing fixed mapping relationships such as continuous clapping corresponding to extended door opening alarm time, vertical index finger sliding corresponding to temperature adjustment, and single hand five fingers together and then spreading outwards corresponding to mode switching. This can meet the basic operation needs of ordinary users.

[0004] However, this technology system has high requirements for the standardization of movement range and joint flexibility, meaning that there are significant blind spots in the adaptation of the disabled user group. Disabled users are often unable to complete the preset standard gestures due to limited joint movement (such as finger curling caused by rheumatoid arthritis), insufficient muscle strength (such as patients with spinal cord injury), or lack of motor coordination (such as patients with cerebral palsy). Existing solutions lack adaptive adjustment mechanisms for special limb characteristics. For example, they do not consider alternative solutions for single-handed operation, simplify the complexity of gestures, or provide error-tolerant compensation for movements, so disabled users still need to rely on remote control or voice control, failing to achieve the accessibility design goal of technology for all.

[0005] It is evident that existing technologies still need improvement and enhancement. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the present invention aims to provide a gesture-based refrigerator interaction method, which solves the adaptation defects of the prior art for disabled users and significantly improves the inclusiveness and universality of gesture interaction.

[0007] The first aspect of this invention provides a gesture-based refrigerator interaction method, comprising: constructing a training sample set and preprocessing it to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set; labeling the preprocessed sample set to obtain a labeled sample set, and constructing a label mapping table based on the labeled sample set; constructing a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the model to be trained based on the labeled sample set to obtain a gesture recognition model; performing lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploying it; acquiring real-time illumination intensity and real-time gesture images, preprocessing the real-time gesture images based on the real-time illumination intensity to obtain a preprocessed real-time image; labeling the preprocessed real-time image based on the label mapping table to obtain a labeled real-time image; inputting the labeled real-time image into the lightweight recognition model to obtain a gesture recognition result, and adjusting the working state of the refrigerator according to the gesture recognition result.

[0008] Optionally, in a first implementation of the first aspect of the present invention, the step of constructing a training sample set and preprocessing it to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set, includes: constructing a training sample set, wherein the training sample set includes a normal sample set and an abnormal sample set, wherein the normal sample set includes multiple normal hand gesture images under normal lighting and multiple normal hand gesture images in dark environments, and the abnormal sample set includes multiple abnormal hand gesture images under normal lighting and multiple abnormal hand gesture images in dark environments; and applying histogram equalization to the normal hand gesture images in dark environments and the dark ring images in the training sample set. The abnormal gesture images in the environment are subjected to low-light grayscale correction processing to obtain a correction sample set; the correction sample set is then subjected to grayscale conversion processing using a weighted grayscale algorithm, and the grayscale conversion training sample set is binarized using an adaptive threshold segmentation algorithm based on the image grayscale histogram to obtain a binarized sample set; the binarized sample set is then denoised using a Gaussian filtering method, and the denoised binarized sample set is then repaired using a morphological restoration algorithm to obtain a repaired sample set; finally, the repaired sample set is subjected to feature extraction processing using a three-dimensional convolutional neural network to obtain a preprocessed sample set including feature vectors.

[0009] Optionally, in a second implementation of the first aspect of the present invention, the step of labeling the preprocessed sample set to obtain a labeled sample set, and constructing a label mapping table based on the labeled sample set, includes: labeling the preprocessed sample set to obtain a labeled sample set, wherein the labeled tags include gesture category, anomaly type, and ambient light level; and constructing a label mapping table by combining the feature vector of the preprocessed sample set and the tags of the labeled sample set, wherein the label mapping table is stored in a key-value pair structure, wherein the key is the hash value of the feature vector, and the value is a set of tags including gesture category tags, anomaly type tags, and ambient light level tags.

[0010] Optionally, in a third implementation of the first aspect of the present invention, the step of constructing a training model using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the training model based on the labeled sample set to obtain a gesture recognition model, includes: constructing a training model, the training model comprising an input layer, a three-dimensional convolutional layer, a CBAM attention layer, an anomaly feature enhancement layer, and an output layer connected in sequence, the CBAM attention layer comprising a channel attention sublayer and a spatial attention sublayer, the anomaly feature enhancement layer employing a residual connection structure to enhance the gradient propagation of anomaly gesture features; dividing the labeled sample set according to a preset partitioning ratio to obtain a training set and a test set; and iteratively training the training model based on the training set and the test set using an Adam optimizer and a cosine annealing algorithm, combined with a cross-entropy loss function and an ambient light loss function, to obtain a gesture recognition model.

[0011] Optionally, in the fourth implementation of the first aspect of the present invention, the step of lightweighting the gesture recognition model to obtain a lightweight recognition model and deploying it includes: pruning the gesture recognition model using an L1 regularized channel pruning algorithm to obtain a compressed recognition model; quantizing the compressed recognition model by using an asymmetric quantization method to quantize 32-bit floating-point parameters into 8-bit integer parameters to obtain a quantized recognition model; distilling the quantized recognition model using inter-layer parameters based on knowledge distillation to obtain a lightweight recognition model; and converting the lightweight recognition model into the TFLite Micro format supported by the refrigerator main controller and deploying it.

[0012] Optionally, in a fifth implementation of the first aspect of the present invention, the refrigerator is equipped with an infrared sensor and a light sensor, the infrared sensor having an infrared supplementary lighting function; the step of acquiring real-time light intensity and real-time gesture image, and preprocessing the real-time gesture image based on the real-time light intensity to obtain a preprocessed real-time image includes: acquiring the real-time light intensity fed back by the light sensor, comparing the real-time light intensity with a preset compensation activation threshold; if the real-time light intensity < the preset supplementary lighting activation threshold, controlling the infrared sensor to perform infrared supplementary lighting, and acquiring the real-time gesture image fed back by the infrared sensor, sequentially performing weak light grayscale correction processing, grayscale conversion processing, binarization processing, denoising processing, repair processing, and feature extraction processing on the real-time gesture image to obtain a preprocessed gesture image including real-time feature vectors; if the real-time light intensity ≥ the preset supplementary lighting activation threshold, acquiring the real-time gesture image fed back by the infrared sensor, sequentially performing grayscale conversion processing, binarization processing, denoising processing, repair processing, and feature extraction processing on the real-time gesture image to obtain a preprocessed gesture image including real-time feature vectors.

[0013] Optionally, in a sixth implementation of the first aspect of the present invention, the step of inputting the labeled real-time image into the lightweight recognition model to obtain a gesture recognition result, and adjusting the working state of the refrigerator according to the gesture recognition result, includes: inputting the labeled real-time image into the lightweight recognition model to obtain a gesture recognition result, wherein the gesture recognition result includes gesture category, anomaly type, ambient light adaptability, and fit; obtaining a preset similarity threshold, comparing the fit and the preset similarity threshold; if the fit ≥ the preset similarity threshold, adjusting the working state of the refrigerator based on the gesture recognition result; if the fit < the preset similarity threshold, generating a gesture adjustment command.

[0014] A second aspect of the present invention provides a gesture-based refrigerator interaction device, comprising: a construction module for constructing and preprocessing a training sample set to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set; a first annotation module for annotating the preprocessed sample set to obtain an annotated sample set, and constructing an annotation mapping table based on the annotated sample set; a training module for constructing a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the model to be trained based on the annotated sample set to obtain a gesture recognition model; a lightweighting module for lightweighting the gesture recognition model to obtain a lightweight recognition model and deploying it; a processing module for acquiring real-time illumination intensity and real-time gesture images, preprocessing the real-time gesture images based on the real-time illumination intensity to obtain a preprocessed real-time image; a second annotation module for annotating the preprocessed real-time image based on the annotation mapping table to obtain an annotated real-time image; and a recognition module for inputting the annotated real-time image into the lightweight recognition model to obtain a gesture recognition result, and adjusting the working state of the refrigerator according to the gesture recognition result.

[0015] A third aspect of the present invention provides a gesture-based refrigerator interaction device, the gesture-based refrigerator interaction device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the gesture-based refrigerator interaction device to perform various steps of the gesture-based refrigerator interaction method described in any of the preceding claims.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the gesture-based refrigerator interaction method described in any of the preceding claims.

[0017] In the technical solution of this invention, the dual-sample set design of normal and abnormal sample sets enables the gesture recognition model to recognize abnormal gestures, breaking through the stringent requirements of traditional solutions for action standardization and filling the adaptation blind spot for disabled users. A three-dimensional convolutional network combined with an attention mechanism is used to construct the model to be trained. The three-dimensional convolutional network can effectively extract the spatial and temporal dynamic features of gestures, while the attention mechanism can focus on key areas of the gesture and reduce the influence of non-critical interference features. Even if disabled users cannot complete standard gestures, the gesture recognition model can still achieve accurate recognition, solving the problem of high requirements for joint flexibility and action coordination in existing technologies. By lightweighting the gesture recognition model, it is ensured that the lightweight model can run efficiently on refrigerator terminal devices, meeting the needs of real-time gesture interaction. Attached Figure Description

[0018] Figure 1A logic flowchart of a gesture-based refrigerator interaction method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a gesture-based refrigerator interaction device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a gesture-based refrigerator interaction device provided in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a gesture-based refrigerator interaction method, apparatus, device, and storage medium. In this invention, the terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the gesture-based refrigerator interaction method in this invention includes: 101. Construct a training sample set and preprocess it to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set; In this embodiment, the normal sample set corresponds to standard gestures that ordinary users can complete, such as tapping with a complete finger or sliding with the index finger. The abnormal sample set corresponds to special forms of gestures that disabled users cannot complete due to missing fingers, skeletal deformities, etc., such as tapping with a missing finger or raising a deformed index finger. The total size of the training sample set is ≥20,000 frames, of which the abnormal sample set covers 3 core anomalies: missing fingers (1 / 3-1 / 2 of the length is missing), missing fingers (1-2 fingers are completely missing), and skeletal deformities (joint extension angle ≤120°). The proportion of real disabled user samples in the abnormal samples is ≥75%, and the proportion of synthetic samples is ≤25%, to ensure the authenticity and diversity of the data. In addition, the proportion of dark environment samples is 30%. Dark environment refers to an ambient light intensity in the range of 5-50 lux, and the background is a common refrigerator door material.

[0021] 102. The preprocessed sample set is labeled to obtain a labeled sample set, and a label mapping table is constructed based on the labeled sample set; In this embodiment, a combination of manual and automated annotation is used to add identity tags to the preprocessed feature vectors and establish a fast mapping relationship between feature vectors and tags. This allows the gesture recognition model to distinguish between normal / abnormal gestures and different abnormal types, while improving the tag matching efficiency during real-time recognition. The annotation mapping table enables quick lookup of feature vectors and corresponding tags, avoiding repeated annotation during real-time recognition, improving interactive response speed, and meeting the real-time control requirements of the refrigerator.

[0022] 103. A three-dimensional convolutional network combined with an attention mechanism is used to construct a model to be trained, and the model to be trained is iteratively trained based on the labeled sample set to obtain a gesture recognition model; In this embodiment, to meet the requirements of gesture dynamism and saliency of abnormal features, a dedicated network structure is designed. Through iterative training on a labeled sample set, the gesture recognition model is able to recognize normal gestures and accurately match abnormal gestures. Compared with existing solutions that use 2D convolutional networks to extract only static features and lack an attention mechanism, this embodiment uses 3D convolution to capture dynamic gestures and combines a CBAM attention layer to focus on abnormal features, thereby solving the problem of poor performance of existing models in recognizing dynamic abnormal gestures and improving the accuracy of gesture recognition for disabled users.

[0023] 104. Perform lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploy it; In this embodiment, to address the hardware limitations of refrigerator main control chips, such as ARM Cortex-M4, a lightweight recognition model is obtained by compressing the size of the gesture recognition model and reducing computing power consumption. This model can run stably in the refrigerator embedded system while retaining core recognition capabilities. This solves the problem that existing models are large in size and have high computing power, making them unsuitable for deployment in refrigerator embedded systems. It ensures that the model can be adapted to the hardware and deployed without sacrificing accuracy.

[0024] 105. Obtain real-time illumination intensity and real-time gesture image, and preprocess the real-time gesture image based on the real-time illumination intensity to obtain a preprocessed real-time image; In this embodiment, an ambient light intensity is collected in real time by a light sensor located 10cm above the refrigerator door. The preprocessing flow of the real-time gesture image is dynamically adjusted based on the collected light intensity to solve the problems of blurred features in dark environments and interference from reflections in bright environments. Furthermore, the user's real-time gestures are captured by the refrigerator's infrared sensor, and the preprocessing logic of the training sample set is reproduced to ensure the consistency between the real-time feature vector and the feature vector during training. After the real-time gesture image is acquired using the refrigerator's infrared sensor, the hand area is first filtered by skin tone ratio and contour ratio to eliminate non-gesture interference.

[0025] 106. Based on the annotation mapping table, the preprocessed real-time image is annotated to obtain an annotated real-time image; In this embodiment, by calculating the hash value of the real-time feature vector, a pre-built label mapping table is queried to quickly match the corresponding gesture category, anomaly type, and ambient light level, generating a labeled real-time image. This provides standardized semantic information for the input of the lightweight recognition model, avoiding repeated learning of label mappings and improving recognition speed. Specifically, if the real-time feature vector matches a certain key value with a degree of ≥80%, a corresponding label is assigned, such as "little finger swipe - reverse direction - illumination 25 lux". If the degree of matching is <80%, the matching fails, such as if the key is not in the mapping table. In this case, temporary labeling is triggered, using the label initially predicted by the lightweight recognition model as a temporary label, which is subsequently synchronized to the cloud to update the label mapping table, avoiding the waste of computing power caused by the lightweight recognition model processing invalid data.

[0026] 107. Input the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjust the working state of the refrigerator according to the gesture recognition result; In this embodiment, the labeled real-time image is input into a lightweight recognition model to obtain gesture recognition results. Based on the fit, it is determined whether to execute the command, ultimately realizing non-contact control of the refrigerator through gestures, forming a complete closed loop from user gestures to refrigerator response. This technical solution effectively solves the problems of disabled users being unable to hold the remote control and voice control being easily interfered with by environmental noise, truly achieving barrier-free interaction. When the refrigerator main controller receives the command, it controls the corresponding module to execute the corresponding operation, such as temperature adjustment, delayed alarm, etc., and then provides real-time feedback on the adjustment results based on the command through the refrigerator APP.

[0027] This invention discloses a gesture-based refrigerator interaction method. Through a dual-sample set design (normal and abnormal sample sets), the gesture recognition model is able to recognize abnormal gestures, overcoming the stringent requirements of traditional methods for standardized movements and filling the adaptation blind spot for disabled users. A three-dimensional convolutional network combined with an attention mechanism is used to construct the training model. The three-dimensional convolutional network can effectively extract the spatial and temporal dynamic features of gestures, while the attention mechanism can focus on key areas of the gesture and reduce the influence of non-critical interference features. Even if disabled users cannot perform standard gestures, the gesture recognition model can still achieve accurate recognition, solving the problem of high requirements for joint flexibility and movement coordination in existing technologies. By lightweighting the gesture recognition model, it is ensured that the lightweight model can run efficiently on the refrigerator terminal device, meeting the needs of real-time gesture interaction.

[0028] In this embodiment of the invention, the construction and preprocessing of the training sample set to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set, including: 201. Construct a training sample set, which includes a normal sample set and an abnormal sample set. The normal sample set includes multiple normal hand gesture images with normal lighting and multiple normal hand gesture images with dark environments. The abnormal sample set includes multiple abnormal hand gesture images with normal lighting and multiple abnormal hand gesture images with dark environments. In this embodiment, the collected normal and abnormal gesture images are deduplicated to remove duplicate frames, and then filtered to remove blurred and occluded frames, thereby improving the effectiveness of the training sample set.

[0029] 202. Histogram equalization is used to perform low-light grayscale correction on normal hand gesture images and abnormal hand gesture images in dark environments in the training sample set to obtain a correction sample set. In this embodiment, to address the issues of narrow grayscale range and confusion between gestures and door background in dark environment samples, the grayscale range is stretched through histogram equalization to improve contrast. Specifically, the grayscale histogram of the gesture image in the dark environment is stretched from a concentrated range (e.g., 80-120) to 0-255, making the finger area (grayscale 120-180) significantly different from the door background (grayscale 80-120). The grayscale variance of the corrected dark environment sample is increased from 15 to 40, improving the clarity of the finger edges and avoiding the loss of details such as the missing finger notch during subsequent binarization.

[0030] 203. The weighted grayscale algorithm is used to perform grayscale conversion on the correction sample set, and an adaptive threshold segmentation algorithm is used to perform binarization on the grayscale converted training sample set based on the image grayscale histogram to obtain a binarized sample set. In this embodiment, a weighted grayscale algorithm is used to perform grayscale conversion on the training sample set, which can improve the contrast between the finger edge and the background and avoid the gap being misjudged as the background during subsequent binarization processing. Based on the image grayscale histogram, the OTSU algorithm is used to calculate the optimal threshold: traversing all possible thresholds T, calculating the inter-class variance corresponding to T, and the T with the largest inter-class variance is the optimal threshold. For abnormal gesture images, the OTSU algorithm will automatically reduce the threshold by 5-10 units to ensure that the gap area is retained as the foreground.

[0031] 204. The binarized sample set is denoised using Gaussian filtering, and the morphological restoration algorithm is used to restore the denoised binarized samples to obtain the restored sample set. In this embodiment, a 5×5 Gaussian convolution kernel is used to smooth isolated noise points in the binarized image, improving the signal-to-noise ratio of the binarized image. This preserves key details of the gesture and solves the problem of blurred edges caused by existing mean filtering, ensuring that subsequent feature extraction can capture the core features of the abnormal gesture. A 3×3 rectangular structuring element is used for closing operations to fill the gaps in the severed fingers and the subtle depressions in the deformed joints. The closing operation includes a dilation step and an erosion step. The dilation step expands the foreground region with the structuring element to fill the gaps, while the erosion step shrinks the foreground region to restore the smoothness of the gesture edges. After repair, the integrity of the abnormal gesture contour is improved, solving the problem of incomplete abnormal gesture contours and avoiding feature vector distortion caused by missing contours, thus laying the foundation for accurate feature extraction in the future.

[0032] 205. A three-dimensional convolutional neural network is used to perform feature extraction processing on the repair sample set to obtain a preprocessed sample set including feature vectors; In this embodiment, the three-dimensional convolutional neural network includes three 3D convolutional layers. The first layer has a 3×3×3 kernel and 32 output channels, extracting basic spatial and temporal features. The second layer has a 3×3×3 kernel and 64 output channels, extracting mid-level features, such as finger joint positions. The third layer has a 3×3×3 kernel and 128 output channels, extracting high-level features, such as continuous taps and combinations of severed fingers. Finally, global average pooling is used to convert the 128-channel features into a 128-dimensional feature vector, with each channel corresponding to one feature value. The 128-dimensional feature vector enables the conversion between images and structured data, preserving key gesture information while significantly reducing data dimensionality, providing an efficient data format for subsequent annotation, storage, and training of the gesture recognition model.

[0033] In this embodiment of the invention, the step of labeling the preprocessed sample set to obtain a labeled sample set, and constructing a label mapping table based on the labeled sample set, includes: 301. The preprocessed sample set is labeled to obtain a labeled sample set. The labeled labels include gesture category, anomaly type and ambient light level. In this embodiment, a clear identity label is added to the preprocessed 128-dimensional feature vector. The label needs to cover what the gesture is (category), the anomaly type (whether it is abnormal and the type of anomaly), and the ambient light level (5 levels). That is, the label is a three-dimensional label, which allows the gesture recognition model to understand the actual meaning of the gesture corresponding to the feature vector, avoiding the gesture recognition model only learning meaningless feature associations. The specific encoding rule of the label can be: 1. Gesture categories: 01 = continuous clapping (extends door opening alarm time), 02 = index finger up (temperature +1℃), 03 = index finger down (temperature -1℃), 04 = fingers spread (mode switching); 2. Abnormal type: 00 = Normal, 01 = Index finger amputation, 02 = Index finger missing, 03 = Skeletal deformity, 04 = Palmar base missing edge; 3. Ambient light level: 00 is <5 lux, 01 is 5-20 lux, 02 is 20-30 lux, 03 is 30-50 lux, and 04 is ≥50 lux; By employing 3D labels, the problem of existing solutions only labeling gesture categories without distinguishing between abnormal gestures is solved. This allows the gesture recognition model to accurately identify the differences between normal and abnormal gestures, such as the difference between normal clapping and clapping with a missing finger, providing a semantic basis for adaptation for users with disabilities.

[0034] 302. Combining the feature vectors of the preprocessed sample set and the labels of the labeled sample set, a label mapping table is constructed. The label mapping table is stored in a key-value pair structure, where the key is the hash value of the feature vector and the value is a set of labels including gesture category labels, abnormality type labels and ambient light level labels. In this embodiment, a hash algorithm is used to convert feature vectors into short-byte keys, and a set of tags is used as values ​​to construct a key-value pair label mapping table. This enables the function of quickly querying tags by inputting feature vectors, avoiding the repeated execution of complex labeling logic during real-time recognition and improving the interactive response speed. Specifically, a simplified version of the SHA-256 hash algorithm is used to generate keys, and ASCII encoding is used to store the generated values. After the label mapping table is constructed, it is synchronously stored in the cloud and the label data area of ​​the refrigerator flash. When the gesture recognition model is iteratively updated, the label mapping table is updated synchronously.

[0035] In this embodiment of the invention, the step of constructing a training model using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the training model based on the labeled sample set to obtain a gesture recognition model, includes: 401. Construct a model to be trained, wherein the model to be trained includes an input layer, a three-dimensional convolutional layer, a CBAM attention layer, an anomaly feature enhancement layer and an output layer connected in sequence. The CBAM attention layer includes a channel attention sub-layer and a spatial attention sub-layer. The anomaly feature enhancement layer adopts a residual connection structure to enhance the gradient propagation of abnormal gesture features. In this embodiment, the input layer is used to receive dynamic image sequences, specifically a three-dimensional tensor (128, 128, 3) and a 5-dimensional ambient light label vector. The ambient light label vector is upscaled using a 1×1×1 convolutional sublayer, and then the upscaled ambient light label vector is embedded into the image feature channel dimension using a feature concatenation sublayer. The three-dimensional convolutional layer includes two convolutional sublayers: the first convolutional sublayer uses a 3×3×3 convolutional kernel with a stride of 1×1×1, 32 output channels, a ReLU activation function, and a "same" padding method (maintaining feature map size). The second convolutional sublayer uses a 5×5×5 convolutional kernel with a stride of 1×1×1, 64 output channels, ReLU activation function, and same padding. These two convolutional sublayers are used to capture the spatial shape of gestures (e.g., finger spread angle) and temporal trajectory (e.g., swipe direction). No pooling operation is applied to the 3D convolutional layer to avoid loss of anomalous features. The CBAM attention layer consists of a channel attention sublayer and a spatial attention sublayer: the channel attention sublayer performs global average pooling on the 64-channel features and generates the output channel using the Sigmoid function. Channel weights are used to enhance key channel features such as joints and severed fingers. The spatial attention sublayer performs global average pooling on the channel-weighted features, reduces the dimensionality to a single channel via 3×3 convolution, and generates spatial weights using the Sigmoid function to filter out areas with filler light spots and focus on abnormal areas such as severed finger gaps. The abnormal feature enhancement layer employs a residual connection structure, adding the features output from the CBAM attention layer to the output features from the second convolutional sublayer of the 3D convolutional layer, followed by 1×1 convolution for forced alignment and ReLU activation to enhance abnormal features. The gradient propagation can avoid the gradient vanishing problem during training and improve the gradient magnitude of abnormal features. The output layer contains two fully connected layers: the first fully connected layer transforms the 64-channel × 128 × 128 feature map into a 1024-dimensional feature vector; the second fully connected layer maps the 1024-dimensional feature vector to an 8-dimensional output (corresponding to 4 types of gestures × 2 states: normal / abnormal). After processing by the Softmax activation function, the probability distribution of the 8 types of results is output. For example, the probability of "continuous tapping - normal" is 0.95, and the probability of "continuous tapping - broken finger" is 0.03.

[0036] 402. The labeled sample set is divided according to a preset division ratio to obtain a training set and a test set; In this embodiment, the labeled sample set is divided into a training set and a test set using a 7:3 ratio. Stratified sampling is used during the division to ensure that the proportion of each type of sample is consistent in the training set and the test set, and to avoid excessive concentration of a certain type of sample in the training set / test set.

[0037] 403. Based on the training set and the test set, the Adam optimizer and cosine annealing algorithm are used, combined with the cross-entropy loss function and the ambient light loss function to iteratively train the model to be trained, so as to obtain the gesture recognition model. In this embodiment, the Adam optimizer efficiently updates the parameters of the model to be trained, effectively solving the problem of slow convergence speed of traditional stochastic gradient descent (SGD). At the same time, the cosine annealing algorithm is combined to dynamically adjust the learning rate, avoiding parameter oscillations caused by excessively high learning rates in the later stages of training. After multiple iterations, the gesture recognition model converges to the optimal state, ensuring recognition accuracy and generalization ability. In addition, during the training process, a joint optimization strategy of cross-entropy loss function and ambient light loss function is adopted, which not only ensures accurate classification of gesture categories, but also calculates the feature distance of the same gesture under different lighting conditions, reducing the difference and significantly reducing the risk of recognition failure caused by differences in lighting scenes.

[0038] In this embodiment of the invention, the step of performing lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploying it includes: 501. The L1 regularized channel pruning algorithm is used to prune the gesture recognition model to obtain a compressed recognition model; In this embodiment, the importance of each channel weight is calculated by L1 regularization, and redundant channels with small absolute weight values ​​are removed. While ensuring minimal loss of accuracy, the size and computing power consumption of the gesture recognition model are greatly reduced, laying the foundation for subsequent quantization processing.

[0039] 502. The compressed recognition model is quantized by using an asymmetric quantization method to quantize the 32-bit floating-point parameters into 8-bit integer parameters to obtain the quantized recognition model. In this embodiment, the 32-bit floating-point parameters (FP32), which account for the majority of storage and computing power in the compressed recognition model, are converted into 8-bit integer parameters (INT8) by using an asymmetric quantization method. This significantly reduces storage overhead and computing power consumption. At the same time, the asymmetric range adapts the parameter distribution, reducing the loss of quantization accuracy. Through quantization processing, the core problem that existing models cannot be deployed on embedded devices is solved.

[0040] 503. The quantized recognition model is distilled using interlayer parameters based on knowledge distillation to obtain a lightweight recognition model; In this embodiment, the original high-precision model (teacher model) that has not been lightweighted is used as a guide to fine-tune the lightweight recognition model (student model) after lightweighting. This allows the student model to learn the knowledge of the teacher model (such as feature distribution and probability output), making up for the accuracy loss caused by pruning and quantization. This ensures that the lightweight recognition model can still accurately recognize the gestures of disabled users without affecting the core interactive functions.

[0041] 504. Convert the lightweight recognition model to the TFLite Micro format supported by the refrigerator's main control unit and deploy it; In this embodiment, the main control chip of the refrigerator can be an ARM Cortex-M4; the lightweight recognition model is converted into the TFLite Micro format supported by the refrigerator's main control chip, and deployed to the approximation system through block loading, hardware adaptation, and other methods to ensure that the lightweight recognition model can run stably and respond to real-time gesture recognition requests; after the refrigerator's main control is powered on, the lightweight recognition model is loaded into RAM in blocks, and during inference, it is loaded and calculated block by block in the order of input layer, 3D convolutional layer, CBAM attention layer, anomaly enhancement layer, and output layer to avoid RAM overflow.

[0042] In this embodiment of the invention, the refrigerator is equipped with an infrared sensor and a light sensor, the infrared sensor having an infrared supplementary lighting function; the step of acquiring real-time light intensity and real-time gesture images, and preprocessing the real-time gesture images based on the real-time light intensity to obtain a preprocessed real-time image includes: 601. Obtain the real-time light intensity fed back by the light sensor, and compare the real-time light intensity with the preset compensation start threshold; In this embodiment, an ambient light sensor is used to monitor the ambient light in real time to determine whether supplemental lighting needs to be activated. The light sensor can be installed 10cm above the front of the refrigerator door. The preset supplemental lighting activation threshold is 50 lux, which can also be manually adjusted by the user through the refrigerator APP.

[0043] 602. If the real-time light intensity is less than the preset supplementary light activation threshold, the infrared sensor is controlled to perform infrared supplementary light and the real-time gesture image fed back by the infrared sensor is acquired. The real-time gesture image is then subjected to weak light grayscale correction processing, grayscale conversion processing, binarization processing, noise reduction processing, repair processing and feature extraction processing in sequence to obtain a preprocessed gesture image including real-time feature vectors. In this embodiment, the infrared sensor is installed 10cm above the front of the refrigerator door, tilted at 15° towards the user interaction area to reduce direct glare from the door. The user interaction area is located 0.5-1.5m away from the refrigerator door. In dark environments, the material of the refrigerator door (stainless steel reflective, glass transparent / semi-transparent) causes a high degree of overlap between the grayscale range of the gesture and the door background. If the contrast is not enhanced, the missing finger notch (grayscale value close to the background) will be misjudged as the door background, and the joint details of the deformed gesture will be blurred, leading to recognition failure. Therefore, enhancing the contrast between the finger edge and the door background is the core prerequisite for ensuring that abnormal gesture features are not lost.

[0044] In this embodiment, the infrared fill light uses an 850nm near-infrared LED, and its fill light intensity dynamically increases by 15% for every 10 lux decrease in ambient light intensity; for example, the fill light intensity is 50% when the ambient light intensity is 5-20 lux, and 35% when the ambient light intensity is 20-30 lux, effectively preventing overexposure of the image.

[0045] 603. If the real-time illumination intensity is greater than or equal to the preset supplementary lighting threshold, then the real-time gesture image fed back by the infrared sensor is acquired, and the real-time gesture image is sequentially subjected to grayscale conversion, binarization, denoising, repair and feature extraction to obtain a preprocessed gesture image including real-time feature vectors. In this embodiment, the preprocessing of real-time gesture images is consistent with the preprocessing of the training sample set, solving the problem of large differences between existing real-time processing and training processing, resulting in recognition bias, and improving the recognition accuracy of the lightweight recognition model. Specifically: 1. In dark environments, weak light grayscale correction is performed first, and adaptive histogram equalization is used in the same way as the dark environment sample processing of the training sample set. 2. Grayscale conversion processing: A weighted grayscale algorithm (RGB weights 0.3 / 0.59 / 0.11) is used, consistent with the processing of the training sample set; 3. Binarization processing: The adaptive OTSU algorithm is used to calculate the threshold. If the threshold is <110 or >190 (extreme lighting), lighting compensation is automatically triggered, the image is recaptured and the threshold is calculated to ensure that the threshold is within the effective range. 4. Denoising and Repair Processing: The Gaussian filtering algorithm, combined with 3×3 rectangular closing operation, ensures that the parameters are completely consistent with those of the training sample set. 5. Feature extraction processing: The same 3D convolutional network as the training sample set is used to extract 128-dimensional real-time feature vectors, and the dimension of the feature vectors is completely matched with that of the training sample set.

[0046] In this embodiment of the invention, the step of inputting the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjusting the working state of the refrigerator according to the gesture recognition result, includes: 701. Input the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, which includes gesture category, anomaly type, ambient light adaptability, and fit degree; In this embodiment, a 128-dimensional real-time feature vector from annotated real-time images is input into a lightweight recognition model. The lightweight recognition model infers in the order of 3D convolutional layer, CBAM attention layer, anomaly feature enhancement layer, and output layer, outputting the probability distribution of 8 types of results. For example, the probability of "continuous tapping - normal" is 0.02, the probability of "continuous tapping - broken finger" is 0.92, and the probabilities of others are <0.05. The category with the highest probability is taken as the gesture category and anomaly type. For example, the probability of 0.92 corresponds to "continuous tapping - broken finger". This probability value is the goodness of fit, which reflects the reliability of recognition. The lightweight recognition model infers and outputs multi-dimensional gesture recognition results, solving the problem that existing recognition only outputs categories without reliability judgment. The goodness of fit provides a quantitative basis for subsequent instruction execution, avoiding blindly executing incorrect instructions.

[0047] 702. Obtain a preset similarity threshold, and compare the fitting degree with the preset similarity threshold; In this embodiment, the preset similarity threshold can be 0.85.

[0048] 703. If the fitting degree is greater than or equal to a preset similarity threshold, the working state of the refrigerator is adjusted based on the gesture recognition result; In this embodiment, four sets of gesture and command mapping relationships are preset, including: continuous tapping (normal / missing finger / deformed) corresponds to powering on or off, raising the index finger (normal / missing finger / deformed) corresponds to a temperature of +1℃, raising the index finger (normal / missing finger / deformed) corresponds to a temperature of -1℃, and spreading all five fingers (normal / multiple missing fingers / deformed) corresponds to switching operating modes.

[0049] 704. If the fitting degree is less than a preset similarity threshold, then a gesture adjustment command is generated; In this embodiment, when the recognition result is unreliable, i.e., the fitting degree is less than the preset similarity threshold, the air conditioner control command is not executed. Instead, a gesture adjustment prompt is generated to guide the user to adjust the gesture posture, such as adjusting the angle or adjusting the distance between the user and the refrigerator, thereby improving the success rate of subsequent gesture recognition and preventing the user from giving up due to recognition failure. Text prompts can be pushed through the refrigerator APP, and the refrigerator display screen can be controlled to display simplified icons, such as the "gesture closer" icon, to suit users with poor eyesight.

[0050] The above describes the gesture-based refrigerator interaction method in the embodiments of the present invention. The following describes the gesture-based refrigerator interaction device in the embodiments of the present invention. Please refer to [link / reference]. Figure 2 One embodiment of the gesture-based refrigerator interaction device in this invention includes: The construction module 801 is used to construct a training sample set and perform preprocessing to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set. The first annotation module 802 is used to annotate the preprocessed sample set to obtain an annotated sample set, and to construct an annotation mapping table based on the annotated sample set; The training module 803 is used to construct a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and to iteratively train the model to be trained based on the labeled sample set to obtain a gesture recognition model. The lightweight module 804 is used to perform lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploy it. Processing module 805 is used to acquire real-time light intensity and real-time gesture image, and preprocess the real-time gesture image based on real-time light intensity to obtain preprocessed real-time image; The second annotation module 806 is used to perform annotation processing on the preprocessed real-time image based on the annotation mapping table to obtain an annotated real-time image; The recognition module 807 is used to input the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjust the working state of the refrigerator according to the gesture recognition result.

[0051] Based on the same ideas as the methods in the above embodiments, the apparatus provided by the present invention can implement the methods in the above embodiments.

[0052] The above combination Figure 2 The gesture-based refrigerator interaction device in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The gesture-based refrigerator interaction device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0053] Figure 3 This is a schematic diagram of the structure of a gesture-based refrigerator interaction device 900 provided in an embodiment of the present invention. The gesture-based refrigerator interaction device 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (e.g., one or more processors) and a memory 920, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 933 or data 932. The memory 920 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the gesture-based refrigerator interaction device 900. Furthermore, the processor 910 may be configured to communicate with the storage media 930 and execute a series of instruction operations in the storage media 930 on the gesture-based refrigerator interaction device 900 to implement the steps of the gesture-based refrigerator interaction method provided in the above-described method embodiments.

[0054] The gesture-based refrigerator interaction device 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input / output interfaces 960, and / or one or more operating systems 931, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The illustrated structure of the gesture-based refrigerator interaction device does not constitute a limitation on gesture-based refrigerator interaction devices. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0055] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a gesture-based refrigerator interaction method.

[0056] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0057] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A gesture-based refrigerator interaction method, characterized by, The method comprises the following steps: constructing a training sample set and performing preprocessing to obtain a preprocessed sample set, wherein the training sample set comprises a normal sample set and an abnormal sample set; performing annotation processing on the preprocessed sample set to obtain an annotated sample set, and constructing an annotation mapping table based on the annotated sample set; constructing a to-be-trained model by using a three-dimensional convolutional network combined with an attention mechanism, and iteratively training the to-be-trained model based on the annotated sample set to obtain a gesture recognition model; performing lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploying the lightweight recognition model; obtaining real-time light intensity and a real-time gesture image, and performing preprocessing on the real-time gesture image based on the real-time light intensity to obtain a preprocessed real-time image; performing annotation processing on the preprocessed real-time image based on the annotation mapping table to obtain an annotated real-time image; inputting the annotated real-time image into the lightweight recognition model to obtain a gesture recognition result, and adjusting the working state of the refrigerator according to the gesture recognition result. 2.The gesture-based refrigerator interaction method of claim 1, wherein, The method comprises the following steps: constructing a training sample set, wherein the training sample set comprises a normal sample set and an abnormal sample set, and the normal sample set comprises multiple normal gesture images with normal light and multiple normal gesture images in a dark environment, and the abnormal sample set comprises multiple abnormal gesture images with normal light and multiple abnormal gesture images in a dark environment; performing weak light gray scale correction processing on the normal gesture images in a dark environment and the abnormal gesture images in a dark environment in the training sample set by using a histogram equalization method to obtain a corrected sample set; performing gray scale conversion processing on the corrected sample set by using a weighted gray scale algorithm, and performing binarization processing on the training sample set after the gray scale conversion processing by using a self-adaptive threshold segmentation algorithm based on an image gray scale histogram to obtain a binarized sample set; performing denoising processing on the binarized sample set by using a Gaussian filtering method, and performing repair processing on the binarized sample after the denoising processing by using a morphological repair algorithm to obtain a repaired sample set; performing feature extraction processing on the repaired sample set by using a three-dimensional convolutional neural network to obtain a preprocessed sample set comprising feature vectors. 3.The gesture-based refrigerator interaction method of claim 2, wherein, The method comprises the following steps: performing annotation processing on the preprocessed sample set to obtain an annotated sample set, and the annotated labels comprise gesture categories, abnormal types, and environmental light levels; constructing an annotation mapping table by combining the feature vectors of the preprocessed sample set and the labels of the annotated sample set, wherein the annotation mapping table is stored in a key-value pair structure, the key is a hash value of the feature vector, and the value is a label set comprising gesture category labels, abnormal type labels, and environmental light level labels. 4.The gesture-based refrigerator interaction method of claim 1, wherein, The method comprises the following steps: A training model is constructed, which includes an input layer, a three-dimensional convolutional layer, a CBAM attention layer, an anomaly feature enhancement layer, and an output layer connected in sequence. The CBAM attention layer includes a channel attention sub-layer and a spatial attention sub-layer. The anomaly feature enhancement layer adopts a residual connection structure to enhance the gradient propagation of anomaly gesture features. The labeled sample set is divided according to a preset division ratio to obtain a training set and a test set; Based on the training and test sets, the Adam optimizer and cosine annealing algorithm, combined with the cross-entropy loss function and the ambient light loss function, are used to iteratively train the model to be trained, thereby obtaining the gesture recognition model. 5.The gesture-based refrigerator interaction method of claim 1, wherein, The step of lightweighting the gesture recognition model to obtain a lightweight recognition model and deploying it includes: The gesture recognition model is pruned using the L1 regularized channel pruning algorithm to obtain a compressed recognition model. The compressed recognition model is quantized by using an asymmetric quantization method to quantize the 32-bit floating-point parameters into 8-bit integer parameters, thus obtaining the quantized recognition model. The quantized recognition model is distilled using interlayer parameters based on knowledge distillation to obtain a lightweight recognition model; Convert the lightweight recognition model to the TFLite Micro format supported by the refrigerator's main controller and deploy it. 6.The gesture-based refrigerator interaction method of claim 1, wherein, The refrigerator is equipped with an infrared sensor and a light sensor, and the infrared sensor has an infrared supplementary light function; The process of acquiring real-time illumination intensity and real-time gesture images, and preprocessing the real-time gesture images based on the real-time illumination intensity to obtain preprocessed real-time images includes: Obtain the real-time light intensity fed back by the light sensor and compare the real-time light intensity with the preset compensation activation threshold; If the real-time light intensity is less than the preset supplementary light activation threshold, the infrared sensor is controlled to perform infrared supplementary light and the real-time gesture image fed back by the infrared sensor is acquired. The real-time gesture image is then subjected to weak light grayscale correction processing, grayscale conversion processing, binarization processing, noise reduction processing, repair processing and feature extraction processing in sequence to obtain a preprocessed gesture image including real-time feature vectors. If the real-time illumination intensity is greater than or equal to the preset supplementary lighting threshold, the real-time gesture image fed back by the infrared sensor is acquired. The real-time gesture image is then subjected to grayscale conversion, binarization, denoising, repair, and feature extraction in sequence to obtain a preprocessed gesture image including real-time feature vectors. 7.The gesture-based refrigerator interaction method of claim 1, wherein, The step of inputting the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjusting the working state of the refrigerator according to the gesture recognition result, includes: The labeled real-time image is input into the lightweight recognition model to obtain gesture recognition results, which include gesture category, anomaly type, ambient light adaptability, and fit. Obtain a preset similarity threshold, and compare the fitting degree with the preset similarity threshold; If the fitting degree is greater than or equal to the preset similarity threshold, the working state of the refrigerator is adjusted based on the gesture recognition result; If the fit is less than a preset similarity threshold, a gesture adjustment command is generated. 8.A gesture-based refrigerator interaction apparatus, characterized by, include: The construction module is used to construct a training sample set and perform preprocessing to obtain a preprocessed sample set, wherein the training sample set includes a normal sample set and an abnormal sample set. The first annotation module is used to annotate the preprocessed sample set to obtain an annotated sample set, and to construct an annotation mapping table based on the annotated sample set; The training module is used to construct a model to be trained using a three-dimensional convolutional network combined with an attention mechanism, and to iteratively train the model to be trained based on the labeled sample set to obtain a gesture recognition model. The lightweight module is used to perform lightweight processing on the gesture recognition model to obtain a lightweight recognition model and deploy it. The processing module is used to acquire real-time light intensity and real-time gesture images, and preprocess the real-time gesture images based on the real-time light intensity to obtain preprocessed real-time images. The second annotation module is used to perform annotation processing on the preprocessed real-time image based on the annotation mapping table to obtain an annotated real-time image; The recognition module is used to input the labeled real-time image into the lightweight recognition model to obtain the gesture recognition result, and adjust the working state of the refrigerator according to the gesture recognition result. 9.A gesture-based refrigerator interaction device, characterized by, The gesture-based refrigerator interaction device includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the gesture-based refrigerator interaction device to perform the steps of the gesture-based refrigerator interaction method as described in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon instructions, the instructions comprising, When the instructions are executed by the processor, they implement the steps of the gesture-based refrigerator interaction method as described in any one of claims 1-7.

Images