Gesture recognition system fusing SLAM (Simultaneous Localization and Mapping) data

Abstract

The invention provides an SLAM data fused gesture recognition system, which comprises an image acquisition module, a gesture recognition module, an SLAM data auxiliary module and a data processing unit, and is characterized in that the image acquisition module is used for acquiring a gesture image of a user; the gesture recognition module performs preliminary recognition on gestures based on a deep learning model; the SLAM data auxiliary module provides three-dimensional space positioning and mapping information of the surrounding environment of the equipment in real time and is used for correcting and enhancing the accuracy of gesture recognition; the data processing unit is responsible for integrating and processing data of all the modules and outputting a final gesture recognition result, and the system has the advantages that better user interaction experience is provided for virtual maintenance equipment, and the system is suitable for gesture recognition application scenes in various complex environments.

Description

Technical Field

[0001] The invention relates to a gesture recognition system integrating SLAM data, and in particular to the technical field of virtual maintenance. Background Art

[0002] During the development and testing of the virtual maintenance equipment software, it was found that there are high requirements for the real-time and accuracy of the gesture recognition system, as follows: 1. Real-time requirements: During the virtual maintenance process, target recognition needs to respond quickly to meet the needs of real-time operations. The response speed from making an action to displaying the corresponding content is tested: Normal environment - the time taken is determined to be 99.9ms; 2. Accuracy requirements: The recognition results must be highly accurate to avoid misoperation or misleading users.

[0003] The existing gesture recognition development based on Yolov8 Pose version has a high misrecognition rate in low-light environment and complex background, and there is a significant delay in the recognition of dynamic gestures. This not only affects the user's real-time interactive experience, but also makes the application unavailable in some scenarios. In addition, due to the limitation of the computing power of the device, the real-time performance of the gesture recognition algorithm cannot be fully guaranteed.

[0004] Through in-depth analysis of the gesture recognition module, the problems are mainly concentrated in the following aspects: 1. Influence of image noise: In low-light environments, the noise in the image increases significantly, making it difficult for the Yolov8 Pose model to accurately extract gesture features, thus affecting recognition accuracy.

[0005] 2. Dynamic gesture processing: When the Yolov8 Pose model processes dynamic gestures, insufficient frame rate and loss of timing information are the main reasons for recognition delay. Limited by the hardware computing power of the device, the model performs poorly in real-time processing of multiple frames of continuous images, resulting in significant delays in the recognition of dynamic gestures.

[0006] The main reasons for the problem are: image noise under low-light conditions makes gesture features blurred in the image, and the Yolov8 Pose model is prone to errors when extracting gesture key points. In addition, dynamic gesture recognition relies on the continuity and accuracy of inter-frame information. However, due to the limited computing power of hardware, the model cannot process a large number of consecutive frames in a short period of time, resulting in loss of timing information and processing delays, which in turn affects the real-time and accuracy of recognition. Summary of the invention

[0007] The technical problem to be solved by the present invention is a gesture recognition system integrating SLAM data.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: A gesture recognition system integrating SLAM data includes the following modules: An image acquisition module, used to obtain a user's gesture image; The gesture recognition module performs preliminary recognition of gestures based on a deep learning model; SLAM data auxiliary module, which provides real-time 3D spatial positioning and mapping information of the device's surrounding environment, used to correct and enhance the accuracy of gesture recognition; The data processing unit is responsible for integrating and processing the data from each module and outputting the final gesture recognition results.

[0009] Furthermore, the image acquisition module adds noise suppression and image enhancement technology in image preprocessing, uses median filtering to suppress image noise and removes random noise caused by insufficient lens illumination. By replacing each pixel with the median of its neighborhood, noise is effectively removed while maintaining edge information. By introducing contrast enhancement and edge sharpening processing, gesture features are better highlighted in the image.

[0010] Furthermore, the gesture recognition module retrains the Yolov8 Pose model to improve the accuracy of gesture recognition in low light and complex backgrounds, and designs and prepares a diverse training dataset.

[0011] Further, the following are several key points of dataset preparation: Multiple lighting conditions: The dataset contains different lighting conditions from low light to high light, simulating lighting changes in different environments to ensure that the model can cope with the impact of lighting changes; Complex background: In addition to standard backgrounds, the dataset covers different complex backgrounds, such as indoor and outdoor environments, different objects and background interference, to improve the robustness of the model in complex scenes; Gesture diversity: The annotations in the dataset cover a variety of gestures, including different gestures, hand positions, and movement speeds, ensuring that the model can accurately recognize different gestures; High-quality annotations: Each image in the dataset is accurately annotated to ensure that the key points of the gestures (such as finger joints, palms, and fingertips) are fully supervised during training.

[0012] Furthermore, after the data set is prepared, the Yolov8 Pose model is trained. The training process can be divided into the following steps: Supervised learning: By using a well-labeled data set, the Yolov8 Pose model is trained using supervised learning. The model parameters are continuously adjusted to minimize the error between the predicted results and the actual labels, and the recognition accuracy of the model is gradually improved. Multiple rounds of training and optimization: Multiple rounds of iterations are performed during the training process. Each round adjusts and optimizes the model. The optimization is not only for recognition accuracy, but also involves the following aspects: Hyperparameter adjustment: including adjustment of learning rate, batch size, and training cycle hyperparameters to ensure that the model can converge in a shorter time and achieve optimal performance; Data enhancement: Improve the generalization ability of the model by enhancing the data (such as rotation, flipping, and brightness adjustment), especially the adaptability to low light and complex backgrounds; Noise processing and robustness improvement: A variety of noise processing methods are used, such as image denoising and filtering technology, to enhance the model's recognition ability in low signal-to-noise ratio environments.

[0013] Furthermore, SLAM data assists recognition: SLAM (Simultaneous Localization and Mapping) data is introduced as an aid. SLAM technology can provide three-dimensional spatial positioning and mapping information of the device's surrounding environment in real time. The spatial data is used to correct and enhance the recognition results of the Yolov8 Pose model.

[0014] Furthermore, the data processing unit lightweighted the Yolov8 Pose model, reduced the model's computational complexity and memory usage through pruning, quantization, and model compression techniques, and packaged the lightweight model into a module that can be integrated with Unity through Python development.

[0015] Furthermore, the pruning is to reduce the model size and the amount of calculation by removing unimportant weights or neurons in the neural network. The pruning strategy is to prune the model layer by layer to remove neurons or connections that have little impact on the final output.

[0016] Furthermore, quantization is the process of converting the floating-point weights in the model into low-precision integer representations, thereby reducing the model's memory usage and computational requirements. The quantization method is: using fixed-point quantization, the model's weights and activation values ​​are converted from floating-point numbers to lower-precision integers.

[0017] Furthermore, model compression optimizes performance by reducing the size and computational complexity of the model. Model compression methods include parameter sharing and low-rank decomposition. Parameter sharing: By sharing some parameters in the model, the total number of parameters in the model can be reduced, thereby reducing the storage requirements of the model; Low-rank decomposition: Use matrix decomposition technology to perform low-rank decomposition on the weights of the convolutional layer, thereby reducing the amount of computation required by the model.

[0018] The beneficial effects of adopting the above technical solution are: The present invention significantly improves the accuracy and stability of gesture recognition by introducing SLAM data as auxiliary information, especially in low-light and dynamic scenes. At the same time, the use of image processing enhancement technology and model lightweight processing enables the gesture recognition algorithm to run efficiently on virtual maintenance equipment with limited computing power. The invention provides a better user interaction experience for virtual maintenance equipment and is suitable for gesture recognition application scenarios in various complex environments.

[0019] The present invention has significantly improved the accuracy and real-time performance of gesture recognition through a series of optimization measures, especially the introduction of retraining and SLAM data-assisted recognition. The improved system can maintain efficient gesture recognition performance under complex lighting and background conditions, and with the assistance of SLAM technology, the stability and accuracy of the model in dynamic scenes are further enhanced. These improvements provide a better user interaction experience for virtual maintenance equipment, allowing the equipment to stably and quickly perform gesture recognition in various complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Selecting a gesture diagram in the present invention; Figure 2 This is a schematic diagram of the confirmation gesture in the present invention; Figure 3 This is a schematic diagram of the menu gesture in the present invention; DETAILED DESCRIPTION

[0021] The present invention provides a gesture recognition system integrating SLAM data, which comprises the following modules: An image acquisition module, used to obtain a user's gesture image; The gesture recognition module performs preliminary recognition of gestures based on a deep learning model; SLAM data auxiliary module, which provides real-time 3D spatial positioning and mapping information of the device's surrounding environment, used to correct and enhance the accuracy of gesture recognition; The data processing unit is responsible for integrating and processing the data from each module and outputting the final gesture recognition results.

[0022] Image acquisition module, in order to solve the problem of gesture recognition in low-light environments, we added noise suppression and image enhancement technology in image preprocessing. Using median filtering to suppress image noise is particularly effective in removing random noise caused by insufficient light in the lens. This method effectively removes noise while maintaining edge information by replacing each pixel with the median value in its neighborhood. In addition, by introducing contrast enhancement and edge sharpening, gesture features are better highlighted in the image. The application of these image processing techniques has significantly improved the recognition accuracy of the Yolov8 Pose model in low-light environments.

[0023] Gesture Recognition Module,To improve the accuracy of gesture recognition, especially in low light and complex backgrounds, we retrained the Yolov8 Pose model.,In order to make the Yolov8 Pose model adaptable to a variety of complex scenarios, it is necessary to design and prepare a diverse training data set.

[0024] The following are several key points in the preparation of the dataset: Multiple lighting conditions: The dataset contains different lighting conditions from low light to high light, simulating the lighting changes in different environments to ensure that the model can cope with the impact of light changes. Complex background: In addition to the standard background, the dataset also covers different complex backgrounds, such as indoor and outdoor environments, different objects and background interference, etc., to improve the robustness of the model in complex scenes. Gesture diversity: The annotations of the dataset cover a variety of gestures, including different gestures, hand postures, and movement speeds, etc., to ensure that the model can accurately recognize different gestures. High-quality annotations: Each image in the dataset is accurately annotated to ensure that the key points of the gesture (such as finger joints, palms, fingertips, etc.) are fully supervised during training.

[0025] After the data set is prepared, the Yolov8 Pose model is trained. The training process can be divided into the following steps:Supervised learning: The Yolov8 Pose model is trained in a supervised learning manner using a well-labeled data set. This method gradually improves the recognition accuracy of the model by continuously adjusting the model parameters to minimize the error between the predicted results and the true annotations.Multiple rounds of training and optimization: Multiple rounds of iterations are carried out during the training process, and the model is adjusted and optimized in each round. Optimization is not only for recognition accuracy, but also involves the following aspects:Hyperparameter adjustment: Including the adjustment of hyperparameters such as learning rate, batch size, and training cycle to ensure that the model can converge in a shorter time and obtain the best performance.Data enhancement: By enhancing the data (such as rotation, flipping, brightness adjustment, etc.), the generalization ability of the model is improved, especially the adaptability to low light and complex backgrounds.Noise processing and robustness improvement: In order to improve the performance in noisy environments, we use a variety of noise processing methods, such as image denoising, filtering and other technologies to enhance the recognition ability of the model in low signal-to-noise ratio environments.

[0026] SLAM data-assisted recognition: In order to further improve the accuracy and stability of gesture recognition, we introduced SLAM (Simultaneous Localization and Mapping) data as an aid. SLAM technology can provide three-dimensional spatial positioning and mapping information of the device's surrounding environment in real time. These spatial data are used to correct and enhance the recognition results of the Yolov8 Pose model. For example, SLAM data helps the model filter background noise and misidentification signals, and correct gesture posture recognition errors through spatial position judgment. This combination significantly improves the robustness of gesture recognition, especially in dynamic and complex environments. The assistance of SLAM data effectively reduces recognition errors and enhances recognition accuracy.

[0027] Data processing unit, in order to ensure that the gesture recognition algorithm can run efficiently on an operating system with limited computing power, we lightweighted the Yolov8 Pose model. Through pruning, quantization, and model compression, the model's computational workload and memory usage were reduced. Finally, Python was used to package the lightweight model into a module that can be integrated with Unity, ensuring that it can run smoothly on virtual maintenance equipment and achieve efficient gesture recognition results.

[0028] The pruning is a technique that reduces the size of the model and the amount of computation by removing unimportant weights or neurons in the neural network. Through pruning, the computational efficiency of the model is significantly improved. Pruning strategy: We pruned the model layer by layer, removing neurons or connections that have little impact on the final output. For example, neurons with weight values ​​close to zero or with small contributions are removed to reduce the amount of computation and memory usage. Performance after pruning: After pruning, the computational complexity of the model is greatly reduced, but it still maintains good recognition accuracy, especially the inference speed on devices with low computing power has been significantly improved.

[0029] Quantization is the process of converting the floating-point weights in a model into low-precision integer representations, thereby reducing the model's memory footprint and computational requirements. Quantization technology is particularly suitable for running deep learning models on embedded devices or low-power hardware. Quantization method: We use fixed-point quantization (such as INT8 quantization) to convert the model's weights and activation values ​​from floating-point numbers (32 bits) to lower-precision integers (such as 8 bits). This greatly reduces the storage space of the model and improves the inference speed, especially on devices whose hardware supports integer operations. Post-quantization performance: After quantization, the model's memory footprint is greatly reduced, while the inference speed is significantly accelerated. On low-power devices, the quantized model can achieve higher execution efficiency without significantly losing accuracy.

[0030] Model compression is a technique that optimizes performance by reducing the size and computational complexity of the model. Commonly used model compression methods include parameter sharing, low-rank decomposition, etc. Parameter sharing: By sharing some parameters in the model, the total number of model parameters is reduced, thereby reducing the storage requirements of the model. Low-rank decomposition: Use matrix decomposition technology, especially low-rank decomposition of convolutional layer weights, to reduce the computational complexity of the model. Performance after compression: The size of the model is greatly reduced after compression, which is suitable for memory-constrained devices. After compression, it can still maintain high recognition accuracy and inference speed, especially in resource-constrained environments, to ensure the efficiency of real-time gesture recognition.

[0031] The present invention significantly improves the accuracy and stability of gesture recognition by introducing SLAM data as auxiliary information, especially in low-light and dynamic scenes. At the same time, the use of image processing enhancement technology and model lightweight processing enables the gesture recognition algorithm to run efficiently on virtual maintenance equipment with limited computing power. The invention provides a better user interaction experience for virtual maintenance equipment and is suitable for gesture recognition application scenarios in various complex environments.

[0032] Specific application examples: The virtual maintenance equipment gesture recognition system sets three gestures for recognition in low light and complex background environments. The three gestures are: "Select", "Confirm" and "Menu", which are defined as follows: Choose a gesture: Make a fist with only the index finger extended, with the palm facing outwards, and avoid covering the index finger, such as Figure 1 As shown, the response situation is that the gesture interaction dotted line appears in the software; Confirmation gesture: pinch the index finger and thumb together, and make a fist with the rest of the fingers, with the palm facing outwards, avoiding covering the area where the index finger and thumb are pinched, such as Figure 2 As shown, the response situation is that the software appears a solid line of gesture interaction; Menu gesture: Stretch out your five fingers with your palms facing the camera. Figure 3 As shown, the response is a pop-up menu panel.

[0033] The measured results are as follows: Gesture Recognition Number of test samples 18 success 16 fail 2 Total response time / Average response time / in conclusion Gesture recognition accuracy rate 88.89% The present invention conducts gesture recognition tests in low light and complex background environments. The improved model has improved recognition accuracy. In the recognition of dynamic gestures, the delay time is also greatly reduced, which significantly improves the user's real-time interactive experience. In addition, through practical application tests, the gesture recognition module can maintain a high recognition stability under different lighting conditions, and the user experience has been greatly improved. The experimental results show that the new algorithm not only improves the recognition accuracy, but also greatly enhances the real-time processing capability of the system.

[0034] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition system integrating SLAM data, characterized in that: It includes the following modules: An image acquisition module, used to obtain a user's gesture image; The gesture recognition module performs preliminary recognition of gestures based on a deep learning model; SLAM data auxiliary module, which provides real-time 3D spatial positioning and mapping information of the device's surrounding environment, used to correct and enhance the accuracy of gesture recognition; The data processing unit is responsible for integrating and processing the data from each module and outputting the final gesture recognition results.

2. A gesture recognition system integrating SLAM data according to claim 1, characterized in that: The image acquisition module adds noise suppression and image enhancement technology in image preprocessing. It uses median filtering to suppress image noise and remove random noise caused by insufficient lens illumination. By replacing each pixel with the median value in its neighborhood, it effectively removes noise while maintaining edge information. By introducing contrast enhancement and edge sharpening processing, gesture features are better highlighted in the image.

3. A gesture recognition system integrating SLAM data according to claim 1, characterized in that: The gesture recognition module retrains the Yolov8 Pose model to improve the accuracy of gesture recognition in low light and complex backgrounds, and designs and prepares a diverse training dataset.

4. A gesture recognition system integrating SLAM data according to claim 3, characterized in that: The following are some key points for dataset preparation: Multiple lighting conditions: The dataset contains different lighting conditions from low light to high light, simulating lighting changes in different environments to ensure that the model can cope with the impact of light changes; Complex background: In addition to standard backgrounds, the dataset covers different complex backgrounds, such as indoor and outdoor environments, different objects and background interference, to improve the robustness of the model in complex scenes; Gesture diversity: The annotations in the dataset cover a variety of gestures, including different gestures, hand positions, and movement speeds, ensuring that the model can accurately recognize different gestures; High-quality annotations: Each image in the dataset is accurately annotated to ensure that the key points of the gestures (such as finger joints, palms, and fingertips) are fully supervised during training.

5. A gesture recognition system integrating SLAM data according to claim 4, characterized in that: After the data set is prepared, the Yolov8 Pose model is trained. The training process can be divided into the following steps: Supervised learning: By using a well-labeled data set, the Yolov8 Pose model is trained using supervised learning. The model parameters are continuously adjusted to minimize the error between the predicted results and the actual labels, and the recognition accuracy of the model is gradually improved. Multiple rounds of training and optimization: Multiple rounds of iterations are performed during the training process. Each round adjusts and optimizes the model. The optimization is not only for recognition accuracy, but also involves the following aspects: Hyperparameter adjustment: including adjustment of learning rate, batch size, and training cycle hyperparameters to ensure that the model can converge in a shorter time and achieve optimal performance; Data enhancement: Improve the generalization ability of the model by enhancing the data (such as rotation, flipping, and brightness adjustment), especially its adaptability to low light and complex backgrounds; Noise processing and robustness improvement: A variety of noise processing methods are used, such as image denoising and filtering technology, to enhance the model's recognition ability in low signal-to-noise ratio environments.

6. The gesture recognition system integrating SLAM data according to claim 1, characterized in that: SLAM data-assisted recognition: SLAM (Simultaneous Localization and Mapping) data is introduced as an aid. SLAM technology can provide three-dimensional spatial positioning and mapping information of the device's surrounding environment in real time. Spatial data is used to correct and enhance the recognition results of the Yolov8 Pose model.

7. The gesture recognition system integrating SLAM data according to claim 1, characterized in that: The data processing unit performs lightweight processing on the Yolov8 Pose model. Through pruning, quantization, and model compression technologies, the model's computational complexity and memory usage are reduced. Through Python development, the lightweight model is packaged into a module that can be integrated with Unity.

8. A gesture recognition system integrating SLAM data according to claim 7, characterized in that: The pruning is to reduce the model size and the amount of calculation by removing unimportant weights or neurons in the neural network. The pruning strategy is to prune the model layer by layer to remove neurons or connections that have little impact on the final output.

9. The gesture recognition system integrating SLAM data according to claim 7, characterized in that: Quantization is the process of converting the floating-point weights in the model into low-precision integer representations, thereby reducing the model's memory usage and computing requirements. The quantization method is: using fixed-point quantization to convert the model's weights and activation values ​​from floating-point numbers to lower-precision integers.

10. The gesture recognition system integrating SLAM data according to claim 7, characterized in that: Model compression is to optimize performance by reducing the size and computational complexity of the model. Model compression methods include parameter sharing and low-rank decomposition. Parameter sharing: By sharing some parameters in the model, the total number of parameters in the model can be reduced, thereby reducing the storage requirements of the model; Low-rank decomposition: Use matrix decomposition technology to perform low-rank decomposition on the weights of the convolutional layer, thereby reducing the amount of computation required by the model.

Images