Intelligent automobile gesture recognition system and recognition method thereof

Abstract

The application relates to the field of artificial intelligence, and in particular to an intelligent automobile gesture recognition system and a recognition method thereof, which comprises an image acquisition module, an infrared sensor is used to capture infrared light; a computing module provides computing power and data processing capability; a hand detection model uses boundary information provided by the image acquisition module to obtain key point information through the computing module, and the hand detection model is detected from top to bottom, and the specific steps are as follows: step 1, collecting a photo: the image acquisition module is used to collect a photo, and the photo is pretreated; step 2, an onnxruntime inference framework is as follows: step 3, mapping a control instruction, the application realizes true man-machine interaction through an artificial intelligence technology, a Qt framework is used on the software, and a visual man-machine interaction interface with high interaction is designed. Gesture behavior recognition functions of collected pictures are realized through an onnxruntime inference architecture and an OpenCV vision library, and the efficiency and stability of system operation are ensured.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to an intelligent vehicle gesture recognition system and its recognition method. Background Technology

[0002] Today's cars are no longer just simple means of transportation, but more like highly integrated intelligent mobile spaces. From the continuous upgrading of intelligent driving assistance systems, with the gradual popularization of functions such as adaptive cruise control and automatic parking, making driving easier and safer, to the increasing intelligence of in-vehicle infotainment systems and the disappearance of traditional buttons, the intelligent in-vehicle infotainment systems appear somewhat redundant and complex in terms of the distribution of function buttons, and the operability is greatly reduced compared to the buttons of traditional in-vehicle infotainment systems.

[0003] Therefore, improving the human-machine interaction function of in-vehicle systems is of paramount importance, and with the market share of smart cockpits gradually increasing, developing an in-vehicle system that can realize corresponding functions by judging the driver's hand gestures is of utmost importance. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes an intelligent vehicle gesture recognition system and its recognition method.

[0005] A smart car gesture recognition system includes: The image acquisition module captures infrared light through an infrared sensor; The computing module provides computing power and data processing capabilities; The hand detection model utilizes boundary information provided by the image acquisition module and obtains key point information through the calculation module, performing top-down detection.

[0006] Furthermore, the backbone network, as a basic module, extracts features from the input image through multi-layer convolution operations.

[0007] Furthermore, the neck network undertakes the core function of feature fusion, optimizing and integrating the features from different levels output by the backbone network.

[0008] Furthermore, the head network, based on the multi-scale feature map output by the neck network, enables the localization and classification of targets in the image.

[0009] A recognition method for a smart car gesture recognition system, the specific steps of which are as follows: Step 1: Acquire photos: Use the image acquisition module to acquire photos and preprocess them; Step 2, the ONNX runtime inference framework is as follows: Step 21: Load preprocessed images: After preprocessing, the images are input into the loaded model in ONNXRuntime for inference. The images first pass through the first hand detection model to obtain an inference result, which will output two multi-dimensional arrays; the first array stores the bounding box information, and the second array stores the category ID and confidence score. Step 22, Hand Detection Model: Using the boundary provided by the detector, and then using the provided boundary information, key point information is obtained through the key point detection model; Step 23: Obtain the bounding box image: Crop the image region containing the obtained bounding box; Step 24: Perform image preprocessing and input it into the hand keypoint model to obtain two two-dimensional array outputs. The first array stores the confidence scores of points along the x-axis, where the maximum index of its dimension is the x-axis coordinate of the keypoint. The second array stores the confidence scores of points along the y-axis, where the maximum index of its dimension is the y-axis coordinate of the keypoint. After obtaining the keypoint information, input the keypoint information into the classification model to obtain the classification result. Step 25, Key Point Detection Model: The algorithm uses the rtmpose model and adopts CSPNEXt as the backbone network to obtain the heat map of key points of the hand; Step 26: Extract key points: Extract n key points using the backbone network, independently classify them along the x and y axes, and generate the final prediction results. Coordinate classification implementation:

[0010] W: Enter the width of the image; H: Height of the input image; : These represent the quantities along the x-axis and y-axis, respectively; The x and y coordinates of keypoints are represented by two independent one-dimensional vectors. Using a scaling factor (k ≥ 1), a one-dimensional vector larger than the image's side length is obtained. For the p-th keypoint, its encoded coordinates are represented as: (2-14); The scaling factor enhances positioning accuracy to a level smaller than a single pixel; Coordinate decoding: (2-15); (2-16); Step 27, Classification Model: Mediapipe hand landmarks are used for auxiliary annotation of the dataset. This algorithm generates 8 different hand shape images. The keypoints are then flipped 90° to ensure that the flipped hand shape remains at 90° regardless of the hand's orientation. The flipping formula is as follows: ; X, Y: represent the coordinates before rotation; θ: represent the rotation angle; X1, Y1: Represent the coordinates after rotation; Keypoint images are generated using a keypoint detection network based on ROI images. Step 28: Obtain hand shape results: Complete the dimensionality reduction and classification tasks through fully connected layers and softmax layers, and obtain the probability distribution information of different gesture categories to obtain hand shape results; Step 3: Map control instructions.

[0011] The beneficial effects of this invention are as follows: This invention achieves true human-computer interaction through artificial intelligence technology. The software utilizes the Qt framework to design a highly interactive and visual human-computer interface. The onnxruntime inference architecture and OpenCV vision library are used to implement gesture recognition functionality for acquired images, ensuring the system's high efficiency and stability during operation. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0013] Figure 1 This is a top-down detection flowchart of the present invention; Figure 2 This is a heat map of the key hand points of the present invention; Figure 3 This is a table of CNN-LSTM network parameters for this invention; Figure 4 This is a diagram illustrating the hand types of the present invention; Figure 5 This is a schematic diagram of the process structure of the present invention; Figure 6 This is a graph showing the change in learning rate according to the present invention; Figure 7 This is a diagram showing the training results of the key points of this invention; Figure 8 This is a diagram showing the training results of the detection model of the present invention; Figure 9 This is a diagram showing the training results of the classification model of the present invention. Detailed Implementation

[0014] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below.

[0015] like Figures 1 to 9 As shown, an intelligent vehicle gesture recognition system includes: The image acquisition module captures infrared light through an infrared sensor, enabling it to clearly capture image details even in harsh low-light environments. The computing module provides powerful computing power and efficient data processing capabilities for the intelligent vehicle gesture recognition system, which can easily handle various complex computing tasks and ensure the smooth and stable operation of the intelligent vehicle gesture recognition system. The hand detection model utilizes boundary information provided by the image acquisition module and obtains key point information through the calculation module, performing top-down detection.

[0016] The image acquisition module includes an industrial-grade USB infrared (IR) camera, which ensures the detail and quality of the generated images and can capture clear and accurate image information to provide strong support for the model's reasoning.

[0017] The computing module utilizes the Jeston Orin Nano development board, which features 8GB of onboard RAM running on a 128-bit bus, providing a high bandwidth of 102GB / s to ensure fast data read / write and processing. Furthermore, it boasts 256GB of storage space and supports both solid-state drive (SSD) storage and microSD card booting, meeting diverse storage and booting needs across various scenarios. In terms of expandability, it features a rich array of high-speed interfaces, enabling rapid connection to various high-performance expansion devices.

[0018] The described hand detection model comprises a backbone network, a neck network, and a head network, enabling the localization and classification of targets in images. Each module has a clear division of labor and works in a progressive manner, collectively ensuring the network's efficient and accurate detection performance.

[0019] The backbone network, as a basic module, extracts features from the input image through multi-layer convolution operations.

[0020] The neck network undertakes the core function of feature fusion, and optimizes and integrates the features of different levels output by the backbone network through a feature pyramid structure.

[0021] The head network, based on the multi-scale feature map output by the neck network, performs the key task of target detection, accurately predicts the bounding box coordinates and category information of the target, and realizes the localization and classification of the target in the image.

[0022] A recognition method for a smart car gesture recognition system, the specific steps of which are as follows: Step 1: Acquire photos: Use the image acquisition module to acquire photos and preprocess them; Step 2, the ONNX runtime inference framework is as follows: Step 21: Load preprocessed images: After preprocessing, the images are input into the loaded model in ONNXRuntime for inference. The images first pass through the first hand detection model to obtain an inference result, which will output two multi-dimensional arrays; the first array stores the bounding box information, and the second array stores the category ID and confidence score. Step 22, Hand Detection Model: The accuracy of the hand detection model determines the accuracy of the hand key points. The hand key point model used in this system is a top-down detection method. The so-called top-down means using the boundary provided by the detector, and then using the provided boundary information to obtain key point information through the key point detection model. The top-down detection flowchart is shown in Figure 1.

[0023] Therefore, choosing a good hand detection algorithm is particularly important. In order to obtain accurate boundary information, the system chose the mmdetection algorithm as the hand detection algorithm. mmdetection is a PyTorch-based object detection toolbox jointly developed by SenseTime and the Chinese University of Hong Kong. Its biggest feature is that it supports customized module insertion, which can customize modules as needed. At the same time, it also supports mixed precision training, which reduces the memory usage during training. Step 23: Obtain the bounding box image: Crop the image region containing the obtained bounding box; Step 24: Perform image preprocessing and input it into the hand keypoint model to obtain two two-dimensional array outputs. The first array stores the confidence scores of points in the x-axis direction, where the maximum index of its dimension is the x-axis coordinate of the keypoint. The second array stores the confidence scores of points in the y-axis direction, where the maximum index of its dimension is the y-axis coordinate of the keypoint. After obtaining the keypoint information, the keypoint information is input into a classification model that uses a classification network with CNN+LSTM to implement hand shape classification to obtain the classification result. Step 25, Keypoint Detection Model: The algorithm uses the rtmpose model and employs CSPNEXt as the backbone network, such as... Figure 2 As shown, the heat at the key points of the hand is obtained; Step 26: Extract key points: Extract n key points using the backbone network, independently classify them along the x and y axes, and generate the final prediction results. Coordinate classification implementation: ; W: Enter the width of the image; H: Height of the input image; : These represent the quantities along the x-axis and y-axis, respectively; The x and y coordinates of keypoints are represented by two independent one-dimensional vectors. Using a scaling factor (k ≥ 1), a one-dimensional vector larger than the image's side length is obtained. For the p-th keypoint, its encoded coordinates are represented as: (2-14); The scaling factor enhances positioning accuracy to a level smaller than a single pixel; Coordinate decoding: (2-15); (2-16); Step 27, Classification Model: The classification model uses a CNN-LSTM image classification model. The prepared dataset, consisting of images of eight different hand gestures, is divided into eight folders, each named according to its classification type. This is trained for 300 epochs. The algorithm generates eight different hand shape images, and a 90° flip is applied to the keypoints to ensure that the flipped hand shape remains at 90° regardless of the hand's orientation. The flipping formula is as follows: ; X, Y: represent the coordinates before rotation; θ: represent the rotation angle; X1, Y1: Represent the coordinates after rotation; Keypoint images are generated using a keypoint detection network based on ROI images. Step 28: Obtain hand shape results: Complete the dimensionality reduction and classification tasks through fully connected layers and softmax layers, and obtain the probability distribution information of different gesture categories to obtain hand shape results; Step 3: Map control instructions.

[0024] This invention achieves true human-computer interaction through artificial intelligence technology. The software uses the Qt framework and designs a highly interactive and visual human-computer interaction interface.

[0025] This invention implements gesture recognition functionality for acquired images using the ONNX runtime inference architecture and the OpenCV vision library, ensuring the system's high efficiency and stability during operation.

[0026] In step S27, with 300 rounds, a combination of multiple learning rate schedulers is used. In the initial few iterations of training, a linear strategy is used to increase the learning rate from a small value to a normal value, and then a different adjustment strategy is used for normal training. Based on this learning rate warm-up optimization strategy, by dynamically adjusting the learning rate in the early stages of training, the model can be guided to explore the solution space with a more reasonable parameter update rhythm. This enhances training robustness and significantly improves model performance. The learning rate changes as follows... Figure 6 As shown in the table below, the learning rate settings are as follows: ; The training set loss gradually decreases with increasing iterations, and the training set accuracy also increases with increasing iterations, approaching 1. Normalized translation error (NME) is a core evaluation metric in keypoint detection. It quantifies the deviation between the model's predictions and the true values ​​by calculating the average Euclidean distance between predicted and ground truth keypoints and standardizing it based on a specific normalization scale. NME evaluates the accuracy of facial keypoint detection by focusing on the relative distances between hand feature points; a lower NME indicates more accurate keypoint predictions. PCK is also a keypoint evaluation metric, representing the percentage of correctly predicted keypoints. It is used to calculate the percentage of correctly predicted keypoints. Figure 7 As shown, with the increase of the number of iterations, the training accuracy and PCK gradually increase, while NME and loss gradually decrease, proving that the training model has a very good fitting effect. NME evaluation metric formula: ; x i This indicates the predicted coordinates of the i-th element; x * i The true value of the i-th coordinate; d is the distance; PCK evaluation index formula: ; T k For threshold; d pi Let Euclidean distance be the Euclidean distance between the predicted value and the actual value of the i-th keypoint for the p-th person; d def p Let be the normalization factor for the p-th individual.

[0027] Analysis of Hand Detection Model Training Results: Loss and Accuracy Evaluation Based on RTMDET Backbone Network Training Figure 8As shown, with training, `loss`, `loss_cls`, and `loss_bbox` all gradually decrease and tend towards 0. `loss_cls` is the classification loss function, used to calculate the difference between the model's predicted class label and the true class label. In object detection tasks, the model needs to predict the class of each object. By comparing the difference between the predicted result and the true label, `loss_cls` helps the model optimize parameters to improve classification accuracy; the lower the `loss_cls`, the more accurate the model's prediction. `loss_bbox` is the bounding box regression loss function, used to calculate the difference between the model's predicted bounding box and the true bounding box. In object detection, in addition to predicting the object's class, the model also needs to predict the object's location. `loss_bbox` helps the model optimize parameters to improve localization accuracy by comparing the difference between the predicted boundary and the true boundary. `bbox_mAP` also increases with the number of training epochs and gradually tends towards 1. `bbox_mAP` refers to the mean evaluation accuracy of bounding boxes in object detection tasks; it comprehensively measures the model's performance in both object location detection and classification. AP5 represents the bounding box mAP (bbox_mAP) metric when the Iou threshold is 0.5. APS represents the bounding box mAP measurement for bounding boxes with pixel areas less than 322 (small scale). APM represents the bounding box mAP measurement for bounding boxes with pixel areas less than 322-962 (medium scale). APS represents the bounding box mAP measurement for bounding boxes with pixel areas less than 962 (large scale). Figure 8 As shown, the coco / bbox_mAP metrics are both close to 1, indicating that the model training effect is good.

[0028] Analysis of Hand Shape Classification Model Training Results: As the number of training iterations increases, the training loss remains near 0, and the validation set value gradually decreases to approximately 7-7.25 and then stabilizes. The graph shows that the values ​​of metrics / accuracy_top1 are between 0.9 and 1. metrics / accuracy_top1 indicates that the classification with the highest probability among the predicted results is correct, and metrics / accuracy_top5 indicates that any of the top five highest probability vectors is considered a correct prediction. The above data demonstrates that the model has a good fit during training. The training results are as follows... Figure 9 As shown.

[0029] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely prisms of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent vehicle gesture recognition system, characterized in that: include: The image acquisition module captures infrared light through an infrared sensor; The computing module provides computing power and data processing capabilities; The hand detection model utilizes boundary information provided by the image acquisition module and obtains key point information through the calculation module, performing top-down detection.

2. The intelligent vehicle gesture recognition system according to claim 1, characterized in that: The image acquisition module includes an industrial-grade USB infrared (IR) camera.

3. The intelligent vehicle gesture recognition system according to claim 1, characterized in that: The computing module uses the Jeston Orin Nano development board.

4. The intelligent vehicle gesture recognition system according to claim 1, characterized in that: The hand detection model includes a backbone network, a neck network, and a head network.

5. The intelligent vehicle gesture recognition system according to claim 4, characterized in that: The backbone network, as a basic module, extracts features from the input image through multi-layer convolution operations.

6. The intelligent vehicle gesture recognition system according to claim 4, characterized in that: The neck network undertakes the core function of feature fusion, optimizing and integrating features from different levels output by the backbone network.

7. The intelligent vehicle gesture recognition system according to claim 4, characterized in that: The head network, based on the multi-scale feature map output by the neck network, enables the localization and classification of targets in the image.

8. A recognition method using an intelligent vehicle gesture recognition system according to any one of claims 1 to 7, characterized in that: The specific steps are as follows: Step 1: Acquire photos: Use the image acquisition module to acquire photos and preprocess them; Step 2, the ONNX runtime inference framework is as follows: Step 21: Load preprocessed images: After preprocessing, the images are input into the loaded model in the ONNX runtime for inference. The images first pass through the first hand detection model to obtain an inference result, which will output two multi-dimensional arrays; the first array stores the bounding box information, and the second array stores the category ID and confidence score. Step 22, Hand Detection Model: Using the boundary provided by the detector, and then using the provided boundary information, key point information is obtained through the key point detection model; Step 23: Obtain the bounding box image: Crop the image region containing the obtained bounding box; Step 24: Perform image preprocessing and input it into the hand keypoint model to obtain two two-dimensional array outputs. The first array stores the confidence scores of points along the x-axis, where the maximum index of its dimension is the x-axis coordinate of the keypoint. The second array stores the confidence scores of points along the y-axis, where the maximum index of its dimension is the y-axis coordinate of the keypoint. After obtaining the keypoint information, input the keypoint information into the classification model to obtain the classification result. Step 25, Key Point Detection Model: The algorithm uses the rtmpose model and adopts CSPNEXt as the backbone network to obtain the heat map of key points of the hand; Step 26: Extract key points: Extract n key points using the backbone network, independently classify them along the x and y axes, and generate the final prediction results. Coordinate classification implementation: ， W: Enter the width of the image; H: Height of the input image; : These represent the quantities along the x-axis and y-axis, respectively; The x and y coordinates of keypoints are represented by two independent one-dimensional vectors. Using a scaling factor (k ≥ 1), a one-dimensional vector larger than the image's side length is obtained. For the p-th keypoint, its encoded coordinates are represented as: (2-14)， The scaling factor enhances positioning accuracy to a level smaller than a single pixel. Coordinate decoding: （2-15）， （2-16）， Step 27, Classification Model: Mediapipe hand landmarks are used for auxiliary annotation of the dataset. This algorithm generates 8 different hand shape images. The keypoints are then flipped 90° to ensure that the flipped hand shape remains at 90° regardless of the hand's orientation. The flipping formula is as follows: ； X, Y: Represent the coordinates before rotation; θ: Represents the rotation angle; X1, y1: Represent the coordinates after rotation; Keypoint images are generated using ROI images via a keypoint detection network. Step 28: Obtain hand shape results: Complete the dimensionality reduction and classification tasks through fully connected layers and Softmax layers, and obtain the probability distribution information of different gesture categories to obtain hand shape results; Step 3: Map control instructions.

Images