Gesture fusion recognition method and device, electronic equipment and storage medium

Abstract

The invention belongs to the technical field of gesture detection, and particularly relates to a gesture fusion recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: determining RTM and DTM according to millimeter wave radar data; recognizing the RTM and the DTM by adopting a double-flow neural network RDCL to obtain a radar gesture recognition result; determining a hand key point for the infrared data by adopting a Mediape library, and determining a static gesture according to the angle, the relative position and the coordinate of the hand key point; determining a dynamic gesture according to the movement direction, the movement track and the rotation angle of the hand key point, and obtaining a visual gesture recognition result according to the static gesture and the dynamic gesture; according to the radar gesture recognition result and the visual gesture recognition result, a weight decision rule is adopted for fusion recognition, and a gesture fusion recognition result is obtained. The method has the beneficial effects that the gesture recognition accuracy is improved, and the system resource consumption during gesture recognition is reduced.

Description

Technical Field

[0001] This invention relates to the field of gesture detection technology, and in particular to a gesture fusion recognition method, device, electronic device, and storage medium. Background Technology

[0002] Existing technologies for gesture fusion typically employ millimeter-wave radar gesture recognition methods using dual-stream deep fusion, which generally suffer from the following drawbacks: (1) If the data acquisition conditions of the two millimeter-wave radars are highly similar (such as the same platform and similar frequencies), the information they provide will have a high degree of overlap and it will be difficult to form an effective complementarity. (2) Dual-stream fusion models usually require complex neural network architectures (such as dual-branch CNN + fusion module), which consume a lot of system resources and have high training costs; (3) It is unable to handle gesture detection in complex scenarios because millimeter waves are prone to producing Doppler blur or sparse point clouds for fast-moving targets. Summary of the Invention

[0003] In order to at least solve one of the technical problems existing in the prior art, the present invention provides a gesture fusion recognition method, device, electronic device and storage medium.

[0004] One aspect of the present invention provides a gesture fusion recognition method, comprising: Acquire infrared and millimeter-wave radar data of gestures, and determine RTM and DTM based on the millimeter-wave radar data; The radar gesture recognition results are obtained by using a dual-stream neural network RDCL to identify RTM and DTM. The infrared data is used to determine key hand points using the Mediapipe library. Based on the angle, relative position, and coordinates of the key hand points, a static gesture is determined. Dynamic gestures are determined based on the direction of movement, trajectory of movement, and rotation angle of key points on the hand. Visual gesture recognition results are obtained based on static and dynamic gestures. The radar gesture recognition results and the visual gesture recognition results are fused and recognized using a weighted decision rule to obtain the gesture fusion recognition result.

[0005] According to the gesture fusion recognition method, acquiring infrared data and millimeter-wave radar data of gestures, and determining RTM and DTM based on the millimeter-wave radar data, includes: The echo signal and reference signal of the millimeter-wave radar data are mixed to obtain an intermediate frequency signal. The intermediate frequency signal is then subjected to a fast Fourier transform to obtain the distance information of each pulse. The distances of each pulse are then spliced ​​together to obtain the RTM. The pulse peak position data of the RTM is extracted, and the pulse peak position data is converted into a data stream through time series analysis. The data stream is then subjected to time-frequency analysis to obtain the Doppler information of the gesture. The DTM is determined based on the Doppler information.

[0006] According to the gesture fusion and recognition method, the radar gesture recognition result is obtained by using a dual-stream neural network RDCL to recognize RTM and DTM, including: The upper branch of the dual-stream neural network RDCL is used to process the RTM to obtain the RTM classification result, and the lower branch of the dual-stream neural network RDCL is used to process the DTM to obtain the DTM classification result. The RTM classification result and the DTM classification result are then spliced ​​spatially and temporally to obtain the radar gesture recognition result.

[0007] According to the gesture fusion and recognition method, determining static gestures based on the angles, relative positions, and coordinates of key hand points includes: Based on the relative positions of the four key hand points, the angle between two key hand points is determined, and then the absolute value of the angle difference between the two key hand points is calculated. Based on the absolute value of the difference and the absolute value, the degree of bending of a single finger is determined. The bending index of a single finger is determined based on the degree of bending of the finger, thereby obtaining the static gesture.

[0008] According to the gesture fusion and recognition method, dynamic gestures are determined based on the movement direction, trajectory, and rotation angle of key hand points. Based on static and dynamic gestures, visual gesture recognition results are obtained, including: Determine the coordinates of the palm position based on key points of the hand; The forward and backward displacement directions are determined based on the palm position coordinates and time series. The degree of palm position coordinate displacement is determined based on the displacement direction and displacement threshold. The degree of palm position coordinate displacement includes the movement direction, movement trajectory and rotation angle of the X and Y coordinates. Based on the degree of displacement of the palm position coordinates and the dynamic gesture template, the dynamic gesture and its type are determined. The types of dynamic gestures include one of the following: directional gestures, path gestures, and rotation gestures.

[0009] According to the gesture fusion recognition method, the gesture fusion recognition result is obtained by using a weighted decision rule based on the radar gesture recognition result and the visual gesture recognition result, including: RadarGesture weights are applied to radar gesture recognition results, and cameraGesture weights are applied to visual gesture recognition results. Then, weight decision rules are used for fusion recognition to obtain gesture fusion recognition results.

[0010] According to the gesture fusion and recognition method described above, the method further includes: Adjust the radarGesture weight and cameraGesture weight based on the gesture type and sampling lighting conditions of the gesture image from the visual gesture recognition results.

[0011] Another aspect of the present invention provides a gesture fusion recognition device, comprising: The first module is used to acquire infrared data and millimeter-wave radar data of gestures, and to determine RTM and DTM based on the millimeter-wave radar data. The second module is used to identify RTM and DTM using a dual-stream neural network RDCL to obtain radar gesture recognition results; The third module is used to determine key hand points using the Mediapipe library from the infrared data, and to determine static gestures based on the angles, relative positions, and coordinates of the key hand points. The fourth module is used to determine dynamic gestures based on the direction of movement, trajectory of movement and rotation angle of key hand points, and to obtain visual gesture recognition results based on static and dynamic gestures. The fifth module is used to perform fusion recognition based on the radar gesture recognition results and the visual gesture recognition results using weighted decision rules, so as to obtain the gesture fusion recognition result.

[0012] Another aspect of the present invention provides an electronic device, including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method as described above.

[0013] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the methods described above.

[0014] The beneficial effects of this invention are as follows: the fusion of millimeter-wave radar and infrared sensors complement each other through heterogeneous sensing advantages, which not only improves the accuracy and robustness of key point detection; but also reduces the negative impact of environmental changes on the accuracy of gesture recognition by detecting gesture recognition features through decision-level fusion; and for stable lighting environments, the weight adjustment results can be cached until changes in ambient light are detected, and the weights can be adjusted in advance based on historical data to predict the trend of light changes and reduce the need for real-time calculation. Attached Figure Description

[0015] Figure 1This is a schematic diagram of the gesture fusion and recognition process according to an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the gesture signal echo according to an embodiment of the present invention.

[0017] Figure 3 This is a flowchart of Doppler information extraction according to an embodiment of the present invention.

[0018] Figure 4 This is a diagram of the RDCL network structure according to an embodiment of the present invention.

[0019] Figure 5 This is a schematic diagram of key hand points according to an embodiment of the present invention.

[0020] Figure 6 This is a flowchart of gesture recognition according to an embodiment of the present invention.

[0021] Figure 7 This is a flowchart of static gesture recognition according to an embodiment of the present invention.

[0022] Figure 8 This is a flowchart of the directional gesture recognition process according to an embodiment of the present invention.

[0023] Figure 9 This is the path / rotation gesture recognition process according to an embodiment of the present invention. Figure 10 This is a weighted decision rule diagram of the gesture results in an embodiment of the present invention.

[0024] Figure 11 This is a diagram illustrating the radar gesture recognition process according to an embodiment of the present invention. (a) shows an upward wave, with PyCharm outputting "up"; (b) shows a downward wave, with PyCharm outputting "down"; (c) shows a leftward wave, with PyCharm outputting "left"; (d) shows a rightward wave, with PyCharm outputting "right"; (e) shows a counter-clockwise rotation, with PyCharm outputting "ccw"; (f) shows a clockwise rotation, with PyCharm outputting "cw"; (g) shows a trajectory S, with PyCharm outputting "s"; (h) shows a trajectory Z, with PyCharm outputting "z"; and (i) shows a trajectory X, with PyCharm outputting "x".

[0025] Figure 12 This is a diagram of the static gesture recognition process of the camera according to an embodiment of the present invention, wherein (a) to (e) are the recognition process diagrams of gestures “1”, “2”, “3”, “4” and “5” respectively.

[0026] Figure 13This is a dynamic gesture recognition process of a camera according to an embodiment of the present invention, wherein (a) to (i) represent the dynamic gesture recognition process of waving upward, waving downward, waving to the left, waving to the right, rotating counterclockwise, rotating clockwise, drawing Z, drawing X and drawing Z in sequence.

[0027] Figure 14 This is a comparison diagram of network models in embodiments of the present invention.

[0028] Figure 15 This is a schematic diagram of a gesture fusion recognition device according to an embodiment of the present invention. Detailed Implementation

[0029] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0030] refer to Figure 1 , Figure 1 This is a schematic diagram of the gesture fusion and recognition process according to an embodiment of the present invention, which includes, but is not limited to, steps S100 to S500: S100 acquires infrared and millimeter-wave radar data of gestures, and determines RTM and DTM based on the millimeter-wave radar data.

[0031] RTM stands for distance-time map; DTM stands for Doppler-time map.

[0032] In some embodiments, the echo signal and reference signal of millimeter-wave radar data are mixed to obtain an intermediate frequency signal. A fast Fourier transform is performed on the intermediate frequency signal to obtain the distance information of each pulse. The distances of each pulse are spliced ​​together to obtain the RTM. The pulse peak position data of the RTM is extracted, and the pulse peak position data is converted into a data stream through time series analysis. The data stream is subjected to time-frequency analysis to obtain the Doppler information of the gesture. The DTM is determined based on the Doppler information.

[0033] As shown in Figure 2, which illustrates the extraction of gesture signal echoes, each row represents a single chirp signal at different distance units, and each column represents the echo signals of different chirps (signals whose carrier frequency changes linearly over the pulse duration) at the same distance unit. The horizontal axis represents the fast time axis, and the vertical axis represents the slow time axis. For each frame of data in a single channel, there are 128 chirp signals, each of which includes 256 sampling points. The size of the reconstructed matrix is ​​128×256.

[0034] Based on the distance measurement principle, the distance information of a gesture is contained in the frequency domain data of the IF signal. The specific operation procedure for extracting the distance information of a gesture is as follows: Figure 2 As shown. Therefore, a fast-time FFT is first performed on each pulse signal to obtain the distance information corresponding to each pulse, and then the information is spliced ​​along the slow time to obtain the distance change information of the gesture.

[0035] like Figure 3 The flowchart shown below illustrates the extraction of Doppler information. Figure 2 The original signal distance information in the illustrated embodiment contains pulse peak position data, where each peak represents the distance information of the object at different time points. These pulse peak position data are rearranged to form a data stream, converting the time-series distance data into a continuous data stream. Then, time-frequency analysis is performed on this rearranged data stream to process and analyze the frequency components of the data within different time windows, thereby obtaining the Doppler information of the gesture.

[0036] S200 uses a dual-stream neural network RDCL to identify RTM and DTM, and obtains radar gesture recognition results.

[0037] In some embodiments, the RTM is processed by the upper branch of the dual-stream neural network RDCL to obtain the RTM classification result, and the DTM is processed by the lower branch of the dual-stream neural network RDCL to obtain the DTM classification result. The RTM classification result and the DTM classification result are then spliced ​​spatially and temporally to obtain the radar gesture recognition result.

[0038] like Figure 4The RDCL network structure diagram shown illustrates that traditional radar recognition methods struggle to significantly improve accuracy. While the trend towards combining deep learning often employs CNNs or RNNs alone for feature extraction, this approach fails to balance spatial resolution and temporal dependencies. Therefore, addressing the issue of information loss in single-scale convolutions, and to enhance accuracy while maintaining real-time performance, the RDCL dual-stream neural network is proposed to improve the recognition of easily confused gestures. The network consists of two distinct branches, with input data from RTM and DTM, respectively. These branches undergo different processing steps and are then concatenated to achieve spatial-temporal fusion. This embodiment of the invention combines the spatial information processing capabilities of CNNs with the temporal information processing capabilities of LSTMs, using an attention mechanism to enhance information. Gesture discrimination is then performed after concatenation at a fully connected layer.

[0039] In some embodiments, an LSTM consists of an input gate, an output gate, a forget gate, candidate memory cells, memory cell states, and hidden states. The forget gate determines the cell state at the previous time step. The formula for calculating the amount of information forgotten is: ; in, It is the weight matrix of the forget gate. It is the bias term of the forgetting gate. It is the hidden state of the previous time step. This is the input for the current time step; It is the sigmoid activation function, which outputs a value between 0 and 1. If the output is close to 0, the information is forgotten; if it is close to 1, the information is retained.

[0040] The input gate determines the amount of new information to be memorized at the current time step, and the formula is as follows:

[0041] in, It is a weight matrix. This is a bias term. Calculate candidate memory units: ; in, It is the weight matrix of the candidate units. It is a bias term. Set the output range to -1 to 1 to ensure the information is not too large.

[0042] The formula for updating the cell state is as follows: ; The core is the cell state It combines the state of the previous time step. and new memory units The former formula retains some information from past memories, while the latter adds new memory information. It can store important information for a long time while gradually adding new information, thus solving the problem. The gradient vanishing problem.

[0043] The output gate determines the hidden state at the current time step. :

[0044] in, It is the weight matrix of the output gate. It is a bias term. Control the outflow of information. Finally, calculate the hidden state using the formula:

[0045] in, Ensure the state is between -1 and 1. Control which information needs to be output.

[0046] The mechanism is a channel attention mechanism. Unlike traditional methods that treat each channel with equal weight, this method dynamically adjusts the importance of each channel, enhancing key channels and suppressing irrelevant ones. The calculation formula is as follows: First, calculate the global feature mean across all channels to extract global information:

[0047] in, and These are the height and width of the feature map. These are the channel pixel values. The channel feature values ​​are input into two fully connected layers to calculate the channel weights:

[0048] in, It is a dimension reduction matrix, which can reduce the amount of computation; It's an upgraded matrix that recovers the number of channels. yes Activation function yes Activation function. Multiplying the channel weights by the original feature map yields a new feature map:

[0049] In this embodiment of the invention, the network adds to the upper and lower branch networks respectively. The mechanism calculates channel attention weights through global average pooling, enabling the model to better focus on the most important features.

[0050] In some embodiments, The upper support network primarily reflects the changes in gestures in the distance-time plane, requiring a focus on extracting local spatial information. Therefore, the upper support network... After a Extracting local spatial features, after a Dimensionality reduction is performed to decrease computational load and improve translation invariance; after another... Extracting deeper features, after a Further dimensionality reduction is performed to improve computational efficiency; then, after one... Extracting global spatial features enhances the ability to distinguish different gestures. After a... Further reduce dimensionality to improve generalization ability; then... Attention mechanisms enhance key information, ultimately The extracted spatial features are compressed to 128 dimensions.

[0051] The input process to the lower branch network primarily reflects the dynamic information of gestures in the Doppler-temporal plane, requiring a focus on extracting temporal features. Therefore, the lower branch network... Two layers were used The first one passed It can learn the speed characteristics of gestures, and Set as Output the complete time series; then pass through another layer Continue processing time series data to learn dependencies over longer time spans. Set as ,let Output the hidden state at the last time step. Then use another... Convert the format (128, ) to a (1×1×128) compatible format and input it into the input field. ,let Focus on the most useful time information; then a Convert the format to a 128-dimensional vector for use by the fully connected layer; finally The extracted time series features are further integrated to output a final 128-dimensional time series feature vector.

[0052] Processed Feature maps and Feature maps in Layer fusion features, with a dimension of 256; after Layers reduce dimensionality, improve computational efficiency, and enable non-linear mapping, making it easier for classifiers to distinguish different categories of gestures. They also allow spatial and temporal information to interact, generating better feature representations. prevent and Overfitting Classify, and finally in The gesture classification results of millimeter-wave radar were obtained.

[0053] S300 uses infrared data The library identifies key hand points and determines static gestures based on the angles, relative positions, and coordinates of these key hand points.

[0054] In some embodiments, the angle between two hand key points is determined based on the relative positions of the four hand key points, and then the absolute value of the angle difference between the two hand key points is calculated. Based on the absolute value of the difference and the absolute value, the degree of bending of a single finger is determined; the number of bent fingers is determined based on the degree of bending of a single finger, and thus a static gesture is obtained.

[0055] like Figure 5 The key points shown are 0-20, representing the palm point and the joints of various gestures. Recognizing these hand joints allows for the identification of both static and dynamic gestures, primarily by analyzing the accumulated joint angles and movement trajectories to determine the state of the fingers.

[0056] In some embodiments, such as Figure 6 The flowchart shown illustrates the gesture recognition process. Static gestures are determined by analyzing the size of the angle and the relative position, while dynamic gestures are recognized by analyzing the initial movement and the motion trajectory and directional changes of consecutive frames.

[0057] like Figure 7 The flowchart shown illustrates the static gesture recognition process. For static gestures, the coordinates of the joints are identified to determine whether a single finger is bent. Then, the gesture is determined based on the degree of bending of the five fingers and the relative positions of the four key points of each finger. For example, to identify the degree of bending of the thumb, the angle 'a' between points 4 and 3 is calculated first, then the angle 'b' between points 2 and 1 is calculated. Finally, the absolute value of the difference between angles 'a' and 'b' is calculated. If the absolute value is less than 12 degrees, the thumb is considered straight. Gestures are distinguished by whether each finger is straight. For example, gesture 1 is indicated by the angles between points 8 and 7 being less than 12 degrees from the angles between points 6 and 5 (meaning the index finger is straight), and the angle differences of the other fingers being greater than 12 degrees (meaning the other fingers are bent). In this case, the gesture is identified as gesture 1.

[0058] S400 determines dynamic gestures based on the movement direction, trajectory, and rotation angle of key hand points, and obtains visual gesture recognition results based on static and dynamic gestures.

[0059] In some embodiments, the hand position coordinates are determined based on key hand points; the forward and backward displacement direction is determined based on the hand position coordinates and time series; the degree of hand position coordinate displacement is determined based on the displacement direction and displacement threshold, wherein the degree of hand position coordinate displacement is the movement direction, movement trajectory and rotation angle of the X and Y coordinates; and the dynamic gesture and its type are determined based on the degree of hand position coordinate displacement and dynamic gesture template, wherein the type of dynamic gesture includes one of directional gesture, path gesture and rotation gesture.

[0060] refer to Figure 8 and Figure 9 . refer to Figure 8 Dynamic gesture recognition is categorized into three types: directional gestures, path gestures, and rotation gestures. Directional gestures include up, down, left, and right movements; path gestures include X, Z, and S movements; and rotation gestures include clockwise and counterclockwise rotations. Gestures are inferred by analyzing the direction, trajectory, and rotation angle changes of key hand points.

[0061] For directional gestures such as up, down, left, and right, the direction of movement is determined by the movement of the palm positions (point 0, point 1, point 5, point 9, point 13, point 17) based on their original positions. When at least three points simultaneously undergo displacement in the same direction that exceeds a displacement threshold, it is determined that directional movement is occurring. The displacement calculation method is as follows:

[0062]

[0063] An upward gesture increases the Y-coordinate of the key point simultaneously; a downward gesture decreases the Y-coordinate simultaneously; a leftward gesture decreases the X-coordinate simultaneously; and a rightward gesture increases the X-coordinate simultaneously. A threshold is used to determine if the gesture meets the standard. The origin of the coordinate system is set to the top left corner, with the right side of the X-axis positive and the bottom of the Y-axis positive. Therefore, when... and When the gesture is in time, it is interpreted as a downward gesture. Displacement threshold; and When, it is judged as a downward gesture; when and When, it is judged as a leftward gesture; when and If so, it is judged as a rightward gesture.

[0064] It's understandable that the rotation gesture is a derivative of the path-based circular gesture; therefore, prioritizing the recognition of path-based gestures hinges on analyzing the finger's movement trajectory. The recognition process for path / rotation gestures is as follows: Figure 9 As shown, the process includes first setting up a list `points_queue` to store keypoint frames at the tip of the index finger. Upon detecting a static action '1', the coordinates are enqueued. The `points_queue` is analyzed every 10-15 frames, while gesture templates for X, Z, S, and O are defined. The origin of the coordinate system is the upper left corner of the viewport. After obtaining the coordinates of the fingertip, the coordinate position is recorded every two frames until the gesture ends or approaches a threshold. After obtaining the trajectory, dynamic time warping is used for template matching. By tracking the movement trajectory of the keypoints at the tip of the index finger, it can be determined whether it conforms to the defined shape.

[0065] For rotation gestures, the procedure is to first present a static action '1', then draw an 'O' shape. After path gesture recognition determines the gesture to be an 'O' shape, the direction of movement at the starting point of the keypoint at the tip of the index finger is calculated to determine the rotation direction. The initial keyframes of the list are analyzed, and then the three adjacent points of the finger coordinates are sequentially taken after N frames. Calculate the vector from the previous point to the next point respectively. and Then use the atan2 function to calculate the rotation angle between the two vectors, the formula is:

[0066]

[0067]

[0068]

[0069]

[0070] in, It calculates the vector from the previous point to the current point. The vector from the current point to the next point is calculated by... and The formula can calculate A positive value indicates counter-clockwise rotation, while a negative value indicates clockwise rotation. By recognizing static gestures and three types of dynamic gestures—directional gestures, path gestures, and rotation gestures—the camera's gesture recognition output can be obtained.

[0071] The S500 uses weighted decision rules to perform fusion recognition based on radar gesture recognition results and visual gesture recognition results to obtain gesture fusion recognition results.

[0072] In some embodiments, reference Figure 10 The weighted decision rule graph for gesture results uses `radarGesture` weights for radar gesture recognition and `cameraGesture` weights for visual gesture recognition. These weighted decision rules are then used for fusion recognition to obtain the final gesture fusion recognition result. For example, let `radarGesture` be the weight for radar gesture results and `cameraGesture` be the weight for camera gesture results, both set to 0.5. For static gestures, experiments have shown that millimeter-wave radar has limited recognition capabilities for static gestures; setting `cameraGesture` to 1 and `radarGesture` to 0, directly using the camera's output, is more effective.

[0073] In some embodiments, the radarGesture weight and cameraGesture weight are adjusted based on the gesture type from the visual gesture recognition result and the sampling lighting conditions of the gesture image. For example, if it is a dynamic gesture, the brightness value of the image is first analyzed to determine the lighting conditions of the current environment. If it is a low-light environment, the radar weight is increased, setting cameraGesture to 0.7 and radarGesture to 0.3; if the lighting is good, the default weights are maintained, i.e., cameraGesture is set to 0.5 and radarGesture to 0.5.

[0074] Unlike common weighted and voting decision-making methods, the decision-making method designed in this section can re-evaluate and adjust weights when ambient light changes. It automatically adjusts weights based on historical accuracy feedback to adapt to different environments and usage conditions. Furthermore, it employs caching technology. For stable lighting environments, it can cache weight adjustment results until a change in ambient light is detected. Based on historical data, it predicts light change trends and adjusts weights in advance, reducing the need for real-time computation.

[0075] For example, Figure 11 This is a diagram illustrating the radar gesture recognition process according to an embodiment of the present invention. (a) shows an upward wave, with PyCharm outputting "up"; (b) shows a downward wave, with PyCharm outputting "down"; (c) shows a leftward wave, with PyCharm outputting "left"; (d) shows a rightward wave, with PyCharm outputting "right"; (e) shows a counter-clockwise rotation, with PyCharm outputting "ccw"; (f) shows a clockwise rotation, with PyCharm outputting "cw"; (g) shows a trajectory S, with PyCharm outputting "s"; (h) shows a trajectory Z, with PyCharm outputting "z"; and (i) shows a trajectory X, with PyCharm outputting "x".

[0076] Conduct actual tests on these 9 gestures, find 6 testers of different ages and genders, and test each gesture 100 times, and the statistics are shown in Table 1.

[0077] Table 1 Radar Gesture Recognition Results

[0078] In Table 1, correct recognition is considered qualified, and incorrect recognition is considered unqualified. The results show that on the self-built gesture dataset, the average recognition accuracy of radar-based gesture recognition reaches 95.0%. Among them, the recognition accuracy of direction gestures is 95.5%, the recognition accuracy of path gestures is 94.7%, and the recognition accuracy of rotation gestures is 94.5%.

[0079] Reference Figure 12 , Figure 12 is the process diagram of the camera static gesture recognition in the embodiment of the present invention. Among them, (a) to (e) are the recognition result diagrams of gestures '1', '2', '3', '4', and '5' in sequence. The gesture recognition of the camera is divided into static and three types of dynamic gesture recognition, namely the static gestures of '1', '2', '3', '4', and '5', and the up, down, left, and right gestures, the draw X, Z, S gestures, and the clockwise and counterclockwise rotation gestures. The experimental environment remains unchanged, and the method of only the hand entering the frame is used to display the visual output results. The purple dots on the video hand represent the recognized key points, and the green lines are connected to form a gesture model, and the recognition result is output in the upper left corner, and the output data is below. The specific determination method is to first judge the straight and bent states of the five fingers, with straight being 1 and bent being 0. Therefore, the finger bending degree of gesture '1' is 0-1-0-0-0, the finger bending degree of gesture '2' is 0-1-1-0-0, the finger bending degree of gesture '3' is 0-1-1-1-0, the finger bending degree of gesture '4' is 0-1-1-1-1, and the finger bending degree of gesture '5' is 1-1-1-1-1.

[0080] Reference Figure 13The camera's dynamic gesture recognition process is shown in Figure 2. Figure 2 shows the recognition result of waving upwards, outputting the recognition result "Swipe Up"; Figure 2 shows the recognition result of waving downwards, outputting the recognition result "Swipe Down"; Figure 2 shows the recognition result of waving to the left, outputting the recognition result "Swipe Left"; Figure 2 shows the recognition result of waving to the right, outputting the recognition result "Swipe Right"; Figure 2 shows the recognition result of rotating counterclockwise, outputting the recognition result "Spin CCW"; Figure 2 shows the recognition result of rotating clockwise, outputting the recognition result "Spin CW"; Figure 2 shows the recognition result of drawing a 'Z', outputting the recognition result "Letter Z"; Figure 2 shows the recognition result of drawing an 'X', outputting the recognition result "Letter X"; Figure 2 shows the recognition result of drawing an 'S', outputting the recognition result "Letter S". The actual recognition effect of these gestures was tested, with each gesture tested 100 times. The statistical results are shown in Table 2.

[0081] Table 2 Camera gesture recognition results

[0082] Table 2 shows that the recognition accuracy rate for static gestures is 97.8%, and the recognition accuracy rate for dynamic gestures is 92.9%, with directional gestures achieving an accuracy rate of 93.3%, path gestures 93.0%, and rotation gestures 92.0%. Experimental results indicate that the average recognition accuracy rate of camera-based gesture recognition on the self-built gesture dataset reaches 94.6%.

[0083] Experiments were conducted on a self-made dataset for different types of network models, such as CNN, LSTM, and Transformer.

[0084] refer to Figure 14 The section describes the accuracy obtained by replacing the network in this chapter with different network models, and classifies the response time levels according to the response speed. The X-axis represents the model used, the Y-axis represents the accuracy, and the style of each bar represents the response time level: a horizontal bar indicates a fast response time, a vertical bar indicates a medium response time, a dotted bar indicates a medium-slow response time, and a diagonal bar indicates a slow response time.

[0085] Experimental results show that the accuracy of CNN alone is only 83.5%, while LSTM improves the accuracy to 87.1%. Using Transformer improves the accuracy but increases computational complexity and slows response time. The model combining CNN and LSTM achieves an accuracy of 90.3%, although the response time is slightly longer. To enhance the model's feature extraction capability, the SE mechanism is used to optimize the combination of CNN and LSTM. Overall, on the self-made dataset, the proposed two-stream neural network RDCL achieves the highest accuracy of 96.4% while maintaining a certain response speed.

[0086] Figure 15 This is a schematic diagram of a gesture fusion recognition device according to an embodiment of the present invention. The device includes a first module 1510, a second module 1520, a third module 1530, a fourth module 1540, and a fifth module 1550.

[0087] The system comprises five modules: a first module for acquiring infrared and millimeter-wave radar data of gestures, and determining RTM and DTM based on the millimeter-wave radar data; a second module for recognizing RTM and DTM using a dual-stream neural network (RDCL) to obtain radar gesture recognition results; a third module for determining hand key points using the Mediapipe library on the infrared data, and determining static gestures based on the angles, relative positions, and coordinates of the hand key points; a fourth module for determining dynamic gestures based on the movement direction, trajectory, and rotation angle of the hand key points, and obtaining visual gesture recognition results based on static and dynamic gestures; and a fifth module for fusing the radar gesture recognition results and visual gesture recognition results using weighted decision rules to obtain gesture fusion recognition results.

[0088] Exemplarily, with the cooperation of the first, second, third, fourth, and fifth modules in the device, the embodiment device can implement any of the aforementioned gesture fusion recognition methods, namely, acquiring infrared data and millimeter-wave radar data of gestures; determining RTM and DTM based on the millimeter-wave radar data; recognizing the RTM and DTM using a dual-stream neural network RDCL to obtain radar gesture recognition results; determining hand key points using the Mediapipe library on the infrared data; determining static gestures based on the angle, relative position, and coordinates of the hand key points; determining dynamic gestures based on the movement direction, trajectory, and rotation angle of the hand key points; obtaining visual gesture recognition results based on static and dynamic gestures; and performing fusion recognition using weighted decision rules based on the radar gesture recognition results and visual gesture recognition results to obtain gesture fusion recognition results. The beneficial effects of this invention are as follows: the fusion of millimeter-wave radar and infrared sensors complement each other through heterogeneous sensing advantages, which not only improves the accuracy and robustness of key point detection; but also reduces the negative impact of environmental changes on the accuracy of gesture recognition by detecting gesture recognition features through decision-level fusion; and for stable lighting environments, the weight adjustment results can be cached until changes in ambient light are detected, and the weights can be adjusted in advance based on historical data to predict the trend of light changes and reduce the need for real-time calculation.

[0089] This invention also provides an electronic device, which includes a processor and a memory; The memory stores the program; The processor executes a program to perform the aforementioned gesture fusion recognition method; the electronic device has the function of carrying and running the gesture fusion recognition software system provided in the embodiments of the present invention, such as a personal computer, minicomputer, mainframe, workstation, network or distributed computing environment, standalone or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0090] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the gesture fusion and recognition method described above.

[0091] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented in the embodiments of this invention. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0092] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned gesture fusion recognition method.

[0093] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, considering the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed in the embodiments of the invention, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0094] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0096] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0097] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0098] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0099] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0100] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A gesture fusion and recognition method, characterized in that, include: Acquire infrared and millimeter-wave radar data of gestures, and determine RTM and DTM based on the millimeter-wave radar data; The radar gesture recognition results are obtained by using a dual-stream neural network RDCL to identify RTM and DTM. The infrared data is used to determine key hand points using the Mediapipe library. Based on the angle, relative position, and coordinates of the key hand points, a static gesture is determined. Dynamic gestures are determined based on the direction of movement, trajectory of movement, and rotation angle of key points on the hand. Visual gesture recognition results are obtained based on static and dynamic gestures. The radar gesture recognition results and the visual gesture recognition results are fused and recognized using a weighted decision rule to obtain the gesture fusion recognition result.

2. The gesture fusion and recognition method according to claim 1, characterized in that, The acquisition of infrared and millimeter-wave radar data for gestures, and the determination of RTM and DTM based on the millimeter-wave radar data, include: The echo signal and reference signal of the millimeter-wave radar data are mixed to obtain an intermediate frequency signal. The intermediate frequency signal is then subjected to a fast Fourier transform to obtain the distance information of each pulse. The distances of each pulse are then spliced ​​together to obtain the RTM. The pulse peak position data of the RTM is extracted, and the pulse peak position data is converted into a data stream through time series analysis. The data stream is then subjected to time-frequency analysis to obtain the Doppler information of the gesture. The DTM is determined based on the Doppler information.

3. The gesture fusion and recognition method according to claim 2, characterized in that, The process of using a dual-stream neural network (RDCL) to identify RTM and DTM to obtain radar gesture recognition results includes: The upper branch of the dual-stream neural network RDCL is used to process the RTM to obtain the RTM classification result, and the lower branch of the dual-stream neural network RDCL is used to process the DTM to obtain the DTM classification result. The RTM classification result and the DTM classification result are then spliced ​​spatially and temporally to obtain the radar gesture recognition result.

4. The gesture fusion and recognition method according to claim 1, characterized in that, Determining static gestures based on the angles, relative positions, and coordinates of key hand points includes: Based on the relative positions of the four key hand points, the angle between two key hand points is determined, and then the absolute value of the angle difference between the two key hand points is calculated. Based on the absolute value of the difference and the absolute value, the degree of bending of a single finger is determined. The bending index of a single finger is determined based on the degree of bending of the finger, thereby obtaining the static gesture.

5. The gesture fusion and recognition method according to claim 1, characterized in that, The process of determining dynamic gestures based on the movement direction, trajectory, and rotation angle of key hand points, and obtaining visual gesture recognition results based on static and dynamic gestures, includes: Determine the coordinates of the palm position based on key points of the hand; The forward and backward displacement directions are determined based on the palm position coordinates and time series. The degree of palm position coordinate displacement is determined based on the displacement direction and displacement threshold. The degree of palm position coordinate displacement includes the movement direction, movement trajectory and rotation angle of the X and Y coordinates. Based on the degree of displacement of the palm position coordinates and the dynamic gesture template, the dynamic gesture and its type are determined. The types of dynamic gestures include one of the following: directional gestures, path gestures, and rotation gestures.

6. The gesture fusion and recognition method according to claim 5, characterized in that, The gesture fusion recognition result is obtained by using a weighted decision rule to perform fusion recognition based on radar gesture recognition results and visual gesture recognition results, including: RadarGesture weights are applied to radar gesture recognition results, and cameraGesture weights are applied to visual gesture recognition results. Then, weight decision rules are used for fusion recognition to obtain gesture fusion recognition results.

7. The gesture fusion and recognition method according to claim 6, characterized in that, The method further includes: Adjust the radarGesture weight and cameraGesture weight based on the gesture type and sampling lighting conditions of the gesture image from the visual gesture recognition results.

8. A gesture fusion recognition device, characterized in that, include: The first module is used to acquire infrared data and millimeter-wave radar data of gestures, and to determine RTM and DTM based on the millimeter-wave radar data. The second module is used to identify RTM and DTM using a dual-stream neural network RDCL to obtain radar gesture recognition results; The third module is used to determine key hand points using the Mediapipe library from the infrared data, and to determine static gestures based on the angles, relative positions, and coordinates of the key hand points. The fourth module is used to determine dynamic gestures based on the direction of movement, trajectory of movement and rotation angle of key hand points, and to obtain visual gesture recognition results based on static and dynamic gestures. The fifth module is used to perform fusion recognition based on the radar gesture recognition results and the visual gesture recognition results using weighted decision rules, so as to obtain the gesture fusion recognition result.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the gesture fusion recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, which is executed by a processor to implement the gesture fusion recognition method as described in any one of claims 1-7.

Images