Millimeter wave radar sign language recognition method and system based on multi-modal feature fusion

Through the multimodal feature fusion method, the Transformer-Pose network and Doppler data are used to process radar echo data to obtain human posture and hand movement information, solving the problem of low recognition accuracy in the existing technology and achieving more High sign language recognition accuracy.

CN119028010APending Publication Date: 2024-11-26CHONGQING INNOVATION CENTER OF BEIJING INSTITUTE OF TECHNOLOGY +1

Patent Information

Application Number
CN202410951366.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing sign language recognition method based on millimeter wave radar ignores human posture information, resulting in low recognition accuracy. Especially when the low-resolution radar system directly inputs distance and angle information, the network framework is insufficient, resulting in recognition accuracy. Low.

Method used

Using multimodal feature fusion method, multi-channel radar echo data is collected through a multi-input and multi-output millimeter wave radar system, and the distance-Doppler-angle sequence and distance-Doppler sequence are obtained in the vertical and horizontal directions. and micro-Doppler diagrams, input the Transformer-Pose network model to obtain the confidence graph of the human body key point, combine joint vectors, bone vectors and Doppler data for feature extraction and fusion, and output sign language recognition results.

Benefits of technology

It improves the accuracy of sign language recognition, weakens the confusion between sign languages ​​with similar postures or similar movement trends, improves the recognition accuracy, and avoids the reduction in accuracy caused by redundancy of low-resolution radar image information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028010A_ABST
    Figure CN119028010A_ABST
Patent Text Reader

Abstract

The invention provides a millimeter wave radar sign language recognition method and system based on multi-modal feature fusion, and the method comprises the steps: constructing a Transform-Pose network through employing the characteristics that the key points of the same human body are closely related in time, and the key points of different human bodies are closely related in space, and training the network through employing a cross-modal supervision mode, thereby achieving the recognition of sign languages of different human bodies, and achieving the recognition of sign languages of different human bodies. A distance-Doppler-angle sequence is processed into fine human body skeleton information, so that the problem that the accuracy of a sign language recognition network is reduced due to information redundancy generated by a low-resolution radar image is avoided; on the other hand, the effect of the human body posture information and the hand motion information on sign language recognition is fully considered, radar echo data are processed into human body skeleton data representing the human body posture information and Doppler data representing the hand motion information, features of the multi-modal data are fused, and a sign language recognition result is obtained; confusion between sign languages with similar postures or similar motion trends is weakened, and sign language recognition accuracy is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of millimeter wave radars, and in particular to a millimeter wave radar sign language recognition method and system based on multimodal feature fusion. Background Art

[0002] Sign language is a tool that uses gestures and changes in posture to simulate language. Sign language recognition technology can automatically translate sign language into text or voice information, and can be applied to disabled people's travel, home life, employment and entrepreneurship, making it easier for disabled people to obtain information and services and fully participate in social life. Compared with other sign language recognition equipment, millimeter wave radar has the advantages of small size, non-contact, no reliance on light, and privacy protection, and has attracted widespread attention.

[0003] At present, the sign language recognition method based on millimeter-wave radar first uses Fourier transform, short-time Fourier transform and other signal processing methods to obtain radar range-Doppler map, range-angle map, micro-Doppler map, etc.; then, these signal processing result maps are input into the deep neural network for feature extraction and feature fusion, and finally the sign language recognition results are output.

[0004] Among the existing radar-based sign language recognition methods, most methods only use the Doppler features in radar data that can reflect posture changes for sign language recognition. This type of sign language recognition network has a simple architecture and is easy to train, but ignores human posture information and is prone to confusion for sign languages ​​with similar hand movement trends. Subsequently, researchers began to input radar data that can reflect target position and motion information, including distance-angle graphs, distance-Doppler graphs, etc., into the sign language recognition network at the same time to reduce confusion and improve sign language recognition accuracy. However, the target distance and angle information obtained from the radar data is limited by the spatial resolution of the radar system and cannot accurately reflect the human posture characteristics; especially for low-resolution millimeter-wave radar systems, directly inputting distance and angle information into the sign language recognition network results in insufficient fitting ability of the existing sign language recognition network framework, resulting in low recognition accuracy. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a millimeter-wave radar sign language recognition method and system based on multimodal feature fusion to solve the technical problem in the prior art that the existing sign language recognition network framework has insufficient fitting ability, resulting in low recognition accuracy.

[0006] The present invention provides a millimeter wave radar sign language recognition method based on multimodal feature fusion, comprising:

[0007] S1. Using a multi-input multi-output millimeter wave radar system to collect multi-channel radar echo data of sign language, and processing the radar echo data to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps respectively;

[0008] S2, inputting the vertical and horizontal distance-Doppler-angle sequences into the Transformer-Pose network model for processing, and outputting a human body key point confidence map;

[0009] S3, selecting human key points from the human key point confidence map, using the position vectors and confidences of the human key points to form joint vectors, and using the joint vectors to form bone vectors;

[0010] S4. Taking the joint vector, bone vector, range-Doppler sequence and micro-Doppler map as input, using a multimodal feature fusion network to extract and fuse the input data features, and outputting a sign language recognition result.

[0011] Optionally, the radar echo data is processed to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps, respectively, including:

[0012] Performing 3D-FFT on the radar echo data to obtain vertical and horizontal range-Doppler-angle sequences;

[0013] Performing 2D-FFT in the range dimension and coherent processing time and multi-channel incoherent accumulation on the radar echo data to obtain a range-Doppler sequence;

[0014] The radar echo data is subjected to distance dimension FFT, MTI processing, time dimension short-time Fourier transform, and distance dimension and space dimension incoherent accumulation to obtain a micro-Doppler map.

[0015] Optionally, the vertical and horizontal range-Doppler-angle sequences are input into

[0016] The Transformer-Pose network model is used to process and output the confidence map of key points of the human body, including:

[0017] The vertical and horizontal distance-Doppler-angle sequences are input into the encoder to output feature maps. The feature maps are summed and input into the Transformer module for feature fusion. The output fused features are passed through the decoder and graph convolutional network to obtain a human key point confidence map.

[0018] Optionally, the encoder and decoder include:

[0019] The encoder consists of three convolution modules, each of which includes a 3D convolution layer and a GRU layer;

[0020] The decoder consists of three convolutional modules, each of which includes a 2D convolutional layer.

[0021] Optionally, the Transformer-Pose network model includes:

[0022]

[0023] Among them, Y RGB Indicates the label value, Y RF Represents the network prediction value, subscript m represents the mth confidence map, and subscripts i and j represent the two-dimensional index position of the pixel.

[0024] Optionally, selecting human key points from the human key point confidence map and using position vectors of the human key points and their confidence levels to form joint vectors comprises:

[0025] The human body key points at least include the head, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip and right hip. The position and confidence of each human body key point are read from the human body key point confidence map, and the vector formed by the horizontal axis coordinates of the position of the human body key point is represented as V x =[x1,x2,...,x9] T , the vector formed by the vertical axis coordinates of the position is represented by V y =[y1,y2,...,y9] T , the confidence vector is expressed as S = [s1, s2, ..., s9] T Among them, x, y and s represent the horizontal coordinate, vertical coordinate and confidence of each key point of the human body respectively. The horizontal coordinate vector, vertical coordinate vector and confidence vector are standardized and expressed as:

[0026]

[0027] Where E(·) represents the average of each human key point in the vector, σ(·) represents the standard deviation of each human key point in the vector, and the joint vector composed of human key points is expressed as

[0028] Optionally, the method of taking the joint vector, bone vector, range-Doppler sequence and micro-Doppler map as input, extracting and fusing the input data features using a multimodal feature fusion network, and outputting a sign language recognition result includes:

[0029] The joint vector is extracted using a graph convolutional network to obtain the first feature vector q joint, use the graph convolutional network to extract the features of the bone vector and obtain the second feature vector q bone , the DNNet-BiLSTM-3D network is used to extract the features of the range-Doppler sequence to obtain the third feature vector q RD ; Use the DNNet-BiLSTM-2D network to extract features from the micro-Doppler image and obtain the fourth feature vector q μD , and fuse the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector. The fusion process is expressed as:

[0030] q merge =a1q joint +a2q bone +a3q RD +a4q μD

[0031] Among them, a1, a2, a3 and a4 are weighted coefficients, and the output is q merge This is the result of sign language recognition.

[0032] The present invention also provides a millimeter wave radar sign language recognition method based on multimodal feature fusion, comprising:

[0033] An echo data acquisition module is used to collect multi-channel radar echo data of sign language using a multi-input multi-output millimeter wave radar system, and process the radar echo data to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps respectively;

[0034] A human key point module is used to input the vertical and horizontal distance-Doppler-angle sequences into the Transformer-Pose network model for processing, and output a human key point confidence map;

[0035] A feature vector acquisition module, used for selecting human key points from the human key point confidence map, using the position vectors of the human key points and their confidences to form joint vectors, and using the joint vectors to form bone vectors;

[0036] The recognition result output module is used to take the joint vector, bone vector, range-Doppler sequence and micro-Doppler map as input, use a multimodal feature fusion network to extract and fuse the input data features, and output a sign language recognition result.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1. Taking advantage of the fact that the key points of the same human body are closely correlated in time and the key points of different human bodies are closely correlated in space, a Transformer-Pose network is constructed. The network is trained using cross-modal supervision to process the distance-Doppler-angle sequence into detailed human skeleton information, avoiding the problem of information redundancy caused by low-resolution radar images and the decrease in the accuracy of the sign language recognition network.

[0039] 2. The role of human posture information and hand movement information in sign language recognition is fully considered. The radar echo data is processed into human skeleton data representing human posture information and Doppler data representing hand movement information. The features of these multimodal data are fused to obtain the result of sign language recognition, which reduces the confusion between sign languages ​​with similar postures or similar movement trends and improves the accuracy of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0043] Figure 2 It is a schematic diagram of the structure of the method flow of the present invention;

[0044] Figure 3 Transformer-Pose network structure diagram of the present invention;

[0045] Figure 4 DNNet-BiLSTM-3D network structure diagram of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other implementation cases obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The functional units with the same labels in the examples of the present invention have the same and similar structures and functions.

[0047] See also Figure 1 The present invention provides a millimeter wave radar sign language recognition method based on multimodal feature fusion, comprising:

[0048] S1. Using a multi-input multi-output millimeter wave radar system to collect multi-channel radar echo data of sign language, and processing the radar echo data to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps respectively;

[0049] S2, inputting the vertical and horizontal distance-Doppler-angle sequences into the Transformer-Pose network model for processing, and outputting a human body key point confidence map;

[0050] S3, selecting human key points from the human key point confidence map, using the position vectors of the human key points and their confidences to form joint vectors, and using the joint vectors to form bone vectors;

[0051] S4. Taking the joint vector, bone vector, range-Doppler sequence and micro-Doppler map as input, using a multimodal feature fusion network to extract and fuse the input data features, and outputting a sign language recognition result.

[0052] See also Figure 2 In this embodiment, S1, a multi-input multi-output millimeter wave radar system is used to collect multi-channel radar echo data of sign language, and the radar echo data is processed to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps respectively.

[0053] The millimeter wave radar system described in the present invention can adopt AWR1642. This embodiment selects to collect sign language of food, and collects at least the sign language of porridge, bread, noodles, wontons, soup, wine, meat, milk, tea and cake through the millimeter wave radar system.

[0054] The multi-input and multi-output AWR1642 radar uses an antenna array to obtain multi-channel data, where the multi-channel data obtained by the horizontal array is horizontal radar data, and the multi-channel data obtained by the vertical array is vertical radar data.

[0055] The horizontal and vertical radar data are subjected to 3D-FFT of range dimension FFT, coherent processing time FFT and space dimension FFT respectively to obtain vertical and horizontal range-Doppler-angle sequences.

[0056] In addition, the radar's multi-channel data is subjected to range-dimensional FFT and coherent processing time FFT to obtain a multi-channel range-Doppler sequence, and then incoherent accumulation is performed along the spatial dimension to obtain a range-Doppler sequence with improved signal-to-noise ratio.

[0057] In addition, the radar's multi-channel data is subjected to range-dimensional FFT to obtain a range-space-pulse sequence; the range-space-pulse sequence is then processed using a two-pulse canceller to suppress stationary clutter interference and achieve MTI; then, the range-space-pulse sequence after MTI processing is subjected to short-time Fourier transform in the time dimension, incoherent accumulation along the range dimension, and incoherent accumulation along the space dimension to obtain a micro-Doppler map.

[0058] S2. Input the vertical and horizontal distance-Doppler-angle sequences into the Transformer-Pose network model for processing, and output a confidence map of key points of the human body.

[0059] See also Figure 3 In this embodiment, both vertical data and horizontal data need to pass through the encoder. The encoder consists of three convolution modules, each of which includes a 3D convolution layer and a GRU layer. The 3D convolution layer doubles the number of feature maps while reducing the size of the feature maps by half, aiming to extract deep features from the input image; the GRU layer captures the temporal dependencies in the input sequence, enhancing the network's understanding of temporal data.

[0060] Next, the output of the encoder for vertical and horizontal data is subjected to feature fusion using the Transformer module. The fused feature data is processed by the decoder to restore the image resolution; the decoder consists of three convolution modules, each of which includes a 2D convolution layer, which reduces the number of feature maps by half while doubling the size of the feature maps. Finally, the feature map output by the decoder passes through a three-layer graph convolutional network (GCN) to obtain a high-resolution confidence map of human key points.

[0061] In order to train the Transformer-Pose network model mentioned above, the output data is labeled using cross-modal supervised learning. In the process of obtaining training data, the millimeter-wave radar and the camera collect signals at the same time. The video data collected by the camera is processed by the pre-trained HRNet network, and the confidence map of the key points of the human body is used as the label value of the millimeter-wave radar data.

[0062] The loss function used to train the Transformer-Pose network is the network prediction value Y RF With label value Y RGB The binary cross entropy between is:

[0063]

[0064] Among them, Y RGB Indicates the label value, Y RFRepresents the network prediction value, subscript m represents the mth confidence map, and subscripts i and j represent the two-dimensional index position of the pixel. After the Transformer-Pose network training is completed, the process of obtaining the confidence map of the key points of the human body from the millimeter-wave radar data can be independently implemented without relying on the camera to collect data.

[0065] S3. Select human key points from the human key point confidence map, use the position vectors of the human key points and their confidences to form joint vectors, and use the joint vectors to form bone vectors.

[0066] The positions and confidences of different human key points are read from the human key point confidence map obtained in step S2, and the human key points of the upper body are selected therefrom, including nine key points of the head, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, and right hip. The selected key points are related to sign language.

[0067] Let the vector formed by the horizontal axis coordinates of the selected human key points be V x =[x1,x2,...,x9] T , the vector formed by the vertical axis coordinates is V y =[y1,y2,...,y9] T , the confidence vector is S = [s1, s2, ..., s9] T , where x, y and s represent the horizontal coordinate, vertical coordinate and confidence of each human joint point respectively. The above vector is normalized, that is,

[0068]

[0069] Where E(·) represents the average of each element of the vector, and σ(·) represents the standard deviation of each element of the vector. Then, the joint vector composed of these key points of the human body is expressed as

[0070] The connection between key points is represented by bone vectors. The key points close to the center of gravity are the source key points, and the key points far from the center of gravity are the target key points. The difference between the position vectors of the target key points and the source key points is the bone vector. For example, let the target key point vector be v i =(x i ,y i ,s i ) T , the source key point vector is v j =(x j ,y j ,s j ) T , where x, y and s represent the horizontal coordinate, vertical coordinate and confidence of the key point respectively, then a single bone vector can be expressed as The elements of all bone vectors are normalized and combined together in the same form as the joint vectors to form a bone vector.

[0071] S4. Taking the joint vector, bone vector, range-Doppler sequence and micro-Doppler map as input, using a multimodal feature fusion network to extract and fuse the input data features, and outputting a sign language recognition result.

[0072] The multimodal feature fusion network provided by this method first uses a graph convolutional network (GCN) to extract features from the joint vector to obtain the first feature vector q joint , use the graph convolutional network to extract the features of the bone vector and obtain the second feature vector q bone , the DNNet-BiLSTM-3D network is used to extract the features of the range-Doppler sequence to obtain the third feature vector q RD ; Use the DNNet-BiLSTM-2D network to extract features from the micro-Doppler image and obtain the fourth feature vector q μD , and fuse the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector. The fusion process is expressed as:

[0073] q merge =a1q joint +a2q bone +a3q RD +a4q μD

[0074] Among them, a1, a2, a3 and a4 are weighted coefficients, and the output is q merge This is the result of sign language recognition;

[0075] Among them, the DNNet-BiLSTM-3D network structure is as follows Figure 4As shown in the figure. The DNNet-BiLSTM-3D network first performs 3×3×3 and 9×9×9 3D convolutions (3D Conv) on the input data, concatenates the two convolution results, and then uses the maximum pooling layer to reduce the amount of data to reduce the computational burden; then, the processed data repeats the above operation twice, and then flattens the data into a one-dimensional vector (flatten); then, the flattened vector is processed by the Bi-LSTM network module, and then passes through two fully connected layers (dense) and a random zeroing layer (dropout) in sequence; finally, after the processed data passes through the last fully connected layer (dense), it is activated by the SoftMax function to obtain the extraction result of the sign language feature vector. The DNNet-BiLSTM-2D network architecture is consistent with the DNNet-BiLSTM-3D network, except that the 3×3×3 and 9×9×9 3D convolutions (3D Conv) are replaced by 3×3 and 9×9 2D convolutions (2D Conv).

[0076] In another embodiment, the Transformer-Pose network is first trained, and experiments are conducted on 2,000 radar and video data collected by 10 experimenters for 10 sign languages. While ensuring that the training set, validation set, and test set do not overlap, 80% of the data are randomly selected as the training set, 10% of the data are used as the validation set, and the remaining data are used as the test set. On the test set, the average precision AP of the key points of the human body concerned by sign language recognition is shown in Table 1.

[0077] Table 1 Human key point recognition performance

[0078] Key Points head shoulder elbow wrist Hip AP 75.8 67.9 53.1 30.2 90.1

[0079] Then, the multimodal data feature extraction and fusion network was trained, and experiments were conducted on 2,000 radar data of 10 sign languages ​​collected by 10 experimenters. Under the condition that the training set, validation set and test set do not overlap, 80% of the data were randomly selected as the training set, 10% of the data were used as the validation set, and the rest of the data were used as the test set. On the test set, the accuracy of sign language recognition using only the range-Doppler sequence as the input network was 53.05%, the accuracy of sign language recognition using only the micro-Doppler map as the input network was 92.12%, and the accuracy of sign language recognition using only the joint vector and bone vector as the input network was 96.79%. The accuracy of sign language recognition of this method was 98.99%.

[0080] Compared with the traditional sign language recognition method of radar signals, the present invention, on the one hand, utilizes the characteristics that the same human key points are closely related in time and different human key points are closely related in space to construct a Transformer-Pose network, and utilizes a cross-modal supervision method to train the network, and processes the distance-Doppler-angle sequence into fine human skeleton information, thereby avoiding the problem that low-resolution radar images generate information redundancy and cause the accuracy of the sign language recognition network to decrease; on the other hand, the present invention fully considers the role of human posture information and hand motion information in sign language recognition, processes radar echo data into human skeleton data representing human posture information and Doppler data representing hand motion information, and fuses the features of these multimodal data to obtain the result of sign language recognition, thereby reducing the confusion between sign languages ​​with similar postures or similar motion trends, and improving the accuracy of sign language recognition.

[0081] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0082] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A millimeter wave radar sign language recognition method based on multimodal feature fusion, It is characterized in that include: S1. Using a multi-input multi-output millimeter wave radar system to collect multi-channel radar echo data of sign language, and processing the radar echo data to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps respectively; S2, inputting the vertical and horizontal distance-Doppler-angle sequences into the Transformer-Pose network model for processing, and outputting a human body key point confidence map; S3, selecting human key points from the human key point confidence map, using the position vectors of the human key points and their confidences to form joint vectors, and using the joint vectors to form bone vectors; S4. Taking the joint vector, bone vector, range-Doppler sequence and micro-Doppler map as input, using a multimodal feature fusion network to extract and fuse the input data features, and outputting a sign language recognition result.

2. The millimeter wave radar sign language recognition method based on multimodal feature fusion as claimed in claim 1, It is characterized in that The radar echo data is processed to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps, respectively, including: Performing 3D-FFT on the radar echo data to obtain vertical and horizontal range-Doppler-angle sequences; Performing 2D-FFT in the range dimension and coherent processing time and multi-channel incoherent accumulation on the radar echo data to obtain a range-Doppler sequence; The radar echo data is subjected to distance dimension FFT, MTI processing, time dimension short-time Fourier transform, and distance dimension and space dimension incoherent accumulation to obtain a micro-Doppler map.

3. The millimeter wave radar sign language recognition method based on multimodal feature fusion as claimed in claim 1, It is characterized in that The vertical and horizontal distance-Doppler-angle sequences are input into the Transformer-Pose network model for processing, and a human body key point confidence map is output, including: The vertical and horizontal distance-Doppler-angle sequences are input into the encoder to output feature maps. The feature maps are summed and input into the Transformer module for feature fusion. The output fused features are passed through the decoder and graph convolutional network to obtain a human key point confidence map.

4. The millimeter wave radar sign language recognition method based on multimodal feature fusion as claimed in claim 3, It is characterized in that The encoder and decoder include: The encoder consists of three convolution modules, each of which includes a 3D convolution layer and a GRU layer; The decoder consists of three convolutional modules, each of which includes a 2D convolutional layer.

5. The millimeter wave radar sign language recognition method based on multimodal feature fusion as claimed in claim 4, It is characterized in that The Transformer-Pose network model includes: Among them, Y RGB Indicates the label value, Y RF Represents the network prediction value, subscript m represents the mth confidence map, and subscripts i and j represent the two-dimensional index position of the pixel.

6. The millimeter wave radar sign language recognition method based on multimodal feature fusion as claimed in claim 1, It is characterized in that The method of selecting a human body key point from the human body key point confidence map and forming a joint vector using the position vector of the human body key point and its confidence level comprises: The human body key points at least include the head, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip and right hip. The position and confidence of each human body key point are read from the human body key point confidence map, and the vector formed by the horizontal axis coordinates of the position of the human body key point is represented as V x =[x 1 ,x 2 ,...,x 9 ] T , the vector formed by the vertical axis coordinates of the position is represented by V y =[y 1 ,y 2 ,...,y 9 ] T , the confidence vector is expressed as S = [s 1 ,s 2 ,...,s 9 ] T Among them, x, y and s represent the horizontal coordinate, vertical coordinate and confidence of each key point of the human body respectively. The horizontal coordinate vector, vertical coordinate vector and confidence vector are standardized and expressed as: Where E(·) represents the average of each human key point in the vector, σ(·) represents the standard deviation of each human key point in the vector, and the joint vector composed of human key points is expressed as 7. The millimeter wave radar sign language recognition method based on multimodal feature fusion as claimed in claim 1, It is characterized in that The method uses the joint vector, bone vector, distance-Doppler sequence and micro-Doppler map as input, uses a multimodal feature fusion network to extract and fuse the input data features, and outputs a sign language recognition result, including: The joint vector is extracted using a graph convolutional network to obtain the first feature vector q joint , use the graph convolutional network to extract the features of the bone vector and obtain the second feature vector q bone , the DNNet-BiLSTM-3D network is used to extract the features of the range-Doppler sequence to obtain the third feature vector q RD ; Use the DNNet-BiLSTM-2D network to extract features from the micro-Doppler image and obtain the fourth feature vector q μD , and fuse the first eigenvector, the second eigenvector, the third eigenvector and the fourth eigenvector. The fusion process is expressed as: q merge =a 1 q joint +a 2 q bone +a 3 q RD +a 4 q μD Among them, a 1 、a 2 、a 3 and a 4 is the weighting coefficient, and outputs q merge This is the result of sign language recognition.

8. A millimeter wave radar sign language recognition method based on multimodal feature fusion, It is characterized in that include: An echo data acquisition module is used to collect multi-channel radar echo data of sign language using a multi-input multi-output millimeter wave radar system, and process the radar echo data to obtain vertical and horizontal range-Doppler-angle sequences, range-Doppler sequences and micro-Doppler maps respectively; A human key point module is used to input the vertical and horizontal distance-Doppler-angle sequences into the Transformer-Pose network model for processing, and output a human key point confidence map; A feature vector acquisition module, used for selecting human key points from the human key point confidence map, using the position vectors of the human key points and their confidences to form joint vectors, and using the joint vectors to form bone vectors; The recognition result output module is used to take the joint vector, bone vector, range-Doppler sequence and micro-Doppler map as input, use a multimodal feature fusion network to extract and fuse the input data features, and output a sign language recognition result.

Citation Information

Patent Citations

  • Posture recognition method, device and equipment and storage medium

    CN111191599A

  • Human body abnormal behavior recognition method based on dense space-time diagram convolutional network

    CN113420608A

  • Cross-modal supervised pedestrian gait recognition method based on millimeter wave radar

    CN117746496A

Cited By

  • Millimeter wave sign language data automatic generation and enhancement method based on robot system

    CN121515222A