Multi-target hand pose estimation method based on millimeter-wave radar

Abstract

This application belongs to the field of millimeter-wave radar perception and hand posture estimation, and discloses a multi-target hand posture estimation method based on millimeter-wave radar, including: Step 1: Create a hand area: Detect all objects in the millimeter-wave radar perception area, use a wake-up detection model to detect predefined wake-up gestures to identify the target user, evaluate the probability of the wake-up gesture, and identify the hand area; Step 2: Construct a range-azimuth-pitch three-dimensional feature cube in the created hand area, wherein the three-dimensional feature cube represents the hand posture characteristics; Step 3: Estimate the hand posture characteristics using the m2HandNet model. This application uses millimeter-wave radar as the research object, realizes hand posture estimation through millimeter-wave radar, supports multi-user and multi-hand interaction, and effectively solves occlusion problems to meet a wide range of application scenarios such as smart home control, smart terminal interaction, virtual reality equipment, and sign language recognition.

Description

Technical Field

[0001] The present application relates to the field of millimeter-wave radar perception and hand posture estimation, and specifically to a multi-target hand posture estimation method based on millimeter-wave radar. Background Art

[0002] With the development of human-computer interaction technology, 3D hand pose estimation (HPE) has gradually played an important role in many applications such as virtual reality, augmented reality, and smart home. Existing HPE systems mainly rely on optical sensors and wearable devices. However, solutions based on wearable devices have high requirements for user usability and are inconvenient to use. Vision-based solutions have greater flexibility but are easily affected by changes in lighting and have significant problems in privacy protection.

[0003] In recent years, millimeter-wave radar-based hand pose estimation has become a promising alternative due to its non-invasive user experience and privacy protection. Millimeter-wave radar can sense and model human hand motion trajectories and micro-movements without relying on lighting or wearing a headset, making it suitable for complex human-computer interaction scenarios.

[0004] However, in real-world scenarios, in order to achieve richer semantic expression for hand pose estimation and support a wider range of downstream applications, HPE systems should support simultaneous multi-user interaction and multi-hand operation. However, due to insufficient spatial resolution and severe occlusion of the interacting hands, existing methods are still limited to the operation prospects of a single user or a single hand. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides a multi-target hand posture estimation method based on millimeter-wave radar. The method locates the hands of the target user, constructs hand features in space, uses the Capon beam algorithm to improve the angular resolution, eliminates occlusions and performs hand posture estimation through m2HandNet. It is a multi-user multi-hand posture estimation method with scalability, occlusion robustness and user adaptability.

[0006] In order to achieve the above objectives, this application is implemented through the following technical solutions:

[0007] This application is a multi-target hand posture estimation method based on millimeter wave radar, which specifically includes the following steps:

[0008] Step 1: Create a hand region: Detect all objects in the millimeter-wave radar sensing area, identify the target user by detecting predefined wake-up gestures using a wake-up detection model, evaluate the probability of the wake-up gesture, and identify the hand region. The wake-up detection model is a binary classification detector built based on support vector data.

[0009] Step 2: construct a range-azimuth-pitch three-dimensional feature cube for the hand area created in step 1, wherein the three-dimensional feature cube represents the hand posture feature;

[0010] Step 3: Estimate the hand posture features of step 2 through the m2HandNet model, where the m2HandNet model includes a distance-robust feature extractor, a spatiotemporal converter, an occlusion elimination module, and a user adaptation module.

[0011] A further improvement of the present application is that the m2HandNet model includes:

[0012] The range-robust feature extractor includes a multi-scale feature extraction module and a feature fusion module for extracting range-robust hand posture features of the hand in the millimeter-wave radar sensing area. The multi-scale feature extraction module includes multiple deconvolution layers for extracting features of the hand in different millimeter-wave radar sensing areas. The feature fusion module includes a cross-attention layer for fusing features of different millimeter-wave radar sensing areas and generating robust features of the hand in the millimeter-wave radar sensing area.

[0013] The spatiotemporal converter includes a spatial converter, a temporal converter and a 3D posture estimation layer;

[0014] The occlusion elimination module includes an inter-hand occlusion elimination module and an internal hand occlusion elimination module;

[0015] The user adaptation module is used to adapt the m2HandNet model to new users, including hand posture modeling and hand posture decoupling.

[0016] A further improvement of the present application is that the creation of the hand region in step 1 specifically includes the following steps:

[0017] Step 1.1. Detect all objects in the millimeter-wave radar sensing area: Perform a range fast Fourier transform on the millimeter-wave radar signal received by each antenna to obtain a range-time spectrum. Perform a Doppler fast Fourier transform on the obtained range-time spectrum to obtain a range-Doppler spectrum. Use the constant false alarm detection algorithm to search for bright spots in the obtained range-Doppler spectrum, which are candidate objects. If the value of a square in the range-Doppler spectrum exceeds a threshold, it is considered that there is a candidate object in that square. For each candidate object, perform an angle fast Fourier transform on the signal of the candidate object received by each antenna to generate a range-azimuth spectrum. Spectrum and distance-elevation Spectrum, estimate the azimuth of the candidate object and pitch angle , get the polar coordinates of each candidate object ( , , ) and the transformed three-dimensional coordinates ( , , );

[0018] Step 1.2: Detect the predefined wake-up gesture to identify the target user: Extract the reflection signal of each object within the millimeter-wave radar sensing area from the millimeter-wave radar , for the reflected signal Apply short-time Fourier transform to obtain the micro-Doppler spectrum of the wake-up gesture , the micro-Doppler spectrum Reflect the time domain characteristics and frequency domain characteristics of the wake-up gesture, detect the wake-up gesture by analyzing the difference between the time domain characteristics and the frequency domain characteristics, and obtain the user positioning information;

[0019] Step 1.3: Evaluate the probability of the wake-up gesture and identify the hand area: Combine the user positioning information with the time domain features and frequency domain features obtained in step 1.2 to evaluate the probability of the wake-up gesture and identify the hand area.

[0020] A further improvement of the present application is that step 1.3 specifically includes the following steps:

[0021] Step 1.3.1. Define an indicator function To measure the micro-Doppler spectrum Whether there is a wake-up gesture, indicator function Defined as:

[0022]

[0023] in, Micro-Doppler spectrum In the wake-up detection model The projection point in the feature space of Wake-up detection model The center point of the hypersphere in the feature space of

[0024] like ≥0, indicating the current micro-Doppler spectrum Includes wake-up gestures;

[0025] Step 1.3.2, by distance , azimuth , pitch angle Extract the micro-Doppler spectrum in the three-dimensional space , calculate the micro-Doppler spectrum of value;

[0026] For satisfaction Distance ≥ 0 Determine the hand distance range of the user;

[0027] Extract the azimuth angle within the hand distance range Range-azimuth RA spectrum and elevation angle Range-elevation RE spectrum;

[0028] In distance-bearing Spectrum plotted plane and distance-elevation angle In the plane where the spectrum is drawn, find the Azimuth ≥ 0 and pitch angle As the hand area, the construction of the hand area is completed.

[0029] A further improvement of the present application is that step 2 constructs a range-azimuth-pitch three-dimensional feature cube as a hand feature, specifically comprising the following steps:

[0030] Step 2.1: In range-azimuth Spectrum plotted plane and distance-elevation angle After the target hand area is located in the plane of spectrum drawing, the Capon beamforming algorithm is used to convert the azimuth angle The resolution is increased from 15° to 1°, resulting in enhanced range-azimuth Spectrum diagram, the pitch angle Resolution increased from 30° to 1°, resulting in enhanced range-elevation Spectrum graph;

[0031] Step 2.2: Enhance the distance-azimuth Spectrum diagram and enhanced distance-elevation angle Spectrum graphs are fused to construct range-azimuth-pitch 3D feature cube as hand pose feature ,in 、 、 Respectively represent distance , azimuth , pitch angle The number of samples on .

[0032] A further improvement of the present application is that in step 3, the distance robust feature extractor extracts features by specifically comprising the following steps:

[0033] Step 3.1.1: Distance-robust feature extractor for hand posture features To zoom,

[0034] Step 3.1.2: Range-azimuth-elevation at different levels In the three-dimensional feature cube, the cross-attention layer is used to analyze the distances , azimuth , pitch angle The weights are assigned and a distance-robust feature extractor is used to extract distance-robust hand posture features.

[0035] A further improvement of the present application is that: in step 3, the spatiotemporal converter converts the azimuth-pitch plane of the hand posture feature into Expand to The characteristic vector of , specifically includes the following steps:

[0036] Step 3.2.1, the spatial transformer performs spatial feature extraction on the distance-robust hand posture features extracted in step 3.1.2;

[0037] Step 3.2.2: The temporal transformer processes the spatial features extracted in step 3.2.1 across multiple frames, learns the temporal dependencies between hand joints across frames, uses the self-attention mechanism to selectively focus on key frames, combines position embedding to preserve frame order, and extracts global spatiotemporal features.

[0038] Step 3.2.3: After obtaining the global spatiotemporal features output by the time converter, the 3D pose estimation layer uses a multi-layer perception structure to predict each frame. Predicted hand pose ;

[0039] Step 3.2.4: Guide the optimization of the m2HandNet model through the weighted joint loss function to reduce the error between the predicted hand posture and the actual hand posture. The weighted joint loss function is expressed as follows:

[0040]

[0041] in, is the number of radar frames, , is the total number of hand joints, is the predicted hand pose for all frames, is the true hand pose for all frames, For the Frame The predicted coordinates of hand joints, For the Frame The real coordinates of the hand joints, For the The weight corresponding to the p-th hand joint point in the frame is defined as:

[0042] ;

[0043] A further improvement of the present application is that in step 3, the occlusion elimination module includes an inter-hand occlusion elimination module and an internal hand occlusion elimination module, which performs inter-hand occlusion elimination and internal hand occlusion elimination on the hand posture features. The inter-hand occlusion elimination module includes right hand posture feature decomposition and left hand posture feature decomposition, and the internal hand occlusion elimination module includes posture deformation estimation and bidirectional posture correction, specifically including the following steps:

[0044] Step 3.3.1, right hand posture feature decomposition: For the occluded right hand posture features, apply the knowledge self-distillation technology and use the hand motion component distiller and interfering component distiller Decompose the occluded right hand posture feature into right hand motion component and right hand interference component;

[0045] Step 3.3.2: To ensure that the right hand motion component and the right hand interference component are not coupled with each other, the mutual information minimization loss function is used for constraint. for:

[0046]

[0047] in, Indicates the The occluded right hand pose features under the frame, Indicates the The occluded right hand motion component under the frame, Indicates the The occluded right-hand interference component under the frame, function express and The mutual information between them is calculated by variational self-distillation method;

[0048] Step 3.3.3, The occluded right hand motion component under the frame By The right-hand interference component of the occluded frame Perform supervised learning to obtain the right hand posture without occlusion between the hands , , Indicates the The frame shows the right hand posture without occlusion of both hands;

[0049] Step 3.3.4, decomposition of left hand posture features: take the right hand motion component as the interference prior, and encode the difference between the interference prior and the occluded left hand posture features into a weight matrix through the anti-interference attention mechanism. The weight matrix is ​​multiplied with the occluded left hand posture features to obtain the left hand posture without occlusion between the two hands. , , Indicates the The left hand posture of the frame without occlusion of both hands;

[0050] Step 3.3.5, posture deformation estimation: Frame and The change of hand joint coordinates between frames is defined as Frame-by-frame estimated hand pose deformation , using the weighted deformation loss function Predicted hand pose deformation for all frames For supervision, use a weighted deformation loss function Defined as:

[0051]

[0052] in, is the predicted hand posture deformation, For realistic hand gesture deformation, is the total number of hand joints, The predicted Frame No. The deformation of the hand joints, The real Frame No. Deformation of each hand joint;

[0053] Step 3.3.6, Bidirectional posture correction: By integrating All frames before the frame and the The results of posture deformation estimation of all frames after the frame are Hand poses without hand occlusions for the frame Make corrections and get Refined hand poses for each frame .

[0054] A further improvement of this application is that: in step 3.3.6, Hand poses without hand occlusions for the frame The correction method is:

[0055] For non- Frame No. Hand poses without hand occlusions for the frame , cumulative Pose deformation estimation results of all frames before the frame or Pose deformation estimation results of all frames after frame , to trace back or deduce to the Frame, get After the frame is deformed, Frame pose estimation , ;

[0056] By putting all The pose deformation estimation of the frame is traced back to the Frame Generation Frame-forward pose deformation estimation and Backward pose deformation estimation ;

[0057] Using the calculated Frame-forward pose deformation estimation and Backward pose deformation estimation Correction to improve the current The estimated accuracy of the frame and obtain the Refined hand poses for each frame :

[0058]

[0059] in, For the The weighting coefficient of the frame, For the The weighting coefficient of the frame, For the The hand poses of the frame without hand occlusion;

[0060] Introducing deformation consistency loss function to refine the hand posture of all frames obtained Supervise and refine the hand poses of all frames The true hand poses for all frames Consistent, the deformation consistency loss function is defined as follows:

[0061]

[0062] in, represents the refined hand poses of all frames, represents the true hand pose for all frames, Indicates that The first frame obtained after deformation Frame refinement of hand pose The predicted coordinates of the joint points, Indicates the Frame refinement of hand pose The real coordinates of the joint points.

[0063] A further improvement of the present application is that: in step 3, the hand posture modeling in the user adaptive module is to refine the left hand posture of all frames. and refined right hand poses for all frames Modeling is done as hand shape and joint motion, hand posture decoupling is based on hand posture modeling, and the refined left hand posture of all frames is and refined right hand poses for all frames Decouple them separately to obtain hand shape information and joint motion information, specifically:

[0064] Step 3.4.1. Hand pose modeling: Refine the left hand pose of all frames and refined right hand poses for all frames Decomposed into hand shape features and joint motion features. Hand shape features include individual hand features and proportions, represented by the length of each hand bone. Joint motion features are the direction and angle of the joints at any time, defined as the 6D rotation of each joint.

[0065] Step 3.4.2, Hand pose decoupling: Refine the left hand pose for all frames and refined right hand poses for all frames Decoupling is performed to guide the m2HandNet model to learn the hand joint motion features that are not related to the user, through the joint motion decoupler and hand-type decoupler Refine the left hand pose for all frames and refined right hand poses for all frames Decoupled into hand shape features and joint motion features respectively;

[0066] Among them, the hand-shaped decoupler Responsible for estimating the length of each hand bone, hand shape decoupler The corresponding hand shape loss function is defined as:

[0067]

[0068] in, Indicates the The predicted length of the root hand bone, Indicates the The true length of the root hand bone, Indicates the total number of bones in the hand; ;

[0069] Step 3.4.3, left hand processing flow loss term: calculate the predicted left hand posture of all frames of the m2HandNet model and the true left hand pose for all frames The error between ,

[0070]

[0071] in, represents the predicted left hand pose for all frames, represents the true left hand pose of all frames, represents the predicted left hand joint bone length for all frames, Indicates the true left hand joint bone length of all frames; represents the refined left hand pose of all frames, represents the predicted left hand pose without hand occlusion for all frames; represents the predicted left hand posture deformation for all frames, represents the true left hand posture deformation of all frames;

[0072] Step 3.4.4. Calculate the right-hand processing flow loss term: Calculate the predicted right-hand pose of all frames predicted by the m2HandNet model and the true right hand pose for all frames The error between :

[0073]

[0074] in, represents the predicted right hand pose for all frames, represents the true right hand pose for all frames, represents the predicted right hand joint bone length for all frames, represents the actual right hand joint bone length of all frames, represents the predicted right hand pose without hand occlusion for all frames, represents the predicted right hand pose deformation for all frames, represents the true right hand posture deformation of all frames;

[0075] Step 3.4.5: Calculate the left-hand processing flow loss term and the right-hand side process flow loss term The sum of:

[0076] .

[0077] The beneficial effects of this application are: the hand posture estimation method proposed in this application has the advantages of supporting multi-hand multi-user interaction, being distance robust, having user adaptability, achieving occlusion elimination, and improving recognition accuracy, specifically:

[0078] (1) Support for multi-hand and multi-user interaction: This application separates the reflected signals of multiple users and multiple hands in three dimensions: distance, speed, and angle. The reflected signals of each user can be used to estimate the posture of multiple users' hands or one hand.

[0079] (2) Distance robustness: This application utilizes a distance robust feature extractor to extract distance robust hand features, which can estimate hand posture at any distance.

[0080] (3) User-adaptive capability: This application introduces a user-adaptive module to extract the hand joint motion features that are independent of the user and can estimate the hand posture of any user.

[0081] (4) Realize occlusion elimination and improve recognition accuracy: This application establishes a hand occlusion removal module, which effectively improves the accuracy of hand posture estimation by eliminating mutual occlusion between the two hands and self-occlusion within a single hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is a flow chart of the hand posture estimation method of the present application.

[0083] Figure 2 It is a system flow chart of the hand posture estimation method in the embodiment of the present application.

[0084] Figure 3 This is the application scenario diagram of this application.

[0085] Figure 4 This is the architecture diagram of the m2HandNet model of this application.

[0086] Figure 5 It is a schematic diagram of the hand joints of this application.

[0087] Figure 6 This is a comparison chart of the user's hand gesture photo of the letter "W", the actual hand gesture, and the estimated result.

[0088] Figure 7 This is a comparison chart of the user's hand gesture photo of the letter "F", the actual hand gesture, and the estimated result.

[0089] Figure 8 This is a comparison chart of the user's hand gesture photo of the letter "Q", the actual hand gesture, and the estimated result.

[0090] Figure 9This is a comparison chart of the user's hand gesture photo of the letter "Z", the actual hand gesture, and the estimated result.

[0091] Figure 10 This is a schematic diagram of the application achieving multi-hand multi-user hand posture estimation and high-precision finger tracking with an average joint error of 9.919mm within an interaction range of 3m. DETAILED DESCRIPTION

[0092] The following diagrams illustrate embodiments of the present application. For clarity, many practical details will be included in the following description. However, it should be understood that these practical details are not intended to limit the present application. In other words, in some embodiments of the present application, these practical details are not essential. Furthermore, to simplify the drawings, some conventional structures and components are depicted in the drawings in a simplified schematic manner.

[0093] like Figure 1-3 As shown, the present application is a multi-target hand posture estimation method based on millimeter wave radar, and the multi-target hand posture estimation method specifically includes the following steps:

[0094] Step 1. Create a hand area: Detect all objects in the millimeter-wave radar sensing area, identify the target user through the wake-up detection model to detect the predefined wake-up gesture, evaluate the probability of the wake-up gesture, and identify the hand area. The wake-up detection model is a binary classification detector built based on support vector data. The predefined wake-up gesture refers to the process of the user interacting with the radar, through a pre-set specific action or gesture to trigger the device's wake-up or start-up function. Step 1 The creation of the hand area specifically includes the following steps:

[0095] Step 1.1. Detect all objects in the millimeter-wave radar sensing area: Perform a range fast Fourier transform on the millimeter-wave radar signal received by each antenna to obtain a range-time spectrum. Perform a Doppler fast Fourier transform on the obtained range-time spectrum to obtain a range-Doppler spectrum. Use the constant false alarm detection algorithm to search for bright spots in the obtained range-Doppler spectrum, which are candidate objects. If the value of a square in the range-Doppler spectrum exceeds a threshold, it is considered that there is a candidate object in that square. Specifically, the constant false alarm detection uses an adaptive threshold to detect objects by tracking background noise. Based on experience, the false alarm rate is set to 10 -4 For each candidate object, the angle fast Fourier transform is performed on the signal of the candidate object received by each antenna to generate the range-azimuth Spectrum and distance-elevation Spectrum, using the existing arrival angle estimation method to estimate the azimuth of the candidate object and pitch angle , get the polar coordinates of each candidate object ( , , ) and the transformed three-dimensional coordinates ( , , ); The conversion here refers to the conversion from polar coordinates to rectangular coordinates;

[0096] Step 1.2: Detect the predefined wake-up gesture to identify the target user: Extract the reflection signal of each object within the millimeter-wave radar sensing area from the millimeter-wave radar , for the reflected signal Apply short-time Fourier transform to obtain the micro-Doppler spectrum of the wake-up gesture , the micro-Doppler spectrum Reflect the time domain characteristics and frequency domain characteristics of the wake-up gesture, detect the wake-up gesture by analyzing the difference between the time domain characteristics and the frequency domain characteristics, and obtain the user positioning information;

[0097] Step 1.3: Evaluate the probability of the wake-up gesture and identify the hand area: Combine the user location information with the time domain and frequency domain features obtained in step 1.2 to evaluate the probability of the wake-up gesture and identify the hand area. In this step, evaluating the probability of the wake-up gesture and identifying the hand area specifically include the following steps:

[0098] Step 1.3.1. Define an indicator function To measure the micro-Doppler spectrum Whether there is a wake-up gesture, indicator function Defined as:

[0099]

[0100] in, Micro-Doppler spectrum In the wake-up detection model The projection point in the feature space of Wake-up detection model The center point of the hypersphere in the feature space of

[0101] like ≥0, indicating the current micro-Doppler spectrum It is likely to include a wake-up gesture;

[0102] Step 1.3.2, by distance , azimuth , pitch angle Extract the micro-Doppler spectrum in the three-dimensional space , calculate the micro-Doppler spectrum of Value, for the ) target users , first in Extract the micro-Doppler spectrum in the interval and measure ;

[0103] For satisfaction Distance ≥ 0 Determine the hand distance range of the user;

[0104] Extract the azimuth angle within the hand distance range Range-azimuth RA spectrum and elevation angle Range-elevation RE spectrum;

[0105] In distance-bearing Spectrum plotted plane and distance-elevation angle In the plane where the spectrum is drawn, find the Azimuth ≥ 0 and pitch angle As the hand area of ​​azimuth and pitch, the construction of the hand area is completed.

[0106] Step 2: Construct a range-azimuth-pitch three-dimensional feature cube in the hand area created in step 1, where the three-dimensional feature cube represents the hand posture feature.

[0107] Constructing a distance-azimuth-pitch three-dimensional feature cube as a hand feature specifically includes the following steps:

[0108] Step 2.1: In range-azimuth Spectrum plotted plane and distance-elevation angle After the target hand area is located in the plane of spectrum drawing, the Capon beamforming algorithm is used to convert the azimuth angle The resolution is increased from 15° to 1°, resulting in enhanced range-azimuth Spectrum diagram, the pitch angle Resolution increased from 30° to 1°, resulting in enhanced range-elevation Spectrum graph;

[0109] Step 2.2: Enhance the distance-azimuth Spectrum diagram and enhanced distance-elevation angle Spectrum graphs are fused to construct range-azimuth-pitch 3D feature cube as hand pose feature ,in 、 、 Respectively represent distance , azimuth , pitch angle The number of samples on .

[0110] Step 3: Estimate the hand posture features of step 2 through the m2HandNet model, which includes a distance-robust feature extractor, a spatiotemporal converter, an occlusion elimination module, and a user adaptation module. Figure 4 As shown, the m2HandNet model includes:

[0111] The distance-robust feature extractor includes a multi-scale feature extraction module and a feature fusion module for extracting distance-robust hand posture features of the hand in the millimeter-wave radar sensing area. The multi-scale feature extraction module includes multiple deconvolution layers for extracting features of the hand in different millimeter-wave radar sensing areas. The feature fusion module includes a cross-attention layer for fusing features of different millimeter-wave radar sensing areas and generating robust features of the hand in the millimeter-wave radar sensing area. The spatiotemporal converter includes a spatial converter, a temporal converter, and a 3D posture estimation layer for learning the spatial features between the fingers of the hand and the temporal features at different moments. The occlusion removal module includes an inter-hand occlusion removal module and an intra-hand occlusion removal module for decoupling the mixed two-hand features to eliminate inter-hand occlusion and intra-hand occlusion. The user adaptation module is used to adapt the m2HandNet model to new users, including hand posture modeling and hand posture decoupling.

[0112] In step 3, the distance robust feature extractor extracts features by specifically comprising the following steps:

[0113] Step 3.1.1: Distance-robust feature extractor for hand posture features To zoom,

[0114] Step 3.1.2: Range-azimuth-elevation at different levels In the three-dimensional feature cube, the cross-attention layer is used to analyze the distances , azimuth , pitch angle The weights are assigned and a distance-robust feature extractor is used to extract distance-robust hand posture features.

[0115] In step 3, the spatiotemporal transformer transforms the hand posture features of the azimuth-pitch plane Expand to The feature vector of is adapted to the input format of the subsequent self-attention mechanism, which specifically includes the following steps:

[0116] Step 3.2.1: The spatial transformer extracts spatial features from the distance-robust hand posture features extracted in step 3.1.2. The extracted spatial features describe the dependencies between fingers and serve as input to the subsequent temporal transformer.

[0117] Step 3.2.2: The temporal transformer processes the spatial features extracted in step 3.2.1 across multiple frames, learns the temporal dependencies between hand joints across frames, uses the self-attention mechanism to selectively focus on key frames, combines position embedding to preserve frame order, and extracts global spatiotemporal features.

[0118] Step 3.2.3: After obtaining the global spatiotemporal features output by the time converter, the 3D pose estimation layer uses a multi-layer perception structure to predict each frame. Predicted hand pose ;

[0119] Step 3.2.4: Guide the optimization of the m2HandNet model through the weighted joint loss function to reduce the error between the predicted hand posture and the actual hand posture. The weighted joint loss function is expressed as follows:

[0120]

[0121] in, is the number of radar frames, , is the total number of hand joints, is the predicted hand pose for all frames, is the true hand pose for all frames, For the Frame The predicted coordinates of hand joints, For the Frame The real coordinates of the hand joints, For the Frame The weight corresponding to each hand joint is defined as:

[0122] ;

[0123] like Figure 4 As shown, the red arrows entering the spatiotemporal converter represent the distance-robust left-hand features extracted by the distance-robust feature extractor and input into the spatiotemporal converter, and the blue arrows represent the distance-robust right-hand features extracted by the distance-robust feature extractor and input into the spatiotemporal converter.

[0124] When the hand posture feature is blocked, the occlusion elimination module includes an inter-hand occlusion elimination module and an internal hand occlusion elimination module, which perform inter-hand occlusion elimination and internal hand occlusion elimination on the hand posture features. The inter-hand occlusion elimination module includes right hand posture feature decomposition and left hand posture feature decomposition, and the internal hand occlusion elimination module includes posture deformation estimation and bidirectional posture correction, specifically including the following steps:

[0125] Step 3.3.1, right hand posture feature decomposition: For the occluded right hand posture features, apply the knowledge self-distillation technology and use the hand motion component distiller and interfering component distiller Decompose the occluded right hand posture feature into right hand motion component and right hand interference component;

[0126] Step 3.3.2: To ensure that the right hand motion component and the right hand interference component are not coupled with each other, the mutual information minimization loss function is used for constraint. for:

[0127]

[0128] in, Indicates the The occluded right hand pose features under the frame, Indicates the The occluded right hand motion component under the frame, Indicates the The occluded right-hand interference component under the frame, function express and The mutual information between them is calculated by variational self-distillation method;

[0129] Step 3.3.3, The occluded right hand motion component under the frame By The right-hand interference component of the occluded frame Perform supervised learning to obtain the right hand posture without occlusion between the hands , , Indicates the The frame shows the right hand posture without occlusion of both hands;

[0130] Step 3.3.4, decomposition of left hand posture features: take the right hand motion component as the interference prior, and encode the difference between the interference prior and the occluded left hand posture features into a weight matrix through the anti-interference attention mechanism. The weight matrix is ​​multiplied with the occluded left hand posture features to obtain the left hand posture without occlusion between the two hands. , , Indicates the The left hand posture of the frame without occlusion of both hands;

[0131] Step 3.3.5, posture deformation estimation: Frame and The change of hand joint coordinates between frames is defined as Frame-by-frame estimated hand pose deformation , using the weighted deformation loss function Predicted hand pose deformation for all frames For supervision, use a weighted deformation loss function Defined as:

[0132]

[0133] in, is the predicted hand posture deformation, For realistic hand gesture deformation, is the total number of hand joints, The predicted Frame No. The deformation of the hand joints, The real Frame No. Deformation of each hand joint;

[0134] Step 3.3.6, Bidirectional posture correction: By integrating All frames before the frame and the The results of posture deformation estimation of all frames after the frame are Hand poses without hand occlusions for the frame Make corrections and get Refined hand poses for each frame .

[0135] In step 3.3.6, Hand poses without hand occlusions for the frame The correction method is:

[0136] For non- Frame No. Hand poses without hand occlusions for the frame , cumulative Pose deformation estimation results of all frames before the frame or Pose deformation estimation results of all frames after frame , to trace back or deduce to the Frame, get After the frame is deformed, Frame pose estimation , ;

[0137]

[0138] By putting all The pose deformation estimation of the frame is traced back to the Frame Generation Frame-forward pose deformation estimation and Backward pose deformation estimation ;

[0139] Using the calculated Frame-forward pose deformation estimation and Backward pose deformation estimation Correction to improve the current The estimated accuracy of the frame and obtain the Refined hand poses for each frame :

[0140]

[0141] in, For the The weighting coefficient of the frame, For the The weighting coefficient of the frame, For the The hand poses of the frame without hand occlusion;

[0142] Introducing deformation consistency loss function to refine the hand posture of all frames obtained Supervise and refine the hand poses of all frames The true hand poses for all frames Consistent, the deformation consistency loss function is defined as follows:

[0143]

[0144] in, represents the refined hand poses of all frames, represents the true hand pose for all frames, Indicates that The first frame obtained after deformation Frame refinement of hand pose The predicted coordinates of the joint points, Indicates the Frame refinement of hand pose The real coordinates of the joint points.

[0145] The hand pose modeling in the user adaptation module refines the left hand pose of all frames and refined right hand poses for all frames Modeling is done as hand shape and joint motion, hand posture decoupling is based on hand posture modeling, and the refined left hand posture of all frames is and refined right hand poses for all frames Decouple them separately to obtain hand shape information and joint motion information, specifically:

[0146] Step 3.4.1. Hand pose modeling: Refine the left hand pose of all frames and refined right hand poses for all frames Decomposed into hand shape features and joint motion features. Hand shape features include individual hand features and proportions, represented by the length of each hand bone. Joint motion features are the direction and angle of the joints at any time, defined as the 6D rotation of each joint.

[0147] Step 3.4.2, Hand pose decoupling: Refine the left hand pose for all frames and refined right hand poses for all frames Decoupling is performed to guide the m2HandNet model to learn the hand joint motion features that are not related to the user, through the joint motion decoupler and hand-type decoupler Refine the left hand pose for all frames and refined right hand poses for all frames Decoupled into hand shape features and joint motion features respectively;

[0148] Among them, the hand-shaped decoupler Responsible for estimating the length of each hand bone, hand shape decoupler The corresponding hand shape loss function is defined as:

[0149]

[0150] in, Indicates the The predicted length of the root hand bone, Indicates the The true length of the root hand bone, Indicates the total number of bones in the hand; ;

[0151] Step 3.4.3, left hand processing flow loss term: calculate the predicted left hand posture of all frames of the m2HandNet model and the true left hand pose for all frames The error between ,

[0152]

[0153] in, represents the predicted left hand pose for all frames, represents the true left hand pose of all frames, represents the predicted left hand joint bone length for all frames, Indicates the true left hand joint bone length of all frames; represents the refined left hand pose of all frames, represents the predicted left hand pose without hand occlusion for all frames; represents the predicted left hand posture deformation for all frames, represents the true left hand posture deformation of all frames;

[0154] Step 3.4.4. Calculate the right-hand processing flow loss term: Calculate the predicted right-hand pose of all frames predicted by the m2HandNet model and the true right hand pose for all frames The error between :

[0155]

[0156] in, represents the predicted right hand pose for all frames, represents the true right hand pose for all frames, represents the predicted right hand joint bone length for all frames, represents the actual right hand joint bone length of all frames, represents the predicted right hand pose without hand occlusion for all frames, represents the predicted right hand pose deformation for all frames, represents the true right hand posture deformation of all frames;

[0157] Step 3.4.5: Calculate the left-hand processing flow loss term and the right-hand side process flow loss term The sum of:

[0158] .

[0159] Figure 6 Shows a photo of a user making a "W" hand gesture, the actual hand pose corresponding to the gesture, and the hand pose estimated by this application. Figure 7 Shows a photo of a user making the letter "F" gesture, the actual hand pose corresponding to the gesture, and the hand pose estimated by this application. Figure 8 Shows a photo of a user making a "Q" hand gesture, the actual hand pose corresponding to the gesture, and the hand pose estimated by this application. Figure 9 Shows a photo of a user making a "Z" hand gesture, the actual hand pose corresponding to the gesture, and the hand pose estimated by this application.

[0160] like Figure 10As shown, the present application achieves multi-hand and multi-user hand pose estimation and high-precision finger tracking with an average joint error of 9.919 mm within an interaction range of 3 m.

[0161] This application utilizes millimeter-wave radar sensing technology to achieve hand posture estimation through millimeter-wave radar. While supporting multi-user and multi-hand interaction, it can effectively solve the occlusion problem and improve distance robustness, thus being applicable to a wide range of human-computer interaction scenarios.

[0162] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A multi-target hand posture estimation method based on millimeter wave radar, characterized by: The multi-target hand posture estimation method specifically comprises the following steps: Step 1: Create a hand region: Detect all objects in the millimeter-wave radar sensing area, identify the target user by detecting a predefined wake-up gesture using a wake-up detection model, evaluate the probability of the wake-up gesture, and identify the hand region. The wake-up detection model is a binary classification detector built based on support vector data. Step 2: construct a range-azimuth-pitch three-dimensional feature cube for the hand area created in step 1, wherein the three-dimensional feature cube represents the hand posture feature; Step 3: Estimate the hand posture features of step 2 through the m2HandNet model, which includes a distance-robust feature extractor, a spatiotemporal converter, an occlusion elimination module, and a user adaptation module, wherein: The m2HandNet model includes: The range-robust feature extractor includes a multi-scale feature extraction module and a feature fusion module for extracting range-robust hand posture features of the hand in the millimeter-wave radar sensing area. The multi-scale feature extraction module includes multiple deconvolution layers for extracting features of the hand in different millimeter-wave radar sensing areas. The feature fusion module includes a cross-attention layer for fusing features of different millimeter-wave radar sensing areas and generating robust features of the hand in the millimeter-wave radar sensing area. The spatiotemporal converter includes a spatial converter, a temporal converter and a 3D posture estimation layer; The occlusion elimination module includes an inter-hand occlusion elimination module and an internal hand occlusion elimination module; The user adaptation module is used to adapt the m2HandNet model to new users, including hand posture modeling and hand posture decoupling.

2. The multi-target hand posture estimation method based on millimeter wave radar according to claim 1, characterized in that: The step 1 of creating the hand region specifically includes the following steps: Step 1.

1. Detect all objects in the millimeter-wave radar sensing area: Perform a range fast Fourier transform (FFT) on the millimeter-wave radar signal received by each antenna to obtain a range-time spectrum. Perform a Doppler FFT on the obtained range-time spectrum to obtain a range-Doppler spectrum. Search for bright spots in the obtained range-Doppler spectrum to identify candidate objects. If the value of a square in the range-Doppler spectrum exceeds a threshold, the square is considered to contain a candidate object. For each candidate object, perform an angle fast Fourier transform on the candidate object signal received by each antenna to generate a range-azimuth (RA) spectrum and a range-elevation (RE) spectrum. Estimate the candidate object's azimuth (α) and elevation (β) angles, and obtain the polar coordinates (r, α, β) and the converted three-dimensional coordinates (x, y, z) of each candidate object. Step 1.2: Detect the predefined wake-up gesture to identify the target user: Extract the reflection signal S of each object within the millimeter-wave radar sensing area from the millimeter-wave radar. r , for the reflected signal S r Apply short-time Fourier transform to obtain the micro-Doppler spectrum μD of the wake-up gesture r , the micro-Doppler spectrum μD r Reflect the time domain characteristics and frequency domain characteristics of the wake-up gesture, detect the wake-up gesture by analyzing the difference between the time domain characteristics and the frequency domain characteristics, and obtain the user positioning information; Step 1.3: Evaluate the probability of the wake-up gesture and identify the hand area: Combine the user positioning information with the time domain features and frequency domain features obtained in step 1.2 to evaluate the probability of the wake-up gesture and identify the hand area.

3. The multi-target hand posture estimation method based on millimeter wave radar according to claim 2, characterized in that: The step 1.3 specifically includes the following steps: Step 1.3.

1. Define an indicator function I wake To measure the micro-Doppler spectrum μD r Is there a wake-up gesture in the indicator function I wake Defined as: in, is the micro-Doppler spectrum μD r In the wake-up detection model The projection point in the feature space of The center point of the hypersphere in the feature space, if I wake ≥0, indicating the current micro-Doppler spectrum μD r Includes wake-up gestures; Step 1.3.2: Extract the micro-Doppler spectrum μD in the three-dimensional space defined by distance r, azimuth angle α, and elevation angle β. r , calculate the micro-Doppler spectrum μD r I wake value; To satisfy I wake The distance r ≥ 0 is determined as the hand distance range of the user; Extract the azimuth angle α∈[α o -60°,α o +60°] and the range-azimuth RA spectrum and elevation angle β∈[β o -60°,β o +60°] range-elevation RE spectrum; In the plane where the range-azimuth RA spectrum is drawn and the plane where the range-elevation RE spectrum is drawn, find the planes that satisfy I wake The azimuth angle α and pitch angle β ≥ 0 are regarded as the hand region, completing the construction of the hand region.

4. The multi-target hand posture estimation method based on millimeter wave radar according to claim 3, characterized in that: Step 2: Construct a range-azimuth-pitch 3D feature cube as a hand feature. This includes the following steps: Step 2.1: After the target hand area is located in the plane plotted by the range-azimuth spectrum (RA) and the plane plotted by the range-elevation spectrum (RE), the Capon beamforming algorithm is used to increase the resolution of the azimuth angle α from 15° to 1°, resulting in an enhanced range-azimuth spectrum. The resolution of the elevation angle β is increased from 30° to 1°, resulting in an enhanced range-elevation spectrum. Step 2.2: Fuse the enhanced range-azimuth RA spectrum with the enhanced range-elevation RE spectrum to construct a range-azimuth-elevation RAE three-dimensional feature cube as the hand posture feature x∈R R×A×E , where R, A, and E represent the number of samples at distance r, azimuth angle α, and elevation angle β, respectively.

5. The multi-target hand posture estimation method based on millimeter wave radar according to claim 4, characterized in that: In step 3, the distance robust feature extractor extracts features by specifically comprising the following steps: Step 3.1.1: Distance-robust feature extractor for hand pose features x∈R R×A×E Zoom in or out; Step 3.1.2: In the distance-azimuth-elevation (RAE) three-dimensional feature cubes at different levels, weights are assigned to each distance r, azimuth angle α, and pitch angle β through the cross-attention layer, and distance-robust hand posture features are extracted using a distance-robust feature extractor.

6. The multi-target hand posture estimation method based on millimeter wave radar according to claim 5, characterized in that: In step 3, the spatiotemporal transformer expands the azimuth-elevation plane AE of the hand posture feature into a feature vector A×E, which specifically includes the following steps: Step 3.2.1, the spatial transformer performs spatial feature extraction on the distance-robust hand posture features extracted in step 3.1.2; Step 3.2.2: The temporal transformer processes the spatial features extracted in step 3.2.1 across multiple frames, learns the temporal dependencies between hand joints across frames, uses the self-attention mechanism to selectively focus on key frames, combines position embedding to preserve frame order, and extracts global spatiotemporal features. Step 3.2.3: After obtaining the global spatiotemporal features output by the time converter, the 3D pose estimation layer uses a multi-layer perception structure to predict the hand pose for each frame t. Step 3.2.4: Guide the optimization of the m2HandNet model through the weighted joint loss function to reduce the error between the predicted hand posture and the actual hand posture. The weighted joint loss function is expressed as follows: Where T is the number of radar frames, 1≤t≤T, N is the total number of hand joints, is the predicted hand pose for all frames, J is the true hand pose for all frames, Predict the coordinates of the p-th hand joint point in the t-th frame, j t,p is the real coordinate of the p-th hand joint point in the t-th frame, is the weight corresponding to the p-th hand joint in the t-th frame, defined as:

7. The multi-target hand posture estimation method based on millimeter wave radar according to claim 1, characterized in that: In step 3, the occlusion elimination module includes an inter-hand occlusion elimination module and an internal hand occlusion elimination module, which perform inter-hand occlusion elimination and internal hand occlusion elimination on the hand posture features. The inter-hand occlusion elimination module includes right hand posture feature decomposition and left hand posture feature decomposition, and the internal hand occlusion elimination module includes posture deformation estimation and bidirectional posture correction, specifically including the following steps: Step 3.3.1, right hand posture feature decomposition: For the occluded right hand posture features, apply the knowledge self-distillation technology and use the hand motion component distiller and interfering component distiller Decompose the occluded right hand posture feature into right hand motion component and right hand interference component; Step 3.3.2: To ensure that the right hand motion component and the right hand interference component are not coupled with each other, the mutual information minimization loss function is used for constraint. for: in, represents the occluded right hand posture feature in the t-th frame, represents the motion component of the right hand that is blocked in the t-th frame, Represents the obscured right-hand interference component under the t-th frame, function express and The mutual information between them is calculated by variational self-distillation method; Step 3.3.3: The occluded right hand motion component in frame t The right-hand interference component of the occluded component under the t-th frame Perform supervised learning to obtain the right hand posture without occlusion between the hands represents the right hand posture without hand occlusion in frame t; Step 3.3.4, decomposition of left hand posture features: take the right hand motion component as the interference prior, and encode the difference between the interference prior and the occluded left hand posture features into a weight matrix through the anti-interference attention mechanism. The weight matrix is ​​multiplied with the occluded left hand posture features to obtain the left hand posture without occlusion between the two hands. represents the left hand posture without occlusion of both hands in the t-th frame; Step 3.3.5, Posture deformation estimation: The change in hand joint coordinates between frame t and frame t-1 is defined as the hand posture deformation estimated at frame t. Use weighted deformation loss function Predicted hand pose deformation for all frames For supervision, use a weighted deformation loss function Defined as: in, is the predicted hand posture deformation, d is the actual hand posture deformation, N is the total number of hand joints, represents the predicted deformation of the p-th hand joint point in the t-th frame, d t,p The actual deformation of the p-th hand joint in the t-th frame; Step 3.3.6, Bidirectional Posture Correction: By integrating the results of posture deformation estimation of all frames before the t-th frame and all frames after the t-th frame, the hand posture without hand occlusion in the t-th frame is corrected. Correction is performed to obtain the refined hand posture of the tth frame 8. The multi-target hand posture estimation method based on millimeter wave radar according to claim 6, characterized in that: In step 3.3.6, the hand pose without hand occlusion in frame t The correction method is: For the k-th frame that is not the t-th frame, the hand pose without hand occlusion Accumulate the pose deformation estimation results of all frames before the tth frame Or the pose deformation estimation results of all frames after the tth frame Go back to the t-th frame and get the pose estimate of the t-th frame after the k-th frame is deformed Generate the forward pose deformation estimate of frame t-1 by backtracking the pose deformation estimates of all T frames to the tth frame and Tt backward pose deformation estimates Use the calculated t-1 frame forward pose deformation estimation and Tt backward pose deformation estimates Correction to improve the estimation accuracy of the current t-th frame and obtain the refined hand posture of the t-th frame Among them, c t is the weighting coefficient of the tth frame, c k is the weighting coefficient of the kth frame, is the hand pose without hand occlusion in the t-th frame; Introducing deformation consistency loss function to refine the hand posture of all frames obtained Supervise and refine the hand poses of all frames Consistent with the true hand pose J of all frames, the deformation consistency loss function is defined as follows: in, represents the refined hand poses of all frames, J represents the true hand poses of all frames, represents the predicted coordinates of the p-th joint point of the hand posture in the t-th frame after deformation of the k-th frame, j t,p Represents the true coordinates of the p-th joint point of the refined hand pose in the t-th frame.

9. The multi-target hand posture estimation method based on millimeter wave radar according to claim 8, characterized in that: In step 3, the hand posture modeling in the user adaptation module refines the left hand posture of all frames and refined right hand poses for all frames Modeling is done as hand shape and joint motion, hand posture decoupling is based on hand posture modeling, and the refined left hand posture of all frames is and refined right hand poses for all frames Decouple them separately to obtain hand shape information and joint motion information, specifically: Step 3.4.

1. Hand pose modeling: Refine the left hand pose of all frames and refined right hand poses for all frames Decomposed into hand shape features and joint motion features. Hand shape features include individual hand features and proportions, represented by the length of each hand bone. Joint motion features are the direction and angle of the joints at any time, defined as the 6D rotation of each joint. Step 3.4.2, Hand pose decoupling: Refine the left hand pose for all frames and refined right hand poses for all frames Decoupling is performed to guide the m2HandNet model to learn the hand joint motion features that are irrelevant to the user, and the joint motion decoupler D motion and hand-type decoupler D shape Refine the left hand pose for all frames and refined right hand poses for all frames Decoupled into hand shape features and joint motion features respectively; Among them, the hand-shaped decoupler D shape Responsible for estimating the length of each hand bone, hand shape decoupler D shape The corresponding hand shape loss function is defined as: in, represents the predicted length of the g-th hand bone, B g represents the true length of the g-th hand bone, G represents the total number of hand bones; g∈G; Step 3.4.3, left hand processing flow loss term: calculate the predicted left hand posture of all frames of the m2HandNet model And the true left hand pose J of all frames L The error between in, represents the predicted left hand pose of all frames, J L represents the true left hand pose of all frames, represents the predicted left hand joint bone length for all frames, B L Indicates the true left hand joint bone length of all frames; represents the refined left hand pose of all frames, represents the predicted left hand pose without hand occlusion for all frames; represents the predicted left hand posture deformation of all frames, d L represents the true left hand posture deformation of all frames; Step 3.4.

4. Calculate the right-hand processing flow loss term: Calculate the predicted right-hand pose of all frames predicted by the m2HandNet model and the true right hand pose J of all frames R The error between in, represents the predicted right hand pose for all frames, J R represents the true right hand pose for all frames, represents the predicted right hand joint bone length for all frames, B R represents the actual right hand joint bone length of all frames, represents the predicted right hand pose without hand occlusion for all frames, represents the predicted right hand pose deformation of all frames, d R represents the true right hand posture deformation of all frames; Step 3.4.5: Calculate the left-hand processing flow loss term and the right-hand side process flow loss term The sum of:

Images