Method for 3D pose estimation of mmwave radar based on physical prior and dynamic spatio-temporal graph

Abstract

The application discloses a millimeter wave radar 3D posture estimation method based on physical prior and dynamic space-time graph, and relates to the technical field of millimeter wave radar human body posture estimation; the method comprises the following steps: synchronously collecting echo signals by using two millimeter wave radars with double visual angles, and obtaining frequency spectrum data of double visual angles and double modes after processing; a coder is spliced to obtain coded Doppler spectrum features and distance spectrum features; a cross-modal fusion module of physical prior is used to explicitly establish the differential kinematics correlation of velocity and distance and perform physical cross-modal fusion, so that cross-modal fusion features after physical semantic alignment are obtained; the cross-modal fusion features are decoded into an initial 3D skeleton coordinate sequence by using a one-dimensional convolution regression head; dynamic space-time graph modeling and a gating fusion module are used for space-time topology optimization to obtain an optimized skeleton coordinate sequence; and a final 3D joint coordinate sequence is obtained by joint mapping of a decoder. The application can obtain high-precision 3D joint coordinate.

Description

Technical Field

[0001] This invention belongs to the field of millimeter-wave radar human posture estimation technology, and particularly relates to a millimeter-wave radar 3D posture estimation method based on physical priors and dynamic spatiotemporal maps. Background Technology

[0002] The development of radar-based human pose estimation technology, designed for all-weather, proactive sensing with privacy protection, aims to overcome the shortcomings of traditional visible light visual perception, such as poor robustness and privacy compliance limitations in low visibility (nighttime, backlight), complex weather conditions (rain, snow, fog), severe obstruction, and crowded scenarios. This technology holds significant engineering and academic value. Currently, this field mainly involves two technical approaches: LiDAR and millimeter-wave radar (MMWR).

[0003] (1) LiDAR Approach: LiDAR generates a 3D point cloud with absolute physical scale and high-precision geometric structure by emitting laser pulses and receiving echoes. Its core advantage lies in being completely unaffected by lighting conditions, and being able to directly characterize the topology of the human skeleton, limb lengths, and motion constraints in 3D space, thereby maintaining stable perception under complex backgrounds and extreme lighting conditions. Existing technologies typically use point cloud representation and deep learning models (such as sparse convolution, graph networks, local surface / voxel modeling, etc.) to extract human pose features, and introduce kinematic priors and physical feasible domain boundaries to constrain the optimization process, in order to overcome the uncertainties caused by sparse sampling and self-occlusion of point clouds.

[0004] (2) Millimeter-wave radar approach: Millimeter-wave radar utilizes frequency-modulated continuous wave (FMCW) or MIMO technology to acquire radar data including dimensions such as range, Doppler velocity, angle, and echo intensity. This approach has moved from theoretical verification to deployment. Typical public benchmarks, such as the HuPR dataset released at WACV 2023, have achieved strict synchronization alignment between millimeter-wave radar and a monocular RGB camera, verifying the advantages of radar in low light and occlusion conditions. Specific methods are divided into two sub-routes based on spectrograms and point cloud sequences:

[0005] The spectrogram method is based on micro-Doppler spectrum, range-Doppler map (RDM) or range-angle map (RAM), and extracts discriminative features such as gait cycle and limb swing through time series modeling.

[0006] Point cloud sequence methods directly process sparse radar point clouds or 4D point cloud sequences (i.e., radar point cloud videos), and utilize networks such as the PointNet series to extract hierarchical spatiotemporal features. Some works further utilize 6D echoes (spatial coordinates XYZ, velocity V, intensity I, time T) to construct a joint representation of "velocity-depth-time" to achieve end-to-end learning.

[0007] Furthermore, the cutting-edge technology node is "MVDoppler-Pose," presented at CVPR 2025. This scheme employs a multi-view spatial detection strategy, using two millimeter-wave radars to jointly observe the human body from the front and side, respectively. At the signal processing end, high-dimensional Doppler spectrum and range spectrum are extracted through Short-Time Fourier Transform (STFT) and Fast Fourier Transform (FFT), respectively, and directly used as input to the deep learning model. This maximizes the preservation of the human body's dynamic spatiotemporal information and achieves the best pose estimation effect to date.

[0008] The aforementioned existing technologies still have the following shortcomings: (1) The lack of physical constraints in multimodal fusion leads to blind fusion and physical distortion. Existing methods (such as MVDoppler-Pose) treat the Doppler spectrum (representing velocity) and the distance spectrum (representing distance) as independent image channels, and use simple feature stitching or conventional shallow attention mechanisms for fusion. This approach completely severs the natural calculus-based physical constraint relationship between velocity and distance (the distance is obtained by integrating velocity, and the velocity is obtained by differentiating distance). When a certain mode is affected by environmental noise or clutter, the network cannot use kinematic priors for cross-validation and feature compensation, resulting in blind mode fusion and severely reducing the specificity and interpretability of pose features.

[0009] (2) Predefined static skeleton graphs cannot adapt to dynamic occlusion and missing joints. Existing methods typically rely on predefined static graph convolutional networks (GCNs) for skeleton topology modeling when extracting 3D joints. Millimeter-wave radar signals are highly susceptible to limb self-occlusion and multipath effects when capturing human motion, resulting in the loss of some joint signals. Traditional models use fixed skeleton connection graphs, which cannot adapt to the dynamic inference requirements of invisible joints, exhibiting great rigidity and significantly increasing estimation errors in occluded scenarios.

[0010] (3) The disconnect between spatiotemporal features leads to poor continuity in long sequence estimation. Traditional temporal modeling often separates spatial graph propagation from temporal feature extraction, failing to establish an efficient dynamic fusion mechanism between "spatial graph features" and "temporal historical features." This makes it difficult for the network to form a stable temporal representation, resulting in severely compromised continuity and consistency of pose estimation in long sequence inputs or dynamically changing scenarios (such as turning around or waving quickly), easily leading to jitter or misalignment artifacts.

[0011] This invention proposes a millimeter-wave radar 3D human pose estimation method based on physical priors and dynamic spatiotemporal graphs. The key technologies are a dynamic spatiotemporal graph modeling and gating fusion module and a physical prior cross-modal fusion module. The physical prior cross-modal fusion module effectively eliminates the physical semantic gap between Doppler and range-spectrum measurements. The dynamic spatiotemporal graph modeling and gating fusion module adaptively captures complex human anatomical topology and kinematic laws. Summary of the Invention

[0012] The purpose of this invention is to provide a 3D attitude estimation method for millimeter-wave radar based on physical priors and dynamic spatiotemporal graphs. This addresses the problems mentioned in the background art, such as existing millimeter-wave radar human attitude estimation methods neglecting the differential kinematic constraints between Doppler and range spectrum during multimodal fusion, resulting in blind fusion and poor noise resistance; existing methods using predefined static skeleton graphs cannot dynamically adapt to joint loss caused by occlusion in radar signals, leading to insufficient attitude estimation robustness; and existing methods having fragmented spatiotemporal feature modeling and lacking a deep spatiotemporal interaction mechanism, resulting in poor continuity of attitude estimation under long sequences or highly dynamic actions.

[0013] To achieve the above objectives, the present invention employs the following technical solution: This invention proposes a 3D attitude estimation method for millimeter-wave radar based on physical priors and dynamic spatiotemporal graphs, comprising the following steps: S1. Data preprocessing: Two millimeter-wave radars deployed on the front and sides of the human body are used to simultaneously collect echo signals. The echo data of each radar is used to extract the Doppler spectrum and range spectrum using short-time Fourier transform and fast Fourier transform respectively, so as to obtain dual-view dual-mode spectrum data. S2. Feature extraction: Based on dual-view, dual-modal spectral data, an encoder is used to concatenate the same modal data from different viewpoints along the sequence dimension to obtain the encoded Doppler spectral features and distance spectral features. S3, Multimodal data feature alignment and fusion; The Doppler spectral features represent velocity and the distance spectral features represent distance. The calculus kinematics relationship between velocity and distance is explicitly established using the physical prior cross-modal fusion module, and physical cross-modal fusion is performed to obtain the Doppler cross-modal fusion features and distance cross-modal fusion features after physical semantic alignment. The Doppler cross-modal fusion features and the distance cross-modal fusion features are decoded into initial 3D skeleton coordinate sequences for different modalities by a one-dimensional convolutional regression head; The initial 3D skeleton coordinate sequences of different modalities are optimized for spatiotemporal topology using dynamic spatiotemporal graph modeling and gating fusion module. The optimized Doppler modal skeleton features and distance modal skeleton features are obtained through dynamic joint relationship learning and deep spatiotemporal interaction. S4, 3D human pose estimation: Flatten and stitch together the Doppler modal skeleton features and the distance modal skeleton features, and obtain the final 3D joint coordinate sequence through joint mapping by the decoder.

[0014] Preferably, the dual-view dual-mode spectral data in S1 includes the frontal view Doppler spectrum, the frontal view distance spectrum, the side view Doppler spectrum, and the side view distance spectrum.

[0015] Preferably, the encoder in S2 is specifically as follows: The encoder consists of several stacked Transformer modules with shared weights. Each Transformer module includes a CNN sub-layer and a cross-view Transformer sub-layer. The CNN sub-layer uses 3×3 convolution to extract local micro-Doppler textures from the spectrogram. The cross-view Transformer sub-layer concatenates the patch sequences from the front and side views along the sequence dimension and calculates multi-head self-attention to achieve implicit alignment and feature extraction from the front and side views.

[0016] Preferably, the physical prior cross-modal fusion module in S3 is specifically as follows: The cross-modal fusion module of the physical prior includes a physical prior computation unit and a bidirectional cross-attention layer; The physical prior computation unit explicitly establishes the calculus kinematic relationship between velocity and distance, generating pseudo-distance features and pseudo-velocity features. The bidirectional cross-attention layer employs two attention branches for the pseudo-distance and pseudo-velocity features: a distance attention branch and a Doppler attention branch. In each attention branch, bidirectional feature interaction fusion is performed under a unified physical dimension through a cross-modal attention mechanism. Finally, the two attention branches maintain dual-stream parallelism and apply residual connections to preserve the feature attributes of the original modality, resulting in physically semantically aligned Doppler cross-modal fusion features and distance cross-modal fusion features.

[0017] Preferably, the physical prior computation unit and the bidirectional cross-attention layer are specifically as follows: The physical prior calculation unit consists of a trapezoidal integrator and a central differencer. The trapezoidal integrator performs discrete integration of the Doppler spectrum features along the time axis to generate pseudo-distance features with physical distance quantities. The central differencer performs discrete differentiation of the distance spectrum features along the time axis to generate pseudo-velocity features with physical velocity quantities. In the bidirectional cross-attention layer, one attention branch uses the distance spectrum feature as the query vector and the pseudo-distance feature generated by integration as the key and value to calculate cross-attention for feature interaction; the other attention branch uses the Doppler spectrum feature as the query vector and the pseudo-velocity feature generated by differentiation as the key and value to calculate cross-attention for feature interaction.

[0018] Furthermore, the processing procedure of the physical prior calculation unit is as follows: The trapezoidal integrator has Doppler spectral characteristics. Perform trapezoidal integration along the time dimension to generate pseudo-distance features. :

[0019] The central differential converter has range spectrum characteristics. By taking the central difference derivative along the time dimension, pseudo-velocity features are generated. :

[0020] in, Indicates the first The original Doppler spectral characteristics of the frame; Indicates the first The original range-spectral characteristics of the frame; The first part of the integral generated represents the first part of the integral. Pseudo-distance features of frames; The derivative generates the first... Pseudo-velocity characteristics of frames; This indicates the current discrete-time frame index; This indicates the sampling time interval between adjacent radar data frames.

[0021] Furthermore, the processing procedure of the bidirectional cross-attention layer is as follows: For the distance attention branch, the distance spectral features are used as the query vector, the pseudo-distance features generated by integration are used as the key and value, and a residual connection is applied. The calculation formula is as follows:

[0022] For the Doppler attention branch, the Doppler spectral features are used as the query vector, the pseudo-velocity features generated by the derivative are used as keys and values, and residual connections are applied. The calculation formula is as follows:

[0023] in, , These represent the distance cross-modal fusion features and Doppler cross-modal fusion features output by the distance attention branch and the Doppler attention branch, respectively. This indicates the cross-attention calculation operation; , , These represent the query vector, key vector, and value vector in the attention mechanism, respectively. Represents the range spectrum characteristics. Indicates pseudo-distance features, Indicates the characteristics of the Doppler spectrum. This indicates a pseudo-velocity feature.

[0024] Preferably, the dynamic spatiotemporal graph modeling and gating fusion module in S3 includes four sub-layers in sequence according to the data flow direction: dynamic graph learning sub-layer, spatial kinematics sub-layer, frame-by-frame temporal aggregation sub-layer, and gating fusion sub-layer. The dynamic graph learning sublayer calculates the self-attention similarity of joint features based on the initial 3D skeleton coordinate sequence, introduces relative position encoding and edge feature multilayer perceptron, dynamically generates the joint association matrix for each frame, and generates spatial graph features based on the joint association matrix. The frame-by-frame temporal aggregation sublayer splices the skeleton features of three adjacent frames and extracts local temporal features through a multilayer perceptron. The spatial kinematics sublayer uses spatial graph features as queries and local temporal features as keys and values ​​to calculate cross attention, thereby obtaining spatiotemporal cross features; The gated fusion sublayer uses a gated mechanism to proportionally weight and fuse spatial graph features and spatiotemporal intersection features, and applies residual connections to output an optimized skeleton coordinate sequence that repairs distortion.

[0025] Furthermore, the processing procedure of the dynamic graph learning sublayer is as follows:

[0026] in, Represents the joint correlation matrix; Represents relative position encoding; Multilayer perceptron representing edge features; Indicates key points With key points The initial spatial edge features between them are used to generate a topological bias after being mapped by a multilayer perceptron; furthermore... The feature dimensions of the query vector and key vector are used to scale the dot product attention to prevent gradient vanishing. The processing procedure of the frame-by-frame temporal aggregation sublayer is as follows:

[0027] in, Represents local temporal characteristics; Represents the skeleton features of a certain frame. This indicates the current discrete-time frame index. and These represent the skeleton features of the previous and next frames, respectively. This represents a multilayer perceptron; Indicates a splicing operation; The processing procedure for the spatial kinematics sublayer is as follows:

[0028] in, Indicates spatiotemporal intersection characteristics, Representing spatial graph features; Indicates querying the projection matrix. Represents the key projection matrix. Represents the projection matrix of values; The processing procedure of the gated fusion sublayer is as follows:

[0029] in, This represents the optimized skeleton coordinate sequence. Represents gating weights; ⊙ represents element-wise multiplication between tensors; Representation layer normalization.

[0030] Preferably, the decoder in S4 consists of a three-layer fully connected network, and the final 3D key point coordinate sequence is obtained through joint mapping of the fully connected network.

[0031] Preferably, the encoder, the physical prior cross-modal fusion module, the regression head, the dynamic spatiotemporal graph modeling and gating fusion module, and the decoder constitute a millimeter-wave radar human posture estimation model, and the average position error per joint is used as the loss function to train and optimize the millimeter-wave radar human posture estimation model.

[0032] Compared with the prior art, the beneficial effects of the present invention are: (1) The method of this invention proposes a physical prior cross-modal fusion module. This invention innovatively constructs a physical cross-modal fusion module, which includes a physical prior calculation unit and a bidirectional cross-modal attention component. First, the calculus kinematic correlation between heterogeneous radar signals is explicitly established through physical prior calculation; trapezoidal integration is performed on the Doppler spectrum features along the time axis to generate pseudo-range features with range dimensions, while central difference differentiation is performed on the range spectrum features to generate pseudo-velocity features with velocity dimensions; second, bidirectional feature interaction is performed under a unified physical dimension, using the original features of one mode as the query and the physical transformation prior features of another mode as the key and value to calculate cross-attention; finally, the bidirectional attention output is deeply fused through layer normalization and residual connection mechanism. This design effectively eliminates the physical semantic gap between Doppler and range spectrum, guides the allocation of attention weights under a unified physical semantic space, and significantly improves the model's ability to filter radar environmental noise and calibrate features.

[0033] (2) The method of this invention proposes a dynamic spatiotemporal graph modeling and gating fusion module. This invention innovatively constructs a dynamic spatiotemporal graph fusion module, which includes three core components: dynamic spatial graph learning, local temporal fusion, and adaptive spatiotemporal gating interaction. In specific implementation, firstly, instead of using the traditional static predefined skeleton graph, the query-key similarity between the features of each joint is calculated, and the geometric bias of the relative position encoding features and edge features of the multilayer perceptron is explicitly superimposed to dynamically generate a unique joint topology association matrix for each frame to complete the spatial graph propagation; secondly, in the temporal dimension, a sliding window is used to stitch together the features of adjacent frames, and local temporal features containing motion inertia are extracted by the multilayer perceptron; finally, the spatiotemporal cross-domain cross attention calculation is performed with the spatial graph features as the query and the temporal features as the key and value, and an adaptive fusion weight is generated through a gating mechanism composed of a two-layer linear network and a Sigmoid function, which dynamically integrates the spatial features and spatiotemporal cross features proportionally and applies residual connections. This design can adaptively capture complex human anatomical topology and kinematic laws, effectively solving the problems of joint distortion and trajectory drift caused by severe blockage of millimeter-wave radar signals and multipath interference. Attached Figure Description

[0034] Figure 1 This is a structural block diagram of the millimeter-wave radar human posture estimation model in this invention; Figure 2 This is a structural block diagram of the physical prior cross-modal fusion module in this invention; Figure 3 This is a structural block diagram of the dynamic spatiotemporal graph modeling and gating fusion module in this invention; Figure 4 This is a visualization of the millimeter-wave radar human pose estimation model in this invention on the MVDoppler-POse dataset. Figure 4 (a) is the real label. Figure 4 (b) is the prediction result. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1: A millimeter-wave radar human pose estimation method based on physical priors and dynamic spatiotemporal graphs is proposed. The method employs a millimeter-wave radar human pose estimation model, and its network architecture is as follows: Figure 1As shown, the model includes a multi-view feature encoder, a physical prior cross-modal fusion module, an initial regression head, a dynamic spatiotemporal graph modeling and gated fusion module, and a late-stage fusion decoder. The input to this model consists of Doppler and distance spectra from both frontal and side views. These are first fed into an encoder built on a Transformer architecture. The encoder's CNN module extracts local micro-Doppler texture, while the Transformer module extracts global context. In multi-view scenarios, cross-view cross-attention (primarily through a cross-attention module) achieves adaptive alignment and feature complementarity between the frontal and side views. The encoded features are then sequentially fed into the physical prior cross-modal fusion module for physical cross-modal fusion, and then preliminarily decoded into an initial 3D keypoint sequence (17×3 coordinates) by a one-dimensional convolutional regression head. Subsequently, this sequence is fed into the dynamic spatiotemporal graph modeling and gated fusion module for spatiotemporal topology optimization. Finally, the dual-modal optimized skeleton features are fused through a fully connected layer to output high-precision final 3D keypoint coordinates. The method mainly includes the following steps: Step 1: Data preprocessing.

[0037] The data was collected by simultaneously acquiring echo signals from two 77GHz millimeter-wave radars deployed on the front and sides of the human body.

[0038] Specifically, in this embodiment, the dataset comes from two millimeter-wave radars (deployed at the front and side of the human body, respectively), which simultaneously acquire raw intermediate frequency echo signals. The echo data from each radar are used to generate a Doppler spectrum using Short-Time Fourier Transform (STFT) and a range spectrum using Fast Fourier Transform (FFT). The output of this unit is four spectrum data streams: frontal view Doppler spectrum, frontal view range spectrum, side view Doppler spectrum, and side view range spectrum.

[0039] Step 2: Feature extraction.

[0040] First, the Doppler spectrum and distance spectrum are extracted using Short-Time Fourier Transform (STFT) and Two-Dimensional Fast Fourier Transform (2D-FFT), respectively, and then Z-score normalization is performed. Subsequently, the dual-view, dual-modal spectral data is fed into a Transformer encoder with shared weights. After extracting local micro-Doppler texture using a CNN, the data is concatenated along the sequence dimension using a cross-view Transformer, and multi-head self-attention is computed to achieve implicit alignment and feature extraction for both frontal and lateral views.

[0041] The encoder is mainly composed of Transformers, with key modules including a CNN sublayer and a cross-view Transformer sublayer. The CNN sublayer uses 3×3 convolutions to extract local micro-Doppler texture features from the spectrogram; the cross-view Transformer sublayer concatenates the patch sequences from the front and side views along the sequence dimension and calculates joint multi-head self-attention to achieve implicit alignment and complementarity of dual-view features.

[0042] Step 3: Multimodal data feature alignment and fusion.

[0043] (1) The physical prior cross-modal fusion module adopts the physical prior cross-modal fusion mechanism.

[0044] To achieve deep alignment of heterogeneous radar signals at the physical quantity level, the encoded features are fed into a physical prior cross-modal fusion module (Physicsprior module). Utilizing the physical properties of Doppler spectrum representing velocity and range spectrum representing distance, the calculus-kinematic relationship between the two is explicitly established. This physical prior cross-modal fusion module includes a physical prior computation unit and a bidirectional cross-attention layer. The module structure is as follows: Figure 2 As shown, this module explicitly establishes the calculus-kinematic relationship between velocity and distance, specifically by analyzing the Doppler spectral characteristics. Perform a discrete trapezoidal integral along the time axis to generate pseudo-distance features. Simultaneously, for distance spectrum characteristics Perform central difference differentiation along the time axis to generate pseudo-velocity features. .

[0045] The physical prior calculation unit consists of a trapezoidal integrator (DIM) and a range differential module (RDM). The DIM performs discrete integration of the Doppler spectral features along the time axis to generate pseudo-range features with physical distance dimensions. The RDM performs discrete differentiation of the range spectral features along the time axis to generate pseudo-velocity features with physical velocity dimensions.

[0046] A bidirectional cross-attention layer constructs a symmetric cross-modal attention mechanism. On one hand, the distance spectral features are used as the query vector, which is generated by integration. Cross-attention is calculated using keys and values; on the other hand, Doppler spectral features are used as query vectors to generate derivatives. Cross-attention is computed as both key and value. Finally, this module outputs physically semantically aligned distance cross-modal fusion features and Doppler cross-modal fusion features, respectively. These two feature streams are kept in parallel and residual connections are applied to preserve the feature attributes of the original modalities. This design ensures deep interaction between modalities under the premise of unified physical dimensions, while also supporting the subsequent regression head in extracting independent initial skeleton sequences. The specific implementation process is as follows: First, a physical prior computation unit is introduced to perform trapezoidal integration on the Doppler spectral features along the time dimension, generating distance features aligned with physical dimensions, as shown below:

[0047] By taking the central difference derivative of the distance spectrum features along the time dimension, a velocity feature aligned with physical dimensions is generated, as shown below:

[0048] in, Indicates the first The original Doppler spectral characteristics of the frame; Indicates the first The original range-spectral characteristics of the frame; The first part of the integral generated represents the first part of the integral. Pseudo-distance features of frames; The derivative generates the first... Pseudo-velocity characteristics of frames; This indicates the current discrete-time frame index; This indicates the sampling time interval between adjacent radar data frames.

[0049] Subsequently, a bidirectional cross-attention interaction is performed, using the physically converted features to guide the attention weight allocation of the other modality. The two attention outputs are then fused by learnable weights to output cross-modal fusion features.

[0050] Specifically, in bidirectional cross-modal attention fusion (primarily using a cross-attention module), the physically transformed features are used as the keys and values ​​for cross-attention, guiding the query of the other modality to interact with the features. The implementation involves the distance modality's query focusing on the distance features K and V generated by the Doppler integral, and the Doppler modality's query focusing on the velocity features K and V generated by the distance derivative. The fusion process is as follows:

[0051]

[0052] in, , These represent the distance cross-modal fusion features and Doppler cross-modal fusion features output by the distance attention branch and the Doppler attention branch, respectively. This indicates the cross-attention calculation operation; , , These represent the query vector, key vector, and value vector in the attention mechanism, respectively. Represents the range spectrum characteristics. Indicates pseudo-distance features, Indicates the characteristics of the Doppler spectrum. This indicates a pseudo-velocity feature.

[0053] This mechanism guides the allocation of attention weights within a unified physical semantic space, enabling noise filtering and feature fusion.

[0054] (2) Initial regression head.

[0055] After the fused features are initially decoded into an initial 3D skeleton coordinate sequence by a one-dimensional convolutional regression head, they are then input into the dynamic spatiotemporal graph modeling and gating fusion module for spatiotemporal topology optimization.

[0056] In this embodiment, specifically, the initial regression head consists of two independent one-dimensional convolutional networks, each containing two convolutional layers (kernel size 3, stride 1, output channels 51, corresponding to 3D coordinates of 17 key points). The input to this module is the fused features of the above outputs, and the output is the initial 3D skeleton coordinate sequence (…). ).

[0057] (3) The dynamic spatiotemporal graph modeling and gating fusion module adopts the dynamic spatiotemporal graph modeling and gating fusion mechanism.

[0058] This invention employs a dynamic spatiotemporal graph modeling and gating fusion module, designing a dynamic spatiotemporal graph modeling and gating fusion mechanism. Internally, this module comprises four sub-layers arranged sequentially according to the data flow direction: Dynamic Adjacency Learning (DGL), Spatial Kinematic MLP (SKM), Framewise Temporal Aggregator (FTA), and Gated Module. The module structure is as follows: Figure 3 As shown. This module does not use the traditional static predefined skeleton map, but instead performs topology optimization through dynamic joint relationship learning and deep spatiotemporal interaction. The input of this module is the initial 3D skeleton coordinate sequence output by the initial regression head mentioned above, and the output is the optimized skeleton coordinate sequence after distortion repair (the tensor shape remains unchanged). The specific processing procedure is as follows: The dynamic graph learning sublayer calculates the self-attention similarity of joint features, superimposes learnable relative position embeddings and edge feature multilayer perceptrons (Edge MLPs), and dynamically generates the joint association matrix for each frame. .

[0059] The frame-by-frame temporal aggregation sublayer splices the skeleton features of three adjacent frames and extracts local temporal features through a multilayer perceptron (MLP).

[0060] The spatial kinematics sublayer uses spatial graph features as queries. Specifically, these spatial graph features... This is achieved by performing matrix multiplication (i.e., dynamic graph convolution aggregation) between the joint association matrix (representing the dynamic topological weights between joints that change with movement) generated by the Dynamic Graph Learning (DGL) sublayer and the input joint node features. This matrix mapping process enables the explicit aggregation of each joint feature with its global and local neighborhood motion constraint information, transforming isolated node coordinate features into a spatial representation containing high-order human anatomical topology and collaborative rules. Subsequently, this sublayer uses local temporal features as keys and values, and calculates the response weights of the aforementioned spatial topological representation to temporal dynamic changes through a cross-domain attention mechanism, thereby achieving deep semantic refinement and coherent fusion of spatial structural features and temporal evolution features.

[0061] The gated fusion sublayer concatenates spatial features with cross-attention output features, generates gate values ​​through two linear networks and a sigmoid function, and then weights and fuses spatial features and spatiotemporal cross features according to the gate ratio. Finally, residual connections and layer normalization are used.

[0062] The specific implementation process of the above four sub-layers is as follows: This stage does not use a static predefined skeleton. Instead, it dynamically generates a unique joint association matrix for each frame by calculating the spatial similarity of joint features and explicitly overlaying relative position embedding with an edge feature multilayer perceptron (Edge MLP). .

[0063] The first layer is the Dynamic Graph Learning (DGL) sublayer. The core of this module lies in driving the Query-Key similarity calculation of node (joint) features through an initial 3D skeleton coordinate sequence, thereby achieving a leap from low-order geometric space to high-order topological semantics. The specific process is as follows: First, the module receives the initial 3D skeleton coordinate sequence output from the previous step and uses a linear projection layer with shared weights to map its low-dimensional spatial coordinates (i.e., X, Y, Z coordinate values) to a high-dimensional implicit feature space, extracting node features with contextual representation capabilities. Then, through two sets of decoupled linear transformations, the high-dimensional node features are projected into a query matrix and a key matrix, respectively. Next, by performing the dot product operation of the query and key, the intrinsic similarity between joints is directly derived and quantified from the deep features of the 3D skeleton coordinates. The network further explicitly superimposes spatial geometric biases generated by relative position embedding and edge feature multilayer perceptron (Edge MLP) to dynamically generate the joint association matrix of the current frame, providing geometric priors between nodes. The specific process is as follows:

[0064] in, Represents the joint correlation matrix; Represents relative position encoding; Multilayer perceptron representing edge features; Indicates key points With key points The initial spatial edge features between them.

[0065] Then, in the temporal dimension, the features of the preceding and following frames are first concatenated and local temporal features are extracted using MLP, and then the spatial graph features are ( As a query, it focuses on cross-references time-series features. ) as keys and values ​​(Key, Value).

[0066] The spatiotemporal interaction and gating fusion in the frame-by-frame temporal aggregation sublayer specifically involves using a sliding window to stitch together features from three adjacent frames and then extracting local temporal motion features through MLP mapping. Next, based on spatial map features For query, time series characteristics Calculate spatiotemporal cross features for key / value pairs. :

[0067]

[0068] in, Represents local temporal characteristics; Represents the skeleton features of a certain frame. This indicates the current discrete-time frame index. and These represent the skeleton features of the previous and next frames, respectively. This represents a multilayer perceptron; Indicates a splicing operation; Indicates spatiotemporal intersection characteristics, Representing spatial graph features; Indicates querying the projection matrix. Represents the key projection matrix. The value projection matrix represents the projection matrix.

[0069] Finally, spatial and spatiotemporal features are proportionally weighted and fused using a gating mechanism, and residual connections are applied to output the optimized skeleton features after distortion repair. .

[0070] The specific process involves calculating adaptive fusion gating weights using a two-layer linear network and a Sigmoid activation function. By utilizing gated weighting to fuse spatial and spatiotemporal intersection features, and applying residual connectivity and layer normalization, as shown below:

[0071] in, This represents the optimized skeleton coordinate sequence. Represents gating weights; ⊙ represents element-wise multiplication between tensors; Representation layer normalization.

[0072] Step 4: 3D human pose estimation.

[0073] Finally, the optimized Doppler mode and distance mode skeleton features are flattened and stitched together, and late joint mapping is performed through a three-layer fully connected network to output the final high-precision 3D key point coordinate sequence.

[0074] The late-stage fusion decoder consists of a three-layer fully connected network. This module flattens and stitches together the optimized frontal and lateral skeleton features, performs late-stage joint mapping through the fully connected network, and finally outputs a high-precision sequence of 3D human joint coordinates.

[0075] Step 5: Model training.

[0076] During model training, the Adam optimizer was used with the average position error per joint (MPJPE) as the loss function.

[0077] Experimental verification: Based on the above-described invention, the following evaluation of its technical effects is proposed: For performance evaluation, this invention was tested and evaluated on the MVDoppler-Pose dataset, as shown in Table 1.

[0078] Table 1. Metrics in the MVDoppler-Pose dataset

[0079] The baseline model used in this invention is the MVDoppler-pose proposed at CVPR (Conference on Computer Vision and Pattern Recognition) 2025. Ablation experiments were conducted on each of the innovative modules of this invention to verify the effectiveness of each module, as shown in Table 2.

[0080] Table 2 Ablation Experiment Results

[0081] As shown in Tables 1 and 2, the test results demonstrate that the model performs excellently on the MVDoppler-Pose dataset, with a mean joint position error (MPJPE) of 56.8 mm and a keypoint accuracy (PCK) of 94.25%.

[0082] Figure 4 Visualize the MVDoppler-POse dataset, where Figure 4 (a) is the real label. Figure 4 (b) shows the prediction results. It can also be seen that the millimeter-wave radar human posture estimation model in this invention can obtain high-precision 3D joint coordinates, and the final prediction results are consistent with the real labels.

[0083] The above description is only for the purpose of helping to understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, any equivalent substitutions or modifications made to the technical solution and inventive concept disclosed in the present invention within the scope of the technology disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A 3D attitude estimation method for millimeter-wave radar based on physical priors and dynamic spatiotemporal graphs, characterized in that, Includes the following steps: S1. Data preprocessing: Two millimeter-wave radars deployed on the front and sides of the human body are used to simultaneously collect echo signals. The echo data of each radar is used to extract the Doppler spectrum and range spectrum using short-time Fourier transform and fast Fourier transform respectively, so as to obtain dual-view dual-mode spectrum data. S2. Feature extraction: Based on dual-view, dual-modal spectral data, an encoder is used to concatenate the same modal data from different viewpoints along the sequence dimension to obtain the encoded Doppler spectral features and distance spectral features. S3, Multimodal data feature alignment and fusion; The Doppler spectral features represent velocity and the distance spectral features represent distance. The calculus kinematics relationship between velocity and distance is explicitly established using the physical prior cross-modal fusion module, and physical cross-modal fusion is performed to obtain the Doppler cross-modal fusion features and distance cross-modal fusion features after physical semantic alignment. The cross-modal fusion module of the physical prior includes a physical prior computation unit and a bidirectional cross-attention layer; The physical prior calculation unit explicitly establishes the calculus kinematic relationship between velocity and distance, generating pseudo-distance features and pseudo-velocity features; The bidirectional cross-attention layer employs two attention branches for pseudo-range and pseudo-velocity features. In each attention branch, bidirectional feature interaction fusion is performed under a unified physical dimension through a cross-modal attention mechanism. Finally, the two attention branches maintain dual-stream parallelism and apply residual connections to preserve the feature attributes of the original modality, resulting in physically semantically aligned Doppler cross-modal fusion features and distance cross-modal fusion features. The Doppler cross-modal fusion features and the distance cross-modal fusion features are decoded into initial 3D skeleton coordinate sequences for different modalities by a one-dimensional convolutional regression head; The initial 3D skeleton coordinate sequences of different modalities are optimized for spatiotemporal topology using dynamic spatiotemporal graph modeling and gating fusion module. The optimized Doppler modal skeleton features and distance modal skeleton features are obtained through dynamic joint relationship learning and deep spatiotemporal interaction. The dynamic spatiotemporal graph modeling and gating fusion module includes four sub-layers in the order of data flow: dynamic graph learning sub-layer, spatial kinematics sub-layer, frame-by-frame temporal aggregation sub-layer, and gating fusion sub-layer. The dynamic graph learning sublayer calculates the self-attention similarity of joint features based on the initial 3D skeleton coordinate sequence, introduces relative position encoding and edge feature multilayer perceptron, dynamically generates the joint association matrix for each frame, and generates spatial graph features based on the joint association matrix. The frame-by-frame temporal aggregation sublayer splices the skeleton features of three adjacent frames and extracts local temporal features through a multilayer perceptron. The spatial kinematics sublayer uses spatial graph features as queries and local temporal features as keys and values ​​to calculate cross attention, thereby obtaining spatiotemporal cross features; The gated fusion sublayer uses a gated mechanism to proportionally weight and fuse spatial graph features and spatiotemporal intersection features, and applies residual connections to output an optimized skeleton coordinate sequence that repairs distortion. S4, 3D human pose estimation: Flatten and stitch together the Doppler modal skeleton features and the distance modal skeleton features, and obtain the final 3D joint coordinate sequence through joint mapping by the decoder.

2. The millimeter-wave radar 3D attitude estimation method based on physical priors and dynamic spatiotemporal diagrams according to claim 1, characterized in that, The encoder in S2 is specifically as follows: The encoder is composed of several Transformer modules with shared weights stacked together. Each Transformer module includes a CNN sub-layer and a cross-view Transformer sub-layer. The CNN sub-layer uses 3×3 convolution to extract local micro-Doppler textures in the spectrogram. The cross-view Transformer sublayer concatenates the patch sequences from the front and side views along the sequence dimension and computes multi-head self-attention, achieving implicit alignment and feature extraction from the front and side views.

3. The millimeter-wave radar 3D attitude estimation method based on physical priors and dynamic spatiotemporal diagrams according to claim 1, characterized in that, The physical prior computation unit and the bidirectional cross-attention layer are as follows: The physical prior calculation unit consists of a trapezoidal integrator and a central differencer. The trapezoidal integrator performs discrete integration of the Doppler spectrum features along the time axis to generate pseudo-distance features with physical distance quantities. The central differencer performs discrete differentiation of the distance spectrum features along the time axis to generate pseudo-velocity features with physical velocity quantities. In the bidirectional cross-attention layer, one attention branch uses the distance spectrum feature as the query vector and the pseudo-distance feature generated by integration as the key and value to calculate cross-attention for feature interaction; the other attention branch uses the Doppler spectrum feature as the query vector and the pseudo-velocity feature generated by differentiation as the key and value to calculate cross-attention for feature interaction.

4. The millimeter-wave radar 3D attitude estimation method based on physical priors and dynamic spatiotemporal diagrams according to claim 3, characterized in that, The processing procedure for the bidirectional cross-attention layer is as follows: in, , These represent the distance cross-modal fusion features and Doppler cross-modal fusion features output by the distance attention branch and the Doppler attention branch, respectively. This indicates the cross-attention calculation operation; These represent the query vector, key vector, and value vector in the attention mechanism, respectively. Represents the range spectrum characteristics. Indicates pseudo-distance features, Indicates the characteristics of the Doppler spectrum. This indicates a pseudo-velocity feature.

5. The millimeter-wave radar 3D attitude estimation method based on physical priors and dynamic spatiotemporal diagrams according to any one of claims 1-4, characterized in that, The decoder in S4 consists of a three-layer fully connected network. The final 3D key coordinate sequence is obtained through joint mapping of the fully connected network.

6. The millimeter-wave radar 3D attitude estimation method based on physical priors and dynamic spatiotemporal graphs according to claim 5, characterized in that, The encoder, the physical prior cross-modal fusion module, the regression head, the dynamic spatiotemporal graph modeling and gating fusion module, and the decoder constitute the millimeter-wave radar human posture estimation model. The average position error per joint is used as the loss function to train and optimize the millimeter-wave radar human posture estimation model.

Images