Human pose reconstruction method based on distance guided double branch network

Abstract

The application discloses a human pose reconstruction method based on a distance-guided double-branch network. The purpose is to solve the problems of the loss of joint information of slow movement or almost static, the lack of unified range prior between different representations, and the difficulty in effectively combining human motion saliency and structural integrity. The method comprises the following steps: acquiring a multi-channel millimeter wave radar echo signal; estimating human distance prior based on absolute median difference; constructing distance-guided point cloud features and distance-guided heat map features; and based on a double-branch network of point cloud and heat map, and using a cross-modal attention mechanism to fuse the two representations. The application improves the accuracy and robustness of human pose reconstruction of the millimeter wave radar, effectively reduces the loss of human information caused by filtering, and is suitable for scenes such as smart elderly care and smart home.

Description

Technical Field

[0001] This invention relates to the fields of radar signal processing and human posture reconstruction, specifically to a human posture reconstruction method based on a distance-guided dual-branch network. Background Technology

[0002] In smart elderly care scenarios, the safety monitoring of seniors is always one of the core requirements. In home-based elderly care, incidents such as falls, fainting spells, and abnormal behavior among elderly people living alone are sudden, have serious consequences, and require timely rescue. Therefore, the ability to identify and monitor elderly behavior around the clock, quickly, and accurately, is of great significance for realizing smart elderly care.

[0003] Existing behavior recognition technologies mainly include wearable devices, optical cameras, and environmental sensors. Wearable devices (wristbands, accelerometers, etc.) can detect information such as heart rate and movement status, but they often suffer from poor compliance (forgetting to wear them / not wanting to wear them), frequent charging and maintenance, and poor wearing comfort. Optical cameras excel in behavior recognition and fall detection, but there are significant privacy concerns, and the cameras are significantly affected by changes in lighting conditions. Environmental sensors (infrared, ultrasonic, etc.) are relatively inexpensive, but their ability to acquire fine-grained human movement and posture information is limited; they can usually only determine whether a person has moved.

[0004] To address the aforementioned issues, this invention proposes a human pose reconstruction technique based on a distance-guided dual-branch network. It utilizes distance-gated prior information to guide the construction of multimodal features, and then leverages these features through a dual-branch network to achieve high-precision human pose reconstruction. This technique offers advantages such as high privacy protection, all-weather and light-resistant performance, and sensitivity to subtle movements. Summary of the Invention

[0005] The purpose of this invention is to provide a human pose reconstruction method based on a distance-guided dual-branch network, which solves the problems of poor privacy protection, loss of information in the static parts of the human body, insufficient utilization of complementary features of various modes, and low accuracy of human pose reconstruction in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A human pose reconstruction method based on a distance-guided dual-branch network includes the following steps: Step S1: Acquisition and preprocessing of multiple-input multiple-output (MIMO) millimeter-wave radar echo signals; Step S2: Estimate the fluctuation intensity of each distance gate based on the Median Absolute Deviation (MAD) and estimate the human body region based on the seed point and region growth strategy to achieve robust human body center distance estimation.

[0007] Step S3: Based on the Capon algorithm, the incoming wave direction is estimated, and the robustly estimated human center distance is used as a guide to construct point cloud features and heat map features from the radar echo data.

[0008] Step S4: Construct a dual-branch encoding-fusion-decoding neural network to extract spatial saliency information from the point cloud and structural integrity information from the heatmap, respectively. Utilize the fusion network to fuse features and regress the coordinates of human skeletal joints through multi-task loss supervision.

[0009] Preferably, step S1 includes human body range estimation. First, the millimeter-wave radar transmits a linear frequency modulated continuous wave and receives echo signals from each channel. The coherent processing time (CPI) includes M pulse repetition intervals (PRI). The time-domain transmitted waveform corresponding to the m-th PRI is: Among them, fast time , For chirp duration, The amplitude of the transmitted signal, For the center frequency, For frequency modulation slope, The imaginary unit, This is the initial phase of the launch.

[0010] If we model the human body as P scattering points, then the signal in the m-th chirp is written as:

[0011] in For the chirp repetition period, It is the propagation delay at the p-th scattering point. Where c is the distance to the target and c is the speed of light; The Doppler frequency shift of the p-th target, where It is the target radial velocity. It is the signal wavelength.

[0012] FMCW receiver mixing and descrambling:

[0013] Substitute equations (1) and (2) into equation (3) and expand the phase, ignoring... The smaller second-order term, neglecting the chirp inner Doppler term under low-speed target conditions, yields:

[0014] Considering multiple channels of the radar, we have:

[0015] Discrete sampled data cube:

[0016] Performing a fast Fourier transform, we get:

[0017] Combined exponent terms (related only to n):

[0018] Substituting equation (8) into equation (7), we have:

[0019] Preferably, step S2, which estimates the fluctuation intensity of each distance gate based on the median absolute deviation, specifically includes performing incoherent energy accumulation on the channel dimension:

[0020] Ignoring the noise term, we have:

[0021] Divide the expression into terms p=r and p≠r:

[0022]

[0023] Therefore, it should be written as:

[0024] in For the noise term, for each pair (p,r) in equations (12) and (13), let:

[0025] but:

[0026] Since P[m,k] is real-valued energy, it can be equivalently written in the form of cosine superposition:

[0027] Since the human body is a continuous target, it often contains multiple scattering points in the vicinity of a range cell. Their micro-Doppler values ​​are different, causing the cross term to fluctuate quasi-periodically with slow time m. The static background usually lacks this oscillation, so it is more stable.

[0028] To further measure the degree of fluctuation, we introduce:

[0029] Human movement can produce occasional large movements that generate instantaneous strong scattering, which can introduce outliers. MAD, as a robust statistic, is less sensitive to outliers and is more stable than variance and standard deviation.

[0030] At this point, B[m,k]≈0 for the static background, then P[m,k]≈A(k), which is approximately a constant, thus making S(k)≈0; while for the human target, there is B[m,k], so P[m,k] will fluctuate with m, and S(k) will increase significantly.

[0031] To reduce the peak values ​​caused by random noise in the distance dimension, the project further performs local smoothing on S(k) in the distance dimension.

[0032]

[0033] Where 2L+1 is the length of the smoothing window.

[0034] Since the human body is a continuous target, its scattering points are continuously distributed in the distance dimension. Therefore, the human body target corresponds to... This often results in continuous high-response intervals rather than isolated single-point peaks. Based on this, the project uses a seed point plus region growing method to obtain the radial range of the human body.

[0035] Within the preset effective distance window Within the range, the point with the largest fluctuation is selected as the seed:

[0036] Set the growth threshold:

[0037] And the interval is obtained by expanding outwards from k0. :

[0038] After obtaining the radial range of the human body, in order to obtain a robust target center position, this paper focuses on the interval... Internal basis Construct the weights and calculate the weighted median center. Let:

[0039] And define the normalized cumulative distribution function (CDF):

[0040] The radial center of the human body is defined as:

[0041] The center is not a simple geometric midpoint, but rather a position where the cumulative fluctuations on both sides of the center each account for about 50%, thus making it less sensitive to local strong peaks or outlier noise.

[0042] Preferably, step S3, the distance-guided estimation of the direction of arrival based on the Capon algorithm, includes the construction of point cloud and heat map, with one branch performing point cloud extraction and the other branch performing azimuth-elevation-range matrix calculation.

[0043] The point cloud path first performs moving target detection:

[0044] Perform FFT on the slow time dimension:

[0045] Channel synthesis yields the RD diagram:

[0046] Using the prior-obtained human body center distance Only before and after it The angle spectrum is calculated within a distance window of meters to reduce computational cost and minimize interference from distant objects.

[0047] The set of distance indices corresponding to the discrete distance axis r(k):

[0048] Conversion relationship between Doppler elements and velocity:

[0049] During point cloud generation, for a certain distance cell Take the Doppler unit with the highest energy in the RD diagram:

[0050] In a fixed distance unit Above, each chirp gives an array observation vector:

[0051] Where Q represents the number of virtual channels used. Next, a windowed local average is performed:

[0052] To avoid ill-conditioned covariance matrices and improve the stability of inverse matrices:

[0053] For a certain candidate angle The guide vector is:

[0054] The Capon (MVDR) spatial spectrum is defined as follows:

[0055] For each distance unit Calculate all angles to form a three-dimensional spectrum:

[0056] For each distance unit Searching for the largest spectral peak:

[0057] Transform the peak point to a Cartesian coordinate system, where x represents left and right, y represents depth, and z represents up and down:

[0058] The adjustment is as follows:

[0059] The final point cloud features are saved as follows:

[0060] The heatmap path uses non-MTI data, within a fixed distance cell. Above, each chirp gives an array observation vector:

[0061] Where Q represents the number of virtual channels used. Next, a windowed local average is performed:

[0062] To avoid ill-conditioned covariance matrices and improve the stability of inverse matrices:

[0063] For a certain candidate angle The guide vector is:

[0064] The Capon (MVDR) spatial spectrum is defined as follows:

[0065] For each distance unit Calculate all angles to form a three-dimensional spectrum:

[0066] Preferably, step S4 utilizes a dual-branch neural network to reconstruct the human posture. The invention employs a dual-branch encoding, fusion, and decoding architecture. The network input includes point cloud data and an azimuth-pitch energy map, and the output is the two-dimensional coordinates of 15 joints of the human body.

[0067] The azimuth-pitch branch uses the distance dimension as an input channel, enabling the network to adaptively learn the fusion method of different distance information. Data is first processed through a 3D convolutional projection layer for channel transformation and spatial downsampling, then fed into a three-layer encoder to extract multi-scale spatiotemporal features. Each encoder layer consists of a Basic Block 3D, containing two 3×3×3 3D convolutions, batch normalization, and ReLU activation, with residual connections mitigating gradient vanishing. After the encoder output, the temporal dimension is compressed through temporal fusion convolutions, resulting in 2D feature maps at three scales.

[0068] The point cloud branch employs an improved PointNet++ architecture. To facilitate fusion and complementation with the azimuth-elevation branch across different spatiotemporal scales, this branch utilizes a three-layer encoder. Each encoder layer includes a Set Abstraction layer, sequentially performing farthest point sampling, spherical neighborhood lookup, local feature extraction, and local max pooling to output point cloud features at different scales. The three encoder layers reduce the number of points from 128 to 64, 32, and 16, respectively, while increasing the feature dimension from 64 to 128 and 256. After global max pooling, the outputs from each layer are integrated using an independent temporal fusion module, ultimately projecting the data into three scale features that match the azimuth-elevation branch.

[0069] The feature fusion module employs a multi-scale cross-self-attention mechanism. First, at each scale, azimuth-pitch features and point cloud features undergo cross-attention and self-attention calculations respectively, capturing cross-modal and intra-modal correlations. The attention output is concatenated with the original features via residual concatenation and then fed into the decoder for feature fusion and upsampling. The decoder consists of 2D residual blocks and upsampling, restoring spatial resolution through multi-layer decoding.

[0070] Next, the main branch extracts differentiable joint coordinates from the heatmap using Soft-Argmax; the auxiliary branch uses a pose graph convolutional network to constrain the structural relationships between joints using the adjacency matrix of the human skeleton, and outputs a joint heatmap.

[0071] The loss function employs a multi-task learning strategy and comprises three components: coordinate regression loss as the primary supervision; GCN heatmap loss providing joint-level heatmap supervision; and YOLO auxiliary loss providing coarse human localization constraints in intermediate layers. The total loss is a weighted sum of these three components.

[0072] The dataset is divided into training, validation, and test sets with a ratio of 70%, 15%, and 15%, respectively. The Adam optimizer is used with an initial learning rate of 0.0001, a batch size of 10, and 100 training epochs.

[0073] Beneficial effects 1. Addressing the problems of privacy leaks, insufficient compliance, poor wearing comfort, frequent charging and maintenance, susceptibility to light exposure, and limited ability to perceive fine-grained human posture in existing behavioral monitoring methods, this invention employs MIMO millimeter-wave radar as the sensing front end. Since the radar signal does not directly image, it fundamentally avoids the privacy leak risks associated with image data collection. Furthermore, it is unaffected by environmental factors such as changes in lighting, darkness at night, or smoke obstruction, achieving stable operation around the clock. Compared to wearable devices such as wristbands and accelerometers, this method is a non-contact sensing method, requiring no active cooperation from the elderly, thus avoiding problems such as insufficient compliance, poor wearing comfort, and frequent charging and maintenance.

[0074] 2. To address the issue of information loss in static areas such as the head and torso caused by MTI filtering in existing moving target detection methods, this invention proposes a robust human center distance estimation method based on Median Absolute Deviation (MAD) and a seed point region growth strategy. Utilizing the physical characteristic that the micro-Doppler cross terms between continuous scattering points of the human body generate quasi-periodic fluctuations over slow time, while the static background lacks such oscillations, MAD characterizes the fluctuation intensity of each distance gate. This method can achieve radial localization of the human body without relying on MTI processing, preserving complete echoes of static human body parts for subsequent heatmap branching, eliminating static part information loss at the source, and providing a reliable distance prior for high-precision attitude reconstruction.

[0075] 3. To address the problems of insufficient utilization of complementary features of various modes, severe far-field clutter interference, high computational complexity, and low accuracy of human pose reconstruction in existing methods, this invention constructs a distance-guided dual-branch encoding-fusion-decoding network. Using robustly estimated human center distance as prior information, the angle spectrum is calculated only within the target's distance window using the Capon (MVDR) algorithm. Point cloud features and azimuth-pitch energy maps are constructed separately, significantly reducing the computational cost of the 3D spectrum and effectively suppressing far-field clutter and multi-target interference. The point cloud branch uses an improved PointNet++ to extract motion saliency information, while the azimuth-pitch branch uses 3D convolution with the distance dimension as the input channel to achieve adaptive distance fusion and extract structural integrity information. The two feature paths simultaneously model cross-modal and intra-modal associations at three scales through a multi-scale cross-self-attention mechanism, fully exploiting the complementarity of point clouds and heatmaps. At the decoding end, a pose graph convolutional network (GCN) based on human skeleton adjacency matrix constraints is used for parallel output. Combined with a multi-task learning strategy consisting of coordinate regression loss, GCN heatmap loss, and YOLO coarse localization auxiliary loss, the network parameters are optimized collaboratively from both joint-level fine supervision and human coarse localization constraints. On a real-world dataset containing 48,600 samples across 3 testers and 9 behaviors, our method achieved a COCO AP of 62.36%, a significant improvement over existing methods. Attached Figure Description

[0076] Figure 1 A schematic diagram of the overall process of the method of the present invention; Figure 2 The flowchart of the robust human body center estimation algorithm established in this invention; Figure 3 A schematic diagram illustrating the problem of information loss in static parts of the human body caused by MTI, which is addressed by this invention; Figure 4 A schematic diagram of the dual-branch network structure of this invention; Figure 5 1. A schematic diagram of human behavior in this invention; Figure 6 The figures shown are actual images of the experimental setup for this invention. Detailed Implementation

[0077] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0078] like Figure 1 As shown, the present invention provides a human pose reconstruction method based on a distance-guided dual-branch network, comprising: Preferably, step S1 includes human body range estimation. First, the millimeter-wave radar transmits a linear frequency modulated continuous wave and receives echo signals from each channel. The coherent processing time (CPI) includes M pulse repetition intervals (PRI). The time-domain transmitted waveform corresponding to the m-th PRI is: Among them, fast time , For chirp duration, The amplitude of the transmitted signal, For the center frequency, For frequency modulation slope, The imaginary unit, This is the initial phase of the launch.

[0079] If we model the human body as P scattering points, then the signal in the m-th chirp is written as:

[0080] in For the chirp repetition period, It is the propagation delay at the p-th scattering point. Where c is the distance to the target and c is the speed of light; The Doppler frequency shift of the p-th target, where It is the target radial velocity. It is the signal wavelength.

[0081] FMCW receiver mixing and descrambling:

[0082] Substitute equations (49) and (50) into equation (51) and expand the phase, ignoring... The smaller second-order term, neglecting the chirp inner Doppler term under low-speed target conditions, yields:

[0083] Considering multiple channels of the radar, we have:

[0084] Discrete sampled data cube:

[0085] Performing a fast Fourier transform, we get:

[0086] Combined exponent terms (related only to n):

[0087] Substituting equation (8) into equation (7), we have:

[0088] Preferably, the step S2, which estimates the fluctuation intensity of each distance gate based on the median absolute deviation, is as follows: Figure 2 As shown, incoherent energy accumulation is performed along the channel dimension:

[0089] Ignoring the noise term, we have:

[0090] Divide the expression into terms p=r and p≠r:

[0091]

[0092] Therefore, it should be written as:

[0093] in For the noise term, for each pair (p,r) in equations (12) and (13), let:

[0094] but:

[0095] Since P[m,k] is real-valued energy, it can be equivalently written in the form of cosine superposition:

[0096] like Figure 3 As shown, since the human body is a continuous target, it often contains multiple scattering points in the vicinity of a range cell. Their micro-Doppler values ​​are different, causing the cross term to fluctuate quasi-periodically with slow time m. The static background usually lacks this oscillation, so it is more stable.

[0097] To further measure the degree of fluctuation, we introduce:

[0098] Human movement can produce occasional large movements that generate instantaneous strong scattering, which can introduce outliers. MAD, as a robust statistic, is less sensitive to outliers and is more stable than variance and standard deviation.

[0099] At this point, B[m,k]≈0 for the static background, then P[m,k]≈A(k), which is approximately a constant, thus making S(k)≈0; while for the human target, there is B[m,k], so P[m,k] will fluctuate with m, and S(k) will increase significantly.

[0100] To reduce the peak values ​​caused by random noise in the distance dimension, the project further performs local smoothing on S(k) in the distance dimension.

[0101]

[0102] Where 2L+1 is the length of the smoothing window.

[0103] Since the human body is a continuous target, its scattering points are continuously distributed in the distance dimension. Therefore, the human body target corresponds to... This often results in continuous high-response intervals rather than isolated single-point peaks. Based on this, the project uses a seed point plus region growing method to obtain the radial range of the human body.

[0104] Within the preset effective distance window Within the range, the point with the largest fluctuation is selected as the seed:

[0105] Set the growth threshold:

[0106] And the interval is obtained by expanding outwards from k0. :

[0107] After obtaining the radial range of the human body, in order to obtain a robust target center position, this paper focuses on the interval... Internal basis Construct the weights and calculate the weighted median center. Let:

[0108] And define the normalized cumulative distribution function (CDF):

[0109] The radial center of the human body is defined as:

[0110] The center is not a simple geometric midpoint, but rather a position where the cumulative fluctuations on both sides of the center each account for about 50%, thus making it less sensitive to local strong peaks or outlier noise.

[0111] Preferably, step S3, the distance-guided estimation of the direction of arrival based on the Capon algorithm, includes the construction of point cloud and heat map, with one branch performing point cloud extraction and the other branch performing azimuth-elevation-range matrix calculation.

[0112] The point cloud path first performs moving target detection:

[0113] Perform FFT on the slow time dimension:

[0114] Channel synthesis yields the RD diagram:

[0115] Using the prior-obtained human body center distance Only before and after it The angle spectrum is calculated within a distance window of meters to reduce computational cost and minimize interference from distant objects.

[0116] The set of distance indices corresponding to the discrete distance axis r(k):

[0117] Conversion relationship between Doppler elements and velocity:

[0118] During point cloud generation, for a certain distance cell Take the Doppler unit with the highest energy in the RD diagram:

[0119] In a fixed distance unit Above, each chirp gives an array observation vector:

[0120] Where Q represents the number of virtual channels used. Next, a windowed local average is performed:

[0121] To avoid ill-conditioned covariance matrices and improve the stability of inverse matrices:

[0122] For a certain candidate angle The guide vector is:

[0123] The Capon (MVDR) spatial spectrum is defined as follows:

[0124] For each distance unit Calculate all angles to form a three-dimensional spectrum:

[0125] For each distance unit Searching for the largest spectral peak:

[0126] Transform the peak point to a Cartesian coordinate system, where x represents left and right, y represents depth, and z represents up and down:

[0127] The adjustment is as follows:

[0128] The final point cloud features are saved as follows:

[0129] The heatmap path uses non-MTI data, within a fixed distance cell. Above, each chirp gives an array observation vector:

[0130] Where Q represents the number of virtual channels used. Next, a windowed local average is performed:

[0131] To avoid ill-conditioned covariance matrices and improve the stability of inverse matrices:

[0132] For a certain candidate angle The guide vector is:

[0133] The Capon (MVDR) spatial spectrum is defined as follows:

[0134] For each distance unit Calculate all angles to form a three-dimensional spectrum:

[0135] Preferably, step S4 utilizes a dual-branch neural network to reconstruct the human pose. For example... Figure 4 As shown, the invention employs a dual-branch encoding, fusion, and decoding architecture. The network input includes point cloud data and azimuth-pitch energy maps, and the output is the two-dimensional coordinates of 15 joints of the human body.

[0136] The azimuth-pitch branch uses the distance dimension as an input channel, enabling the network to adaptively learn the fusion method of different distance information. Data is first processed through a 3D convolutional projection layer for channel transformation and spatial downsampling, then fed into a three-layer encoder to extract multi-scale spatiotemporal features. Each encoder layer consists of a Basic Block 3D, containing two 3×3×3 3D convolutions, batch normalization, and ReLU activation, with residual connections mitigating gradient vanishing. After the encoder output, the temporal dimension is compressed through temporal fusion convolutions, resulting in 2D feature maps at three scales.

[0137] The point cloud branch employs an improved PointNet++ architecture. To facilitate fusion and complementation with the azimuth-elevation branch across different spatiotemporal scales, this branch utilizes a three-layer encoder. Each encoder layer includes a Set Abstraction layer, sequentially performing farthest point sampling, spherical neighborhood lookup, local feature extraction, and local max pooling to output point cloud features at different scales. The three encoder layers reduce the number of points from 128 to 64, 32, and 16, respectively, while increasing the feature dimension from 64 to 128 and 256. After global max pooling, the outputs from each layer are integrated using an independent temporal fusion module, ultimately projecting the data into three scale features that match the azimuth-elevation branch.

[0138] The feature fusion module employs a multi-scale cross-self-attention mechanism. First, at each scale, azimuth-pitch features and point cloud features undergo cross-attention and self-attention calculations respectively, capturing cross-modal and intra-modal correlations. The attention output is concatenated with the original features via residual concatenation and then fed into the decoder for feature fusion and upsampling. The decoder consists of 2D residual blocks and upsampling, restoring spatial resolution through multi-layer decoding.

[0139] Next, the main branch extracts differentiable joint coordinates from the heatmap using Soft-Argmax; the auxiliary branch uses a pose graph convolutional network to constrain the structural relationships between joints using the adjacency matrix of the human skeleton, and outputs a joint heatmap.

[0140] The loss function employs a multi-task learning strategy and comprises three components: coordinate regression loss as the primary supervision; GCN heatmap loss providing joint-level heatmap supervision; and YOLO auxiliary loss providing coarse human localization constraints in intermediate layers. The total loss is a weighted sum of these three components.

[0141] Preferably, such as Figure 5 As shown, a dataset of 9 behaviors was constructed, containing test results from 3 testers. The training set has 34,020 samples, the validation set has 7,290 samples, and the test set has 7,290 samples. The test scenarios are as follows. Figure 6As shown. The Adam optimizer was used, with an initial learning rate of 0.0001, a batch size of 10, 100 training epochs, and an L2 regularization parameter of 0.0001. The training environment consisted of an NVIDIA A100 graphics card.

[0142] Preferably, as shown in Table 1, during the verification phase, the accuracy of each joint and the overall accuracy are obtained using the COCO AP evaluation index. Experimental results show that the proposed method achieves an AP of 62.36% on the actual test dataset. This represents a significant improvement over existing methods and effectively enables human pose reconstruction.

[0143] Table 1

[0144] The above specific embodiments only illustrate the technical principles of the present invention. The module names and parameter settings in this description can be adjusted according to actual application scenarios and are not limited thereto. Those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments, and such modifications and substitutions do not depart from the technical spirit and protection scope of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A human pose reconstruction method based on distance-guided dual-branch network, characterized in that, Includes the following steps: Step S1: The millimeter-wave radar transmits a linear frequency modulated continuous wave, receives echo signals from each channel, and mixes the echo signals to obtain a deskewing signal; Fourier transform is performed along the fast time dimension to map the radar echo data to the range domain, resulting in a three-dimensional data matrix, whose three dimensions correspond to slow time sampling, range cells, and channels, respectively. Step S2: The human body region is calculated from the radar echo. First, incoherent energy accumulation is performed on the three-dimensional data matrix in the channel dimension. Then, MAD statistics are performed on the slow time dimension under each range gate to obtain the fluctuation intensity of each range gate. Next, the fluctuation intensity is locally smoothed. Based on the smoothed fluctuation intensity, the radial distance range of the human target is extracted using a seed point and region growth strategy. Finally, a weighted median estimate is performed on this range to obtain a robust estimate of the human body center distance. Step S3: Use a fixed number of distance gates before and after the estimated center distance as the human feature construction region. Within this region, construct two types of human representations: the first is a point cloud representation, and the second is a heat map representation. Step S4: Construct a dual-branch neural network. The heatmap branch uses 3D convolution to extract multi-scale spatiotemporal features, while the point cloud branch uses hierarchical set abstraction to extract multi-scale geometric features. Cross-modal interaction is performed through the feature fusion module, and multi-loss supervised training is performed with synchronously acquired optical labels to output the coordinates of human skeleton nodes.

2. The method as described in claim 1, characterized in that, In step S1, the transmitting signal carrier frequency of the linear frequency modulation continuous wave is , the frequency modulation slope is , and the pulse width is ; the de-skewed beat frequency signal is a function of fast time , slow time , and the receiving channel , and the expression is: ； in, The total number of scattering points. For the first Complex scattering coefficients at each scattering point For the first Beat frequency of each scattering point For the first Round-trip delay at each scattering point For the target distance, At the speed of light, For the first Doppler frequencies at each scattering point Radial velocity, For wavelength, The pulse repetition period, It is the imaginary unit.

3. The method as described in claim 1, characterized in that, In step S2, the expression for incoherent energy accumulation in the channel dimension is: ； in For slow time indexing, For distance cell index, For the receive channel index; The median absolute deviation of the distance-slow time power spectrum along the slow time dimension is statistically analyzed to obtain the expression for the fluctuation intensity of each distance cell: ； The expression for locally smoothing the wave intensity along the distance dimension is: ； The radial distance interval is weighted and the median is estimated to obtain the expression for the distance to the center of the human body: 。 4. The method as described in claim 1, characterized in that, In step S3, the method for constructing the point cloud representation is as follows: The radar 3D data matrix is ​​subjected to moving target display filtering. Then, Capon beamforming is performed on each range cell within the human feature construction area to obtain the azimuth-elevation-range 3D spectral matrix. Two-dimensional spectral peak detection is performed on the 3D spectral matrix along the range cell to extract the azimuth angle, elevation angle and spectral power in each range cell. The detected peak coordinates are mapped from the spherical coordinate system to the Cartesian coordinate system to obtain the 3D point cloud.

5. The method as described in claim 1, characterized in that, In step S3, the method for constructing the heatmap representation is as follows: Without applying target display filtering to the radar 3D data matrix, Capon beamforming is performed on each range cell within the human feature construction area to obtain the azimuth-elevation 2D spectrum. The azimuth-elevation spectra of each range cell are stacked along the range dimension to construct an azimuth-elevation-range heatmap.

6. The method as described in claim 1, characterized in that, In step S4, the method for extracting the heatmap features is as follows: The distance dimension of the heatmap is replaced with the channel dimension to preserve spatial information. After three-dimensional convolutional projection, it is input into a multi-scale three-dimensional residual coding network. In three stages, the spatial resolution is gradually reduced and the receptive field is expanded to obtain a multi-scale three-dimensional feature set. Convolutional aggregation is performed on the three-dimensional features at each scale along the time dimension to output multi-scale heatmap features.

7. The method as described in claim 1, characterized in that, In step S4, the method for extracting the point cloud features is as follows: The point cloud representation is input into the network, and the center point is selected by sampling the farthest point. Based on this, the neighborhood features are aggregated by sharing a multilayer perceptron and max pooling to obtain a multi-scale center point feature set. Max pooling is performed along the center point dimension to obtain frame-level feature vectors for the center point features at each scale. The frame-level feature vectors of multiple consecutive frames are concatenated in time order and then fused temporally through a fully connected layer to output multi-scale point cloud features.

8. The method as described in claim 1, characterized in that, In step S4, the fusion method is as follows: The multi-scale point cloud features are extended to the same spatial dimension as the heatmap features. At each scale, the point cloud’s own attention features, the heatmap’s self-attention features, the point cloud-guided heatmap’s cross-attention features, and the heatmap-guided point cloud’s cross-attention features are calculated. The self-attention features, cross-attention features, and features from the previous scale are concatenated along the channel dimension to obtain the fused feature result.

9. The method as described in claim 1, characterized in that, In step S4, the multiple losses are as follows: The primary monitoring signal is coordinate regression loss. ； in, The total number of key points. For the first Predicted coordinates of each key point For the first The actual coordinates of each key point; The heatmap loss generated by the graph convolutional network is used as a structural constraint: ； in, The output of the graph convolutional network is the first... Predicted heatmap of key points For the first The actual heatmap of each key point is a binary cross-entropy function. Spatial constraint based on the loss of human presence in the intermediate layer: ； in, , The feature map space size, A label for the actual existence of the human body. To predict the probability of human existence; The weighted summation constitutes the total loss function: ； in, , , The weights are used as coefficients, and the total loss function is used to train the dual-branch neural network to output the coordinates of human skeletal nodes.

Images