A human posture estimation method based on radar point cloud imaging and multi-dimensional feature fusion

Abstract

The application discloses a human posture estimation method based on radar point cloud imaging and multi-dimensional feature fusion, relates to the cross technical field of computer vision and radar signal processing, and comprises three key technical links: firstly, target resolution is improved through spatial energy distribution estimation, a least square estimation algorithm is used to reconstruct the three-dimensional spatial distribution of the target by utilizing the positive correlation between radar signal energy and target reflection area; secondly, a structured multi-dimensional point cloud matrix is constructed, and a distance-velocity layered sorting strategy is used to convert sparse radar point clouds into high information density image representation; finally, a multi-dimensional feature fusion posture estimation network is designed, three-dimensional convolution, a multi-head attention mechanism and a gated recurrent unit are integrated, and the collaborative extraction of space-time features is realized. The application solves the problems of sparse target features, noise sensitivity and poor universality in traditional millimeter wave radar posture estimation.

Description

Technical Field

[0001] This invention relates to the field of computer vision and radar signal processing, and in particular to a method for human pose estimation based on radar point cloud imaging and multidimensional feature fusion. Background Technology

[0002] Human pose estimation is an important research area in computer vision and sensor perception, aiming to reconstruct the three-dimensional spatial position of human joints by acquiring target motion information. Traditional vision-based sensor methods have made significant progress, often relying on deep learning techniques, such as extracting the three-dimensional skeletal structure and joint information of the human body from two-dimensional images. However, traditional vision-based pose estimation methods often exhibit low robustness and accuracy when faced with occlusion, suboptimal lighting conditions, or viewpoint distortion. Furthermore, vision-based pose estimation methods may not only involve privacy breaches but also face risks such as image tampering and malicious surveillance.

[0003] The electromagnetic waves emitted by millimeter-wave radar, after reflection from the target, contain the target's motion characteristics. Therefore, target attitude information can be estimated by analyzing the reflected signals. In recent years, attitude estimation techniques based on millimeter-wave radar point clouds have become a research focus in the field of attitude estimation. As a non-contact sensing technology, millimeter-wave radar has unique advantages. First, millimeter-wave signals contain more dimensional information than vision, such as velocity and distance, enabling millimeter-wave radar to obtain more robust three-dimensional coordinates for attitude estimation. Second, millimeter-wave radar is unaffected by changes in lighting conditions, allowing point cloud construction systems based on millimeter-wave radar to operate in low-light or nighttime environments, overcoming the limitations of traditional visual methods in low-light environments.

[0004] Although millimeter-wave radar has broad application prospects in attitude estimation, attitude estimation methods based on millimeter-wave radar currently face two problems. First, the target features captured by millimeter-wave radar are too sparse and susceptible to noise interference; second, traditional prediction models utilize a single feature dimension, have limited recognition capabilities, and cannot fully utilize the multi-dimensional features of millimeter-wave radar. Therefore, they can only provide high estimation accuracy under specific actions and lack universality. Summary of the Invention

[0005] To improve the noise resistance and universality of millimeter-wave radar attitude estimation, this invention proposes a target attitude estimation method based on radar point cloud imaging and multi-dimensional feature fusion. This method primarily addresses the technical problems of insufficient resolution, poor universality, and low information utilization in millimeter-wave radar point cloud-based attitude estimation methods. Based on the relationship between radar signal energy and target reflective area, this invention proposes a spatial estimation-based target detection method to enhance the resolution of millimeter-wave radar when sensing targets. Simultaneously, leveraging the multi-dimensionality of radar signal features, this invention constructs a multi-dimensional target point cloud image matrix, increasing information density and enhancing noise resistance, effectively reducing the prediction difficulty of millimeter-wave radar attitude estimation. Finally, employing a multi-dimensional feature fusion attitude estimation network further improves information utilization efficiency, enhances the universality of attitude estimation, and effectively reduces the computational cost of network training.

[0006] Therefore, the present invention adopts the following technical solution:

[0007] This invention provides a method for human pose estimation based on radar point cloud imaging and multi-dimensional feature fusion, comprising the following steps:

[0008] Transmit and receive frequency-modulated continuous wave signals via a radar array;

[0009] The received signal is processed by fast Fourier transform in the range dimension and fast Fourier transform in the velocity dimension to obtain the range-velocity two-dimensional energy matrix of the target; based on the range-velocity two-dimensional energy matrix, the three-dimensional spatial energy distribution of the target is reconstructed by the least squares estimation algorithm, and the effective target area is filtered by the exponential threshold function;

[0010] The effective target region is sorted by distance-velocity hierarchy to construct a three-channel point cloud image matrix containing distance, velocity, and energy information;

[0011] The point cloud image matrix is ​​input into a multi-dimensional feature fusion network for processing. The multi-dimensional feature fusion network sequentially extracts local spatial features through a three-dimensional convolution module, captures global feature relationships through a multi-head attention mechanism module, and extracts temporal motion features through a gated recurrent unit module, and finally outputs the skeletal point coordinates of the target pose.

[0012] Furthermore, the received signal is processed by range-dimensional Fast Fourier Transform and velocity-dimensional Fast Fourier Transform to obtain the target's range-velocity two-dimensional energy matrix, including:

[0013] After processing the received echo signal into an intermediate frequency signal, the intermediate frequency signal is subjected to fast Fourier transform in the range dimension and fast Fourier transform in the velocity dimension to obtain the range-velocity matrix of the target range and velocity.

[0014] By selecting target points and combining different virtual antennas at the target points along the horizontal and vertical directions, and then performing a fast Fourier transform in the angle dimension, the coordinates of the target in two vertical directions at the same distance and speed can be obtained.

[0015] The energy distribution of the target in space is estimated by using the distribution results in the horizontal and vertical directions, resulting in a two-dimensional energy matrix of the target in terms of range and velocity.

[0016] Furthermore, the energy distribution of the target in space is estimated using the distribution results in the horizontal and vertical directions, including:

[0017] For an antenna sequence at a given selection point The subsequences corresponding to the horizontal and vertical directions are: , Then, perform Fast Fourier Transform on these two subsequences respectively to obtain the frequency domain representations of the horizontal and vertical angles: ;in, and These represent the frequency domain distributions of the horizontal and vertical angles, respectively.

[0018] Convert the frequency domain result to amplitude: ;in , The frequency domain distribution modulus of the elevation angle represents the energy distribution of the target in the horizontal and vertical directions at a certain distance and velocity.

[0019] Furthermore, based on the aforementioned range-velocity two-dimensional energy matrix, a least squares estimation algorithm is used to reconstruct the three-dimensional spatial energy distribution of the target, including: the energy distribution of the target in the horizontal and vertical directions. , The sizes are respectively n , m At this distance and speed, the overall energy distribution of the target in space is as follows: Then X and , The following relationship exists:

[0020] ;

[0021] Where X is the spatial energy distribution matrix; the summation of the elements in the rows and columns of X yields a coefficient matrix. , Each row of this vector contains 1s and 0s; that is:

[0022] ;

[0023] Among them, for Each of its actions includes m Row vectors of 1s ,for Each of its actions contains n Row vectors of 1s By multiplying the coefficient matrix A with X, the sum of the rows and columns is obtained. , :

[0024] ;

[0025] From this underdetermined system of equations, the spatial energy distribution matrix X is estimated and solved using the least squares method: .

[0026] Furthermore, the exponential threshold function filters the effective target region, including:

[0027] The target reflective area is predicted based on the received energy peak value. g The spatial energy distribution matrix X is used to detect the target's range using an exponential threshold function f. G :

[0028] ;

[0029] Where P, Q, and C are the coefficients of the function. G This represents the percentage of the detected target area relative to the size of matrix X. During detection, targets are sorted downwards according to their peak energy, until the detected area reaches a certain percentage. G .

[0030] Furthermore, a multi-dimensional target point cloud image matrix construction method is employed to perform range-velocity hierarchical sorting of the effective target region, constructing a three-channel point cloud image matrix containing range, velocity, and energy information, including:

[0031] After detecting the target range at different distances and speeds, G Sort by distance and speed, first sorting by distance in ascending order; if distances are the same, then sorting by speed in descending order; finally, all target selection points will be arranged by distance from nearest to farthest, and by speed from largest to smallest for the same distance. The multidimensional feature matrix S, s This represents the total number of target points selected in a radar transmission frame.

[0032] Merge the multidimensional feature matrices S along the first dimension for each pixel in the point cloud matrix L. First, select the sequences at the same positions in the second and third dimensions of S. Its size is s Then, the index value with the smaller distance is selected; if the distances are the same, the index with the larger speed is selected, finally obtaining the point cloud image matrix L:

[0033] ;

[0034] Each element It is the feature index selected from the multidimensional feature matrix S after sorting by distance and velocity;

[0035] Based on the index position in the point cloud image matrix L, the corresponding distance, velocity, and energy values ​​are found in the multidimensional feature matrix S and filled into the point cloud image matrix L to construct a point cloud image containing distance, velocity, and energy.

[0036] Further, the point cloud image matrix is ​​input into a multi-dimensional feature fusion network for processing, including:

[0037] The distance matrix R, velocity matrix D, and energy matrix E are superimposed according to the channels:

[0038] ;

[0039] in, This is the multidimensional feature image matrix after channel overlay, and its size is... ; i For different channel indices of the multidimensional feature image matrix H;

[0040] Spatial features are extracted from the multidimensional feature image matrix H after channel stacking using convolutional blocks. The convolutional block consists of a convolutional neural network, batch normalization, non-linear activation layer, and pooling layer. For each channel stacked multidimensional feature image matrix H, the convolutional block convolves the three channels of the matrix together to capture the local spatial features of the image matrix H.

[0041] After extracting local features from the convolutional block, a multi-head attention mechanism is introduced to weight the global information;

[0042] After extracting the spatial features of the multidimensional point cloud image, a gated recurrent unit is introduced to extract the temporal features of the point cloud; the gated neural unit outputs 25 pose estimation coordinates, each of which contains two coordinate values, horizontal and vertical.

[0043] After obtaining the attitude estimation coordinates, the attitude estimation coordinates are mapped onto a multidimensional feature image matrix, and the three-dimensional spatial coordinates are obtained based on the distance information to form a three-dimensional attitude estimation result.

[0044] Furthermore, the multi-head attention mechanism includes an 8-head multi-head attention layer, which introduces a speed similarity constraint in the attention score calculation.

[0045] ;

[0046] in, This represents the score of the original attention layer. The adjusted attention score , The first in the space i , j The speed at each point For strong constraint coefficients, This is due to differences in speed tolerance.

[0047] The advantages and positive effects of this invention are as follows: By leveraging the relationship between radar signal energy and target reflective area, this invention proposes a target detection method based on spatial estimation. This method effectively overcomes the shortcomings of traditional millimeter-wave radar in resolving small-scale targets. Simultaneously, this invention utilizes the multidimensionality of radar signal features to construct a multidimensional target point cloud image matrix, increasing data information utilization and enhancing noise resistance. This effectively improves the universality and noise resistance of the attitude estimation model, thereby enhancing the system's robustness in dynamic environments. Furthermore, this invention employs a multidimensional feature fusion attitude estimation network to further improve information utilization efficiency, making the attitude estimation system more universal and capable of efficient estimation under various scenarios and conditions while reducing network training costs. Ultimately, this achieves the goals of improved resolution, enhanced universality, and increased information utilization in millimeter-wave radar attitude estimation algorithms. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a target attitude estimation method based on radar point cloud imaging and multi-dimensional feature fusion in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of spatial energy distribution and detection results in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the point cloud matrix construction process in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the multidimensional feature pose estimation network structure in an embodiment of the present invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0055] This invention proposes a target attitude estimation method based on radar point cloud imaging and multi-dimensional feature fusion. First, a spatial estimation method is used to measure the energy trend of target reflection at different distances, and the resolution of the radar point cloud is improved by filtering the target range. Second, a multi-dimensional target point cloud image matrix is ​​constructed by integrating multi-dimensional features at different distances, thereby increasing information density while reducing prediction difficulty. Finally, a multi-dimensional feature fusion attitude estimation network is designed. This network can fuse image matrices of different dimensions and combine temporal information to predict the human posture at the current moment, ensuring complete utilization of information and thus guaranteeing the efficiency and accuracy of attitude estimation results.

[0056] like Figure 1 As shown in the figure, this embodiment of the invention provides a target attitude estimation method based on radar point cloud imaging and multi-dimensional feature fusion. This embodiment performs continuous attitude estimation on human targets moving in a daily indoor environment and displays the real-time skeletal points of the target. The present invention places the transceiver device 1m above the ground.

[0057] The attitude estimation system is configured as follows: The system operates on a Texas Instruments IWR6843 FMCW radar transceiver; the transceiver operates at 60 GHz with a bandwidth of 3.49 GHz, providing a range resolution of 4.3 cm, a velocity resolution of 7 cm / s, and a frame rate of 10 fps; the transceiver uses a 3-transmit 4-receive antenna, which can form a 12-element virtual array with the help of TDM-MIMO mode, providing an angular resolution of 29°.

[0058] The target attitude estimation method specifically includes the following steps:

[0059] S1. Signal acquisition and processing;

[0060] The system transmits and receives frequency-modulated continuous wave signals via a MIMO radar array.

[0061] S2. Target detection and spatial reconstruction based on spatial estimation method;

[0062] In the target detection method based on spatial estimation, a two-dimensional energy matrix of range-velocity is constructed from the received intermediate frequency echo signal, and effective target points are screened. Utilizing the characteristics of a MIMO radar virtual array, direction-of-arrival (DOA) estimation is performed in both the horizontal and vertical antenna dimensions to obtain the angular spectrum distribution in two orthogonal dimensions: horizontal azimuth and elevation. Then, by jointly optimizing the joint distribution characteristics of the horizontal and vertical angular spectra, a mapping model between the two-dimensional angular spectrum and three-dimensional spatial coordinates is established using least-squares estimation. Combined with the constraint of target motion continuity, a refined reconstruction of the spatial distribution of human targets within the range-velocity unit is achieved. Finally, a target reflective area prediction model is constructed based on the signal energy peak value, and an exponential threshold function is used to dynamically screen effective target regions within different range-velocity units.

[0063] More specifically, after processing the received echo signal into an intermediate frequency signal, a Fast Fourier Transform (FFT) in the range dimension and a Fast Fourier Transform (FFT) in the velocity dimension are performed on the signal to obtain the range-velocity matrix of the target's range and velocity. By filtering target points and combining different virtual antennas at the target points along the horizontal and vertical directions, and further performing an angle-dimensional FFT, the coordinates of the target in the two vertical directions at the same range and velocity can be obtained. However, the coordinates in the two orthogonal directions alone cannot obtain the spatial distribution information of the target at that range and velocity. In order to obtain a more comprehensive estimation of the human body angle, this embodiment uses the distribution results in the horizontal and vertical directions to estimate the target distribution in space.

[0064] Specifically, for an antenna sequence at a given selection point The subsequences corresponding to the horizontal and vertical directions are: , Then, perform Fast Fourier Transform on these two subsequences respectively to obtain the frequency domain representations of the horizontal and vertical angles: ;

[0065] in, and These represent the frequency domain distributions of the horizontal and vertical angles, respectively. Subsequently, this invention converts the frequency domain results into amplitude:

[0066] ;

[0067] in , These are the frequency domain distribution moduli of the elevation angle. They represent the energy distribution of a target in the horizontal and vertical directions at a certain distance and velocity.

[0068] Although the energy distribution in the horizontal and vertical directions has been calculated separately, these results do not fully reflect the overall spatial distribution characteristics. To obtain more comprehensive spatial distribution information, it is necessary to further integrate the data in the horizontal and vertical directions and consider their interactions. Therefore, this invention introduces least squares estimation to calculate the overall spatial distribution characteristics of the target.

[0069] If the energy distribution of the target in the horizontal and vertical directions , The sizes are respectively n , m At this distance and speed, the overall energy distribution of the target in space is as follows: Then X and , The following relationship exists:

[0070] ;

[0071] Here, X is the spatial energy distribution matrix. The summation of the elements in the rows and columns of X yields a coefficient matrix. , Each row of this vector contains 1s and 0s. That is:

[0072] Among them, for Each of its actions includes m Row vectors of 1s ,for Each of its actions contains n Row vectors of 1s Matrix multiplication allows us to multiply the coefficient matrix A by X to obtain the sum of the rows and columns. ,

[0073] ;

[0074] From this underdetermined system of equations, the spatial energy distribution matrix X is estimated and solved using the least squares method:

[0075] ;

[0076] Since matrix A is invariant in multiple equations for different distances and velocities, the first three terms of different equations (i.e., The calculations are completely identical, eliminating the need for repeated calculations. This process significantly reduces the computational load, thereby ensuring the real-time performance of the entire system.

[0077] After estimating the target energy trend at different distances and speeds, the target needs to be detected. Since the target's reflective area is positively correlated with the energy level in the received signal, the target's reflective area can be detected based on the energy peak value of the received signal. Figure 2 As shown, for those with a peak value of g For the spatial energy distribution matrix X, this invention uses an exponential control function f to detect the target's range. G : Where P, Q, and C are the coefficients of the function. G This represents the percentage of the detected target area relative to the size of matrix X. During detection, targets are sorted downwards according to their peak energy, until the detected area reaches a certain percentage. G .

[0078] The specific implementation includes the following steps:

[0079] S21. Coarse estimation of point cloud spectrum: In this embodiment, data processing algorithms such as distance-dimensional fast Fourier transform, constant false alarm rate detection, and velocity-dimensional fast Fourier transform are used to obtain coarsely estimated point cloud distance and velocity spectrum, which are used as points to be screened.

[0080] S22. Coarse estimation of spatial energy spectrum: By selecting target points and combining different virtual antennas at the target points along the horizontal and vertical directions, the spatial spectrum distribution is obtained through FFT along the antenna dimension. For a given antenna sequence at a selected point... The antenna indices corresponding to the horizontal and vertical directions are respectively , Extract the subsequences corresponding to these indices. , Then, perform Fast Fourier Transform on these two subsequences respectively to obtain the frequency domain representations of the horizontal and vertical angles: ;in, and These represent the frequency domain distributions of the horizontal and vertical angles, respectively, with a magnitude of 64. The frequency domain results are then converted to amplitude:

[0081] ;

[0082] S23. High-Density Spectrum Estimation: Least squares estimation is introduced to calculate the overall spatial distribution characteristics of the target, yielding a more comprehensive spatial distribution profile. This includes obtaining the energy distribution in both horizontal and vertical directions. , The size of each target is 64. At this distance and speed, the overall spatial distribution of the target is as follows: .

[0083] During the summation of the elements in the rows and columns of X, there is a coefficient matrix. , Each row of the vector contains 1s and 0s.

[0084] Among them, for Each row contains a row vector of 64 ones. ,for Each row is a row vector containing 64 ones. Matrix multiplication allows us to multiply the coefficient matrix A by X to obtain the sum of the rows and columns. , : ;

[0085] From this underdetermined system of equations, the least squares method is used to estimate the solution matrix X: ;

[0086] Since matrix A is invariant in multiple equations for different distances and velocities, the first three terms of different equations (i.e., They are exactly the same, so there is no need to calculate them again.

[0087] S24. Multi-dimensional Target Point Cloud Image Screening: Targets are screened based on estimated energy trends at different distances and velocities. The target reflective area is predicted based on the received energy peak value. For targets with peak values... g The spatial energy distribution matrix X is used to filter the target range using an exponential control function f: ;

[0088] in, G This represents the percentage of the selected target area relative to the size of matrix A. During the selection process, targets are sorted downwards based on their peak energy, until the area of ​​the selected portion reaches the calculated percentage. G value.

[0089] S3. A multi-dimensional target point cloud image matrix construction method is used to sort the effective target area by distance-velocity hierarchy, and a three-channel point cloud image matrix containing distance, velocity and energy information is constructed.

[0090] In the multi-dimensional point cloud image generation method, a hierarchical sorting strategy is first adopted based on the detected target region: prioritizing ascending order by distance, and then organizing in descending order by velocity for the same distance, forming a structured three-dimensional feature matrix. Through a spatial index mapping mechanism, the distance, velocity, and energy information in the original feature matrix are precisely associated with the image pixel coordinates, achieving multi-parameter fusion-based three-dimensional point cloud visualization. This method effectively solves the information overload problem in dense target scenes through energy adaptive filtering and hierarchical sorting strategies, improving point cloud data quality and subsequent processing efficiency.

[0091] More specifically, after detecting the target range at different distances and speeds, G Sort by distance and speed, first sorting by distance in ascending order; if distances are the same, then sorting by speed in descending order. Finally, all target selection points will be arranged by distance from nearest to farthest, and by speed from largest to smallest for points at the same distance. The multidimensional feature matrix S, s This represents the total number of target points selected in a radar transmission frame.

[0092] Finally, the size is The multidimensional feature matrix S is merged into three The point cloud matrix L is first merged along the first dimension. For each pixel in the point cloud matrix L... First, select the sequences at the same positions in the second and third dimensions of S. Its size is s Then, the index value with the smaller distance is selected; if the distances are the same, the index with the larger speed is selected, finally obtaining the point cloud image matrix L:

[0093] ;

[0094] Each element It is the feature index selected from matrix S after sorting by distance and velocity.

[0095] After obtaining the point cloud matrix L, the corresponding distance, velocity, and energy values ​​can be found in S based on the index positions in L, and this information is then filled into matrix L. For example... Figure 3 As shown, a point cloud matrix containing distance, velocity, and energy can be constructed in this way.

[0096] In practice, the following specific steps are included:

[0097] S31. Multidimensional target point sorting: After filtering out the target ranges at different distances and speeds, sort them by distance and speed. First, sort by distance in ascending order. If the distances are the same, sort by speed in descending order.

[0098] The final target is that all selection points will be arranged in order of distance from nearest to farthest, and in order of speed from greatest to least at the same distance. The multidimensional feature matrix S, s This represents the total number of target points selected in a radar transmission frame.

[0099] S32. Generation of Multidimensional Target Point Cloud Image: Generate a multidimensional target point cloud image of size [size missing]. The multidimensional feature matrix S is merged into three The point cloud image matrix L.

[0100] First, the multidimensional feature matrices are merged along the first dimension. For each pixel in the point cloud image matrix L... First, select the sequences at the same positions in the second and third dimensions of S. Its size is s Then, the index value with the smaller distance is selected; if the distances are the same, the index with the larger speed is selected, finally obtaining the point cloud image matrix L:

[0101] ;

[0102] Based on the index position in L, the corresponding distance, velocity, and energy values ​​are found in the original matrix K, and this information is filled into matrix L to construct a point cloud image containing distance, velocity, and energy.

[0103] S4. Input the point cloud image matrix into a multi-dimensional feature fusion network for processing to obtain the coordinates of the skeleton points of the target pose;

[0104] In this embodiment, the designed multi-dimensional feature pose estimation network performs multi-granular feature extraction and fusion in both time and space from multi-dimensional point cloud images. For example... Figure 4 As shown, the distance matrix R, velocity matrix D, and energy matrix E are first superimposed according to the channels:

[0105] ;

[0106] in, This is the multidimensional feature image matrix after channel overlay, and its size is... . i For different channel indices of the multidimensional feature image matrix H.

[0107] Subsequently, spatial features are extracted from the point cloud matrix H using convolutional blocks, which consist of a convolutional neural network, batch normalization, nonlinear activation layers, and pooling layers. For each point cloud matrix H, the convolutional block convolves all three channels of the matrix together to capture the local spatial features of the image matrix H.

[0108] The output feature map from the convolutional block contains local spatial feature information, but relying solely on local features may ignore global relationships, affecting target recognition. Therefore, this invention introduces a multi-head attention mechanism after extracting local features from the convolutional block to weight global information. Multi-head attention can focus on multidimensional information in the feature map from different perspectives and adaptively adjust weights, enhancing the expression of global features and improving the model's ability to recognize targets.

[0109] Human motion is typically continuous and temporally correlated, meaning there is a regularity and correlation between the current point cloud image and the preceding and following frames. Therefore, relying solely on a single frame of point cloud image for feature extraction may lead to the loss of temporal information, affecting the model's ability to understand and predict motion trends. To address this, this invention, after extracting the spatial features of the multidimensional point cloud image, introduces a gated recurrent unit to extract the temporal features of the point cloud. Ultimately, the gated neural unit outputs 25 pose estimation coordinates, each containing both horizontal and vertical coordinate values.

[0110] In practice, firstly, the 3D point cloud feature spectral map is processed by a convolution kernel with a size of [missing information]. The convolutional layer extracts spatial features, resulting in 128-dimensional features, which are then input into a multi-head attention layer with 8 heads. A speed similarity constraint is introduced into the attention score calculation.

[0111] ;

[0112] in, This represents the score of the original attention layer. The adjusted attention score , The first in the space i , j The velocity of each point. This forces the model to focus on point clouds with similar velocities, suppressing erroneous associations between points with excessively different velocities. Among these... The strong constraint coefficient is set to 0.3. The speed tolerance is set at 0.07 m / s. Finally, the temporal relationship between multidimensional point cloud features is learned through gated neural units, and the 64-dimensional abstract features are mapped to... The pose matrix is ​​calculated. Here, 25 represents the predicted number of human skeleton points, and 2 represents the radial and horizontal coordinates relative to the radar. After obtaining the target pose estimation coordinates, this invention maps them onto a multi-dimensional feature image matrix and obtains the three-dimensional spatial coordinates based on its distance information to form a three-dimensional pose estimation result.

[0113] Regarding the training strategy, this invention proposes a joint training framework that integrates the physical characteristics of millimeter-wave radar with a deep learning model. A Doppler-kinematic coupling loss function is constructed: the network-predicted joint coordinate differential velocity is constrained by the radar's measured radial velocity, forcing the attitude estimation to conform to the electromagnetic wave scattering law. Simultaneously, a joint motion smoothness constraint is introduced, suppressing prediction jitter through the second-order difference between adjacent frame coordinates. During joint optimization, the physical constraint loss and attitude regression loss are weighted and fused, and the network parameters are updated synchronously during backpropagation. Specifically, based on the physical consistency constraint between joint motion velocity and radar Doppler measurements, a Doppler-kinematic coupling loss function is proposed:

[0114] ;

[0115] in The Doppler-kinematic coupling loss function is... T This represents the total time for sample collection. This represents the predicted total number of key points. It is predicted by the network. Time of the first Two-dimensional coordinates of each joint yes At that moment, in the first With each joint as the center and the radius as... The average velocity of the point cloud within the spatial circle.

[0116] After obtaining the target pose estimation coordinates, this invention maps them onto a multi-dimensional feature image matrix H, and obtains three-dimensional spatial coordinates based on its distance information to form a three-dimensional pose estimation result.

[0117] In the method of the above embodiments, the relationship between radar signal energy and target reflective area is used to improve the resolution of millimeter-wave radar when identifying small targets, and a new image structure is constructed to increase the information density of data and enhance the anti-noise capability. Then, the point cloud image is input into a multi-dimensional feature attitude estimation network for prediction, which improves the information utilization rate and solves the problems of poor universality and low robustness of existing attitude estimation methods.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human pose estimation method based on radar point cloud imaging and multi-dimensional feature fusion, characterized in that, Comprising the following steps: Transmit and receive a frequency-modulated continuous wave signal through a radar array; The received signal is subjected to a range dimension fast Fourier transform and a velocity dimension fast Fourier transform processing, and a distance-velocity two-dimensional energy matrix of the target is obtained; based on the distance-velocity two-dimensional energy matrix, a least square estimation algorithm is used to reconstruct a three-dimensional space energy distribution of the target, and an exponential threshold function is used to screen an effective target area; wherein the exponential threshold function screening the effective target area comprises: predicting the target reflection area size based on the accepted energy peak value, for a space energy distribution matrix X with a peak value of g , an exponential threshold function f is used to detect the range of the target G : ; wherein P, Q, C are function coefficients, G represents the percentage of the detected target range in the size of the matrix X, and during detection, the peak energy is sequentially sorted downward until the detected area percentage reaches G ; The effective target area is distance-speed layered and sequenced, and a three-channel point cloud image matrix containing distance, speed and energy information is constructed, including: after detecting the target range under different distances and speeds G sequencing by distance and speed, first by ascending order of distance, and if the distance is the same, by descending order of speed value; finally, all the screening points of the target are arranged in descending order of distance from near to far, and in descending order of speed from large to small under the same distance a multi-dimensional feature matrix S, s the total number of targets screened in a radar transmission frame; the multi-dimensional feature matrix S is merged along the first dimension, and for each pixel point in the point cloud matrix L , first select the sequence at the same position in the second and third dimensions in S , the size of which is s , then select the smaller index value in distance, and if the distance is the same, select the larger index in speed, and finally obtain the point cloud image matrix L: ； wherein each element is a feature index selected from the multi-dimensional feature matrix S after sorting by distance and speed; according to the index position in the point cloud image matrix L, the corresponding distance, speed and energy values are found in the multi-dimensional feature matrix S, and are filled into the point cloud image matrix L, to construct a point cloud image containing distance, speed and energy. Input the point cloud image matrix into a multi-dimensional feature fusion network for processing, and the multi-dimensional feature fusion network sequentially extracts local spatial features through a three-dimensional convolution module, captures global feature relationships through a multi-head attention mechanism module, and extracts time sequence motion features through a gate recurrent unit module, and finally outputs the skeletal point coordinates of the target pose.

2. The human pose estimation method based on radar point cloud imaging and multi-dimensional feature fusion according to claim 1, characterized in that, Performing range dimension fast Fourier transform and velocity dimension fast Fourier transform processing on the received signal to obtain a target range-velocity two-dimensional energy matrix, including: After processing the received echo signal into an intermediate frequency signal, performing range dimension fast Fourier transform and velocity dimension fast Fourier transform processing on the intermediate frequency signal to obtain a target range-velocity matrix of range and velocity; Screening target points and combining different virtual antennas at the target points along the horizontal and height directions, and further performing angle dimension fast Fourier transform to obtain the coordinates of the target in the same range and velocity in two vertical directions; Estimating the energy distribution of the target in space using the distribution results in the horizontal and height directions to obtain a target range-velocity two-dimensional energy matrix.

3. The human pose estimation method based on radar point cloud imaging and multi-dimensional feature fusion according to claim 2, characterized in that, Estimating the energy distribution of the target in space using the distribution results in the horizontal and height directions, including: Antenna sequence for a given screening point where the sub-sequences corresponding to the horizontal, height directions are , and the fast Fourier transform is applied to each of the two sub-sequences to obtain the frequency domain representation of the horizontal, height angles: ; where, and denote the frequency domain distribution of the horizontal, height angles, respectively. Transforming the frequency domain results into amplitudes: ; wherein , is the frequency domain distribution length of the elevation angle, representing the energy distribution of the target in the horizontal and elevation directions at a certain distance and velocity.

4. The human pose estimation method based on radar point cloud imaging and multi-dimensional feature fusion according to claim 3, characterized in that, Based on the range-velocity two-dimensional energy matrix, reconstructing the three-dimensional spatial energy distribution of the target using a least squares estimation algorithm, including: If the energy distribution of the target in the horizontal and height direction is , , the size of n , m , the overall energy distribution of the target in space at this distance and speed is , then X and , have the following relationship: ; where X is a spatial energy distribution matrix; and there is a coefficient matrix , whose each row is a vector containing 1, 0; that is: ； where each row of A contains ones and each row of X contains m ones , where each row of A contains n ones and each row of X contains ones , : ; From the underdetermined equation set, the spatial energy distribution matrix X is solved using least squares estimation: 。 5. The human pose estimation method based on radar point cloud imaging and multi-dimensional feature fusion according to claim 1, characterized in that, Inputting the point cloud image matrix into a multi-dimensional feature fusion network for processing, including: Stacking the range matrix R, the velocity matrix D, and the energy matrix E by channel: ； wherein, is the multi-dimensional feature image matrix after channel superposition, and the size of is ; i is the different channel index of the multi-dimensional feature image matrix H. Using a convolution block to extract spatial features from the multi-dimensional feature image matrix H after channel stacking, and the convolution block is composed of a convolution neural network, batch normalization, a nonlinear activation layer, and a pooling layer; for each multi-dimensional feature image matrix H after channel stacking, the convolution block convolves the 3 channels of the matrix together to capture the local spatial features of the image matrix H; After the convolution block extracts the local features, a multi-head attention mechanism is introduced to weight the global information; After extracting the spatial features of the multi-dimensional point cloud image, a gate recurrent unit is introduced to extract the time sequence features of the point cloud; the gate neural unit outputs 25 pose estimation coordinates, each coordinate containing two coordinate values of horizontal and vertical; After obtaining the pose estimation coordinates, the pose estimation coordinates are mapped to the multi-dimensional feature image matrix, and three-dimensional spatial coordinates are obtained according to the distance information to form a three-dimensional pose estimation result.

6. The human pose estimation method based on radar point cloud imaging and multi-dimensional feature fusion according to claim 1, characterized in that, The multi-head attention mechanism includes: a multi-head attention layer with 8 heads, and a velocity similarity constraint is introduced in the attention score calculation: ； wherein, denotes the score of the original attention layer, denotes the adjusted attention score, , denotes the velocity of the i , j point in space, is a strong constraint coefficient, is a velocity tolerance difference.

Images