Human body posture reconstruction method based on radar 2D-DOA spectrogram

Abstract

The invention discloses a human body posture reconstruction method based on a radar 2D-DOA spectrogram, is applied to the technical field of human body posture reconstruction, and aims to solve the problem that echo information of a human body target is lost due to the fact that a main implementation mode of an existing radar-based human body posture reconstruction method is to carry out human body posture representation through an RA spectrogram or point cloud data extracted by a radar. The method comprises the following steps: establishing a human body target radar signal model by analyzing radar echo propagation characteristics of a human body target in a close-range indoor home scene; the method comprises the following steps: acquiring radar echo and attitude image information of a human body target by using a three-transmitting four-receiving millimeter wave radar and an RGB camera, extracting 2D-DOA spectrogram features and attitude tags of the human body target in combination with a radar signal processing algorithm and a computer vision extraction method, and constructing a human body attitude data set; constructing a multi-scale spatiotemporal feature extraction and human body posture reconstruction model based on a radar 2D-DOA spectrogram in combination with the time continuity feature of the human body posture; and training the extracted human body posture reconstruction model through the constructed human body posture data set to obtain a radar-based human body posture reconstruction model, thereby realizing human body posture reconstruction in a close-range indoor home scene.

Description

Technical Field

[0001] The present invention belongs to the field of human body posture reconstruction, and in particular relates to a human body posture reconstruction technology. Background Art

[0002] In the field of human posture reconstruction, the existing mainstream technical approaches focus on achieving human posture reconstruction through computer vision and wearable devices. However, human posture reconstruction in close-range indoor home scenarios requires consideration of comfort and privacy security, and existing methods cannot meet user needs. Radar has the advantages of being non-contact, privacy-protecting, and working all day and all night. Therefore, how to achieve radar-based human posture reconstruction has important research value.

[0003] In recent years, some studies have begun to explore radar-based human pose reconstruction methods and have made some progress. In their research, Zheng et al. proposed a new method, RadarFormer, which introduced a self-attention mechanism for the first time, directly realizing human perception from radar echoes, skipping traditional imaging algorithms and achieving end-to-end signal processing. (ZHENG Zhijie, ZHANG Diankun, LIANG Xiao, et al. RadarFormer: End-to-end human perception withthrough wall radar and transformers[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 35(10): 4319-4332.). Sengupta et al. first proposed using millimeter-wave radar for human pose estimation. Human pose estimation is performed by using the target point cloud features obtained by millimeter-wave radar, accurately estimating the positions of more than 15 different human joints, and achieving high-precision human pose estimation in dynamic and complex environments (SENGUPTA A, JIN Feng, ZHANG Renyuan, et al. mm-Pose: Real-time human skeletal posture estimation using mmWave radars and CNNs [J]. IEEE Sensors Journal, 2020, 20 (17): 10032-10044.). Xie et al. proposed using two TI millimeter-wave radars MMWCAS-RF-EVM to obtain the horizontal RA and vertical RA spectra of the target for human pose representation, and using multiple cameras to simultaneously extract the 3D skeleton label of the target. The 3D pose reconstruction of the human body is achieved through the constructed RPM framework (XIE Chunyang, ZHANG Dongheng, WU Zhi, et al. RPM: RF-based pose machines [J]. IEEE Transactions on Multimedia, 2024, 26: 637-649.).Zhao et al. proposed using a frequency-modulated continuous wave radar equipped with horizontal and vertical dual antenna arrays to generate a two-dimensional heat map of the target's corresponding direction, using vision to obtain human pose labels, and fitting them through the constructed RF-Pose to achieve three-dimensional human pose prediction based on RF signals (ZHAO Mingmin, LI Tianhong, ABU ALSHEIKH M, et al. Through-wall human pose estimation using radio signals[C]. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 7356-7365.).

[0004] However, existing radar-based human pose reconstruction methods primarily rely on radar-generated spectra or point cloud data to represent human poses, which results in missing echo information about human targets. Furthermore, the radar systems employed often have numerous antennas and large arrays, making the systems complex and bulky. Therefore, research on a human pose reconstruction system based on a small-array millimeter-wave radar is of great practical significance. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a human posture reconstruction method based on radar 2D-DOA spectrum. Human posture is represented by the radar 2D-DOA spectrum, and human posture reconstruction in close-range indoor home scenes is achieved by constructing a multi-scale spatiotemporal feature extraction and human posture reconstruction model.

[0006] The technical solution adopted by the present invention is: a human body posture reconstruction method based on radar 2D-DOA spectrum, comprising:

[0007] S1. Focusing on human posture reconstruction in indoor close-range home scenes, we study the radar feature characterization method of human targets and characterize human posture features through radar 2D Direction of Arrival (2D-DOA) spectrogram.

[0008] S2. Based on the radar 2D-DOA spectrum and the temporal continuity characteristics of the human target posture, a human posture reconstruction model based on multi-scale spatiotemporal features is constructed;

[0009] S3. Acquire the 2D-DOA spectrogram and posture label of the human target through synchronous acquisition by radar and RGB camera, construct a human posture dataset based on the 2D-DOA spectrogram and posture label of the human target, and train the human posture reconstruction model constructed in step S2 based on the human posture dataset to obtain a radar-based human posture reconstruction model;

[0010] S4. Input the target 2D-DOA spectrum acquired by the radar into the radar-based human posture reconstruction model obtained in step S3 to obtain the target posture reconstruction result, thereby realizing radar-based human posture reconstruction.

[0011] Beneficial effects of the present invention: The present invention is based on a three-transmitter, four-receiver millimeter-wave radar for human posture perception, which has the advantages of small radar array size, compact system size, simple structure and insensitivity to light. In the posture reconstruction method, 2D-DOA is used to characterize the human posture, which has the advantages of intuitive features and comprehensive characterization of human posture information, and can effectively improve the accuracy of posture reconstruction. In the posture reconstruction model, considering the temporal continuity of human posture, human posture is characterized by multi-frame 2D-DOA spectrum features, and a temporal feature deep extraction module is constructed to extract multi-frame spectrum information to realize temporal feature extraction of human posture. At the same time, considering the movement changes of human posture at different time scales, multiple different time observation windows are constructed through multi-scale spatiotemporal feature splitting to realize multi-scale spatiotemporal feature extraction. Finally, after the multi-scale spatiotemporal features are fused, the mapping from features to posture results is realized through the posture reconstruction module to realize high-precision human posture reconstruction. The method of the present invention has the following advantages:

[0012] 1. The proposed human posture reconstruction method based on MIMO millimeter-wave radar realizes human perception through a three-transmitter and four-receiver small array radar, effectively reducing the system size and making it have the advantages of small size, light weight and simple structure.

[0013] 2. The proposed human posture reconstruction method based on MIMO millimeter-wave radar cleverly uses the 2D-DOA spectrum generated by radar to represent human posture, which can more effectively represent human posture information;

[0014] 3. The proposed human posture reconstruction model based on radar 2D-DOA spectrograms reconstructs human posture by extracting radar spectrum features at multiple spatiotemporal scales. This can effectively improve the model's ability to extract features from radar spectra and the spatiotemporal features of human posture, further improving the model's human posture reconstruction effect.

[0015] 4. The present invention can be used in the fields of human-computer interaction, motion monitoring and medical rehabilitation assessment in indoor scenarios, providing effective support for contactless intelligent interaction, sports health and medical rehabilitation assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the processing flow of the proposed method.

[0017] Figure 2 Schematic diagram of the scene for indoor human posture reconstruction.

[0018] Figure 3 The real and virtual antenna arrays for the three-transmit, four-receive FMCW millimeter-wave radar;

[0019] Among them, (a) is the real antenna array of the radar, and (b) is the schematic diagram of the virtual antenna array.

[0020] Figure 4 This is the process of extracting human posture labels based on images.

[0021] Figure 5 This is the processing algorithm flow for radar 2D-DOA spectrum estimation.

[0022] Figure 6 A schematic diagram of a frame of radar raw data, TR map, RD map and 2D-DOA spectrum;

[0023] Among them, (a) is a schematic diagram of a frame of radar raw data, (b) is the TR map after processing the frame data, (c) is the RD map after processing the frame data, and (d) is the 2D-DOA spectrum after processing the frame data.

[0024] Figure 7 The overall framework diagram of the human posture reconstruction model based on multi-scale spatiotemporal features.

[0025] Figure 8 is the loss curve of the model training process

[0026] Figure 9 Perception front-end and software testing interface for a radar-based human pose reconstruction system.

[0027] Among them, (a) is the perception front end of the radar-based human posture reconstruction system, and (b) is the software test interface of the radar-based human posture reconstruction system. DETAILED DESCRIPTION

[0028] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0029] The present invention establishes a human target radar signal model by analyzing the radar echo propagation characteristics of human targets in indoor scenes; uses a three-transmitter, four-receiver millimeter-wave radar and an RGB camera to obtain the radar echo and posture image information of the human target, combines the radar signal processing algorithm with the computer vision extraction method to extract the 2D-DOA spectrum features and posture labels of the human target, and constructs a human posture dataset; combines the time continuity characteristics of human posture to construct a multi-scale spatiotemporal feature extraction and human posture reconstruction model based on the radar 2D-DOA spectrum; trains the proposed human posture reconstruction model with the constructed human posture dataset to obtain a radar-based human posture reconstruction model, which realizes human posture reconstruction in close-range indoor home scenes.

[0030] The method flow chart of the present invention is as follows Figure 1 As shown, the following steps are included:

[0031] Step 1: Radar echo model analysis of human targets

[0032] Considering the target radar echo model in a close-range indoor home scene, the scene diagram of human posture reconstruction based on radar is as follows: Figure 2 As shown in the figure, the radar and RGB camera are mounted on a tripod at a height of 1.2m from the ground. First, in order to realize the perception of human targets in the scene, the system uses a three-transmitter and four-receiver FMCW (Frequency-Modulated Continuous Wave) millimeter wave radar with a frequency range of 60GHz-64GHz. The real and virtual antenna arrays of the radar are shown in the figure. Figure 3 As shown in Figure 2, for this radar, the time domain expression of its transmitted signal is:

[0033]

[0034] Where t represents time, A is the signal amplitude, f0 is the starting frequency of the radar, B is the effective bandwidth of the radar transmission signal, T c is the period of the radar signal.

[0035] Then, for the human target, considering that it contains multiple scattering points as an extended target, a multi-scattering point model is established. It is assumed that it contains K scattering points and is located d meters away from the radar. At this time, the target echo received by the radar is shown in the following formula.

[0036]

[0037] Where r(t) represents the echo signal received by the radar receiving antenna, K represents the number of scattering points of the target, β represents the loss of radar wave in free space, and τ k S represents the transmission delay of the electromagnetic wave from the transmitter to the kth reflector and then reflected to the radar receiver.noise (t) represents the electromagnetic background noise in the radar detection scenario.

[0038] After mixing and filtering the signal, the intermediate frequency signal of the human target posture echo is obtained:

[0039]

[0040] Where, Represents the phase offset of the radar echo. After the above analysis, the human target radar echo model of a single virtual channel is obtained.

[0041] Furthermore, for radar echoes from a MIMO array, since the array spacing is half the wavelength, the time delays between different virtual antennas remain consistent, and the differences are mainly reflected in the phase offset of the echoes. Therefore, for MIMO millimeter-wave radar, the intermediate frequency signal expression for the human target echo from different virtual antennas is:

[0042]

[0043] Where yn(t) is the intermediate frequency signal obtained by analyzing the nth channel, is the phase offset of the nth channel. By analyzing the phase offsets between different channels, the DOA of the target is estimated.

[0044] Step 2: Human pose dataset construction

[0045] In such Figure 2 In the indoor human posture reconstruction scenario shown in the figure, radar and RGB camera are used to synchronously collect radar echoes and image information of human targets, and a human posture dataset is constructed. In terms of synchronization between radar and RGB image data, heterogeneous sensor data frame synchronization is adopted. By synchronizing radar data frames with image data frames, heterogeneous sensor data synchronization is achieved. After obtaining radar data and image data, in order to extract the posture label of the human target, YOLOv11-pose is used to extract posture joints from each frame of image data collected by the RGB camera to obtain the human skeleton. The processing flow is as follows: Figure 4 In order to extract the radar 2D-DOA spectrum, the radar signal preprocessing algorithm is used to complete the extraction from the radar raw data to the 2D-DOA spectrum.

[0046] Step 3: Radar signal preprocessing

[0047] The algorithm flow of radar 2D-DOA spectrum estimation is as follows Figure 5 As shown, the specific process is as follows:

[0048] First, the radar passes through y n(t) is sampled to obtain the radar raw data matrix. In order to extract the human target distance on the raw data matrix, pulse compression is performed within a slow time of the raw data matrix to obtain the target time-range map (TR map). Its calculation formula is as follows:

[0049]

[0050] Where R(k,n) is the echo signal strength at the kth range cell at the nth time, C(μ,n) represents the complex data in the μth row and nth column of the radar's raw data matrix, and N is the number of pulse compression points. The above calculation yields the TR map of the reflector captured by the radar, which provides the distance between the radar and the human target. This distance also changes over time as the human target moves.

[0051] Next, to extract the target Doppler, the Doppler spectrum is estimated on the TR map of the target frame to obtain the range-Doppler map (RD map) of the target frame. The calculation formula is as follows.

[0052]

[0053] Where w(μ) represents the window function, R(k,m-μ) represents the value of the m-μth chirp signal at the kth range bin, and N represents the number of pulse compression points. The RD map of the target is obtained by processing all range bins in the current frame using the above method.

[0054] Then, two-dimensional constant false alarm rate (2D-CFAR) detection is performed on the target RD map to obtain the RD map unit containing the human target. The 2D-DOA spectrum of the RD map unit containing the human target is estimated using the minimum mean square distortionless response (MVDR) algorithm. The calculation method of MVDR estimation of 2D-DOA is as follows.

[0055] calculate Figure 3 The steering vectors for the MIMO array shown are:

[0056]

[0057] in, is the horizontal steering vector, is the steering vector in the vertical direction, T represents the matrix transpose, and I and J correspond to the number of channels in the horizontal and vertical directions, respectively.

[0058] For the RD map unit containing the target, extract the RD map data of 12 virtual channels composed of three-transmit and four-receive radars, and calculate the autocorrelation matrix as follows:

[0059] R=E{d(K,M)d H (K,M)},

[0060] Where, d(K,M)=[D1(K,M),D2(K,M),...,D 12 (K, M)] is the RD map unit data of all 12 channels containing the target, D i (K, M) represents the data in the Kth row and Mth column of the RD map of the i-th channel, and H represents the conjugate transpose. Combined with the above calculations, the 2D-DOA spectrum is estimated, and its power spectrum is:

[0061]

[0062] Among them, θ and Represents the horizontal angle and vertical angle respectively. After the above processing, the 2D-DOA spectrum on the RD map unit is obtained. After the above estimation is performed on all the RD map units containing human targets, the 2D-DOA spectrum of all targets is obtained. After superimposing the results of all RD map units, the final 2D-DOA spectrum is obtained for human posture feature characterization. A schematic diagram of a frame of radar raw data of a human target, TR map, RD map and 2D-DOA spectrum is shown in the figure. Figure 6 shown.

[0063] Step 4: Human body posture reconstruction model construction

[0064] Since human posture has temporal continuity, in order to effectively extract the radar 2D-DOA spectrum features and the temporal continuity of human posture, a human posture reconstruction model based on multi-scale spatiotemporal features is constructed. The model consists of multi-scale spatiotemporal feature splitting, temporal feature depth extraction module, multi-scale temporal feature fusion and posture reconstruction module. The overall structure diagram of the model is shown in the figure. Figure 7As shown; among them, the multi-scale spatiotemporal feature splitting completes the splitting of the input 2D-DOA at different time scales, and obtains 2D-DOA spectrogram sequences with different time steps; the temporal feature depth extraction module completes the temporal feature extraction of the current input 2D-DOA spectrogram sequence, and obtains the spatiotemporal depth information of the current 2D-DOA spectrogram sequence; the multi-scale temporal feature fusion is used to fuse the 2D-DOA spectrogram features extracted at different time scales, and obtain the multi-scale spatiotemporal features of the human body posture; the posture reconstruction module completes the mapping of the input multi-scale spatiotemporal features of the human body posture to the output human body posture reconstruction result, and obtains the final output human body posture.

[0065] Step 5: Model training and testing

[0066] In order to demonstrate the effectiveness of the proposed method, a human posture dataset containing 16,800 samples was constructed in step 2 to train and test the model. During the training process, the loss function is defined as:

[0067]

[0068] Among them, ||·|| represents the second-order norm, J represents the number of human joints, O i Represents the position of the i-th joint point output after fitting, Represents the geometric center of all joint points after fitting, Indicates the position of the i-th joint point in the label, Indicates the geometric center of all nodes in the label.

[0069] After the system extracts the posture features of the human target using the data preprocessing method described in step S3, it trains the human posture reconstruction model based on multi-scale spatiotemporal features constructed in step S4 in combination with the calculation method of the loss function to obtain a radar-based human posture reconstruction model. In model training, the final human posture reconstruction model is obtained through a fixed 1000 rounds of training each time. The loss curve of the model training process is shown in the figure below. Figure 8 As shown. Finally, the model is deployed on the computing analysis and display platform to realize the real-time reconstruction of human posture. The computing analysis and display platform is composed of a high-performance computer. The platform first obtains the radar data of the human target in real time, and obtains the 2D-DOA spectrum of the human target after the radar signal preprocessing described in step 3. Then, the real-time 2D-DOA spectrum is sent to the human posture reconstruction model obtained after training to complete the real-time reconstruction from radar signal to human posture. The perception front end and software test interface of the radar-based human posture reconstruction system are shown in the figure. Figure 9 shown.

[0070] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A human body posture reconstruction method based on radar 2D-DOA spectrogram, characterized in that: include: S1. Focusing on human posture reconstruction in indoor close-range home scenes, we study the radar feature characterization method of human targets and characterize human posture features through radar 2D-DOA spectrogram; S2. Based on the radar 2D-DOA spectrum and the temporal continuity characteristics of the human target posture, a human posture reconstruction model based on multi-scale spatiotemporal features is constructed; S3. Acquire the 2D-DOA spectrogram and posture label of the human target through synchronous acquisition by radar and RGB camera, construct a human posture dataset based on the 2D-DOA spectrogram and posture label of the human target, and train the human posture reconstruction model constructed in step S2 based on the human posture dataset to obtain a radar-based human posture reconstruction model; S4. Input the target 2D-DOA spectrum acquired by the radar into the radar-based human posture reconstruction model obtained in step S3 to obtain the target posture reconstruction result, thereby realizing radar-based human posture reconstruction.

2. The human body posture reconstruction method based on radar 2D-DOA spectrum according to claim 1, characterized in that: The acquisition process of the radar 2D-DOA spectrum in step S1 is: After mixing and filtering the target echo received by the radar, the intermediate frequency signal of the human target posture echo is obtained; Sampling the intermediate frequency signal to obtain the radar raw data matrix; Perform pulse compression on a slow time of the original radar data matrix to obtain the target time-range spectrum; Doppler spectrum estimation is performed on the target time-range spectrum of one frame of the target to obtain the range-Doppler spectrum of one frame of the target; A two-dimensional constant false alarm detection is performed on the range-Doppler spectrum of the target to obtain a range-Doppler spectrum unit containing the human target, and a 2D-DOA spectrum estimation is performed on the range-Doppler spectrum unit containing the human target.

3. The method for human body posture reconstruction based on radar 2D-DOA spectrum according to claim 2, characterized in that: The target echo received by the radar is expressed as: Where r(t) represents the target echo received by the radar; t represents time; A is the signal amplitude; f0 is the starting frequency of the radar; B is the effective bandwidth of the radar transmission signal; T c is the period of the radar signal; K is the number of scattering points of the target; β is the loss of radar wave in free space; τ k S represents the transmission delay of the electromagnetic wave from the transmitter to the kth reflector and then reflected to the radar receiver; noise (t) represents the electromagnetic background noise in the radar detection scenario.

4. The method for human body posture reconstruction based on radar 2D-DOA spectrum according to claim 3, characterized in that: Specifically, the 2D-DOA spectrum is estimated using the least mean square undistorted response algorithm. The calculation process is as follows: Calculate the steering vector of the radar array as: Among them, v(ψ y ) is the horizontal steering vector, v(ψ z ) is the vertical steering vector, T represents the matrix transpose, and I, J correspond to the number of channels in the horizontal and vertical directions, respectively; For the range-Doppler spectrum unit containing the target, extract the range-Doppler spectrum data and calculate the autocorrelation matrix as follows: R=E{d(K,M)d H (K,M)}, Where d(K,M) is the range-Doppler spectrum data of all channels containing the target, d(K,M)=[D1(K,M),D2(K,M),...,D 12 (K,M)];D i (K, M) represents the data in the Kth row and Mth column of the range-Doppler spectrum of the i-th channel; H represents the conjugate transpose; Perform 2D-DOA spectrum estimation, and its power spectrum is: Among them, θ and Represents the horizontal angle and vertical angle respectively.

5. The method for human body posture reconstruction based on radar 2D-DOA spectrum according to claim 4, characterized in that: The human posture reconstruction model based on multi-scale spatiotemporal features in step S2 includes: a multi-scale spatiotemporal feature splitting module, a temporal feature depth extraction module, a multi-scale temporal feature fusion module and a posture reconstruction module; the multi-scale spatiotemporal feature splitting completes the splitting of the input 2D-DOA spectrogram at different time scales to obtain a 2D-DOA spectrogram sequence with different time steps; the temporal feature depth extraction module completes the temporal feature extraction of the current input 2D-DOA spectrogram sequence to obtain the spatiotemporal depth information of the current 2D-DOA spectrogram sequence; the multi-scale temporal feature fusion is used to fuse the 2D-DOA spectrogram features extracted at different time scales to obtain the multi-scale spatiotemporal features of the human posture; the posture reconstruction module completes the mapping of the input multi-scale spatiotemporal features of the human posture to the output human posture reconstruction result to obtain the final output human posture.

6. The method for human body posture reconstruction based on radar 2D-DOA spectrum according to claim 5, characterized in that: In step S3, the posture label is specifically extracted from each frame of image data collected by the RGB camera through a computer vision extraction algorithm to obtain the human skeleton.

7. The method for reconstructing human posture based on radar 2D-DOA spectrum according to claim 6, characterized in that: The computer vision extraction algorithm is YOLOv11-pose.

8. The method for human body posture reconstruction based on radar 2D-DOA spectrum according to claim 7, characterized in that: During the training process in step S3, the loss function is defined as: Among them, J represents the number of human joints, O i Represents the position of the i-th joint point output after fitting, O c Represents the geometric center of all joint points after fitting, Indicates the position of the i-th joint point in the label, Indicates the geometric center of all nodes in the label.

Images