A human posture reconstruction method based on radar 2D-DOA spectrum diagram

Abstract

The application discloses a human posture reconstruction method based on a radar 2D-DOA spectrum diagram, and is applied to the technical field of human posture reconstruction, and aims at the problem that the main implementation mode of the existing human posture reconstruction method based on a radar is to perform human posture representation through RA spectrum diagrams or point cloud data extracted by the radar, and echo information of a human target is lost; the application establishes a human target radar signal model by analyzing radar echo propagation characteristics of the human target in a close-range indoor home scene; radar echoes and posture image information of the human target are acquired by using a three-transmitting-four-receiving millimeter wave radar and an RGB camera, 2D-DOA spectrum diagram characteristics and posture labels of the human target are extracted in combination with a radar signal processing algorithm and a computer vision extraction method, and a human posture dataset is constructed; a multi-scale space-time feature extraction and human posture reconstruction model based on the radar 2D-DOA spectrum diagram is constructed in combination with time continuity characteristics of the human posture; the human posture reconstruction model is trained through the constructed human posture dataset, a human posture reconstruction model based on the radar is obtained, and human posture reconstruction in the close-range indoor home scene is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of human pose reconstruction, and particularly relates to a human pose reconstruction technology. BACKGROUND

[0002] In the field of human pose reconstruction, the existing mainstream technical approach is focused on realizing human pose reconstruction through computer vision and wearable devices. However, in the near-distance indoor home scene, the comfort and privacy security need to be considered for human pose reconstruction, and the existing method cannot meet the user demand. Since radar has the advantages of non-contact, privacy protection, all-weather and all-day work, how to realize human pose reconstruction based on radar has important research value.

[0003] In recent years, some studies have begun to explore radar-based methods for human posture reconstruction and have made some progress. In the research of Zheng et al., a new method, RadarFormer, was proposed, which for the first time introduces a self-attention mechanism to directly realize human perception from radar echoes, skipping traditional imaging algorithms and achieving end-to-end signal processing. (ZHENG Zhijie, ZHANG Diankun, LIANG Xiao, et al. RadarFormer: End-to-end human perception with through-wall radar and transformers[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 35(10): 4319-4332.). Sengupta et al. first proposed using millimeter-wave radar for human posture estimation. Human pose estimation is performed using target point cloud features acquired by millimeter-wave radar, accurately estimating the positions of more than 15 different human joints, achieving high-precision human pose estimation in dynamic and complex environments (SENGUPTA A, JIN Feng, ZHANG Renyuan, et al. mm-Pose: Real-time human skeletal posture estimation using mmWave radars and CNNs[J]. IEEE Sensors Journal, 2020, 20(17): 10032-10044.). Xie et al. proposed using two TI millimeter-wave radars MMWCAS-RF-EVM to acquire the horizontal and vertical RA spectra of the target for human pose representation, and using multiple cameras to simultaneously extract the 3D skeleton label of the target, and using the constructed RPM framework to realize the 3D pose reconstruction of the human body (XIE Chunyang, ZHANG Dongheng, WU Zhi, et al. RPM: RF-based pose machines[J]. IEEE Transactions on Multimedia, 2024, 26: 637-649.).Zhao et al. proposed generating a two-dimensional thermal image of the target's direction using a frequency-modulated continuous wave radar equipped with horizontal and vertical dual antenna arrays, acquiring human pose labels using vision, and fitting the constructed RF-Pose to achieve three-dimensional human pose prediction based on RF signals (ZHAO Mingmin, LI Tianhong, ABU ALSHEIKH M, et al. Through-wall human pose estimation using radio signals[C]. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018:7356-7365.).

[0004] However, existing radar-based human posture reconstruction methods primarily rely on radar-extracted RA spectra or point cloud data for human posture representation, which suffers from a lack of echo information about the human target. Furthermore, the radar systems used often involve numerous antennas and large array sizes, resulting in complex system structures and enormous size. Therefore, researching a human posture reconstruction system based on a small-array millimeter-wave radar is of significant practical importance. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a human posture reconstruction method based on radar 2D-DOA spectrograms. The method uses radar 2D-DOA spectrograms to characterize human posture and, through the construction of a multi-scale spatiotemporal feature extraction and human posture reconstruction model, achieves human posture reconstruction in close-range indoor home scenarios.

[0006] The technical solution adopted in this invention is: a human posture reconstruction method based on radar 2D-DOA spectral maps, comprising:

[0007] S1. To reconstruct human posture in indoor close-range home scenarios, a radar feature representation method for human targets is studied. Human posture features are represented by radar two-dimensional direction of arrival (2D-DOA) spectrum.

[0008] S2. Based on radar 2D-DOA spectrograms and combined with the temporal continuity characteristics of human target posture, construct a human posture reconstruction model based on multi-scale spatiotemporal features.

[0009] S3. Acquire the 2D-DOA spectrum and pose label of the human target by synchronously collecting data with radar and RGB camera. Construct a human pose dataset based on the 2D-DOA spectrum and pose label of the human target. Train the human pose reconstruction model constructed in step S2 based on the human pose dataset to obtain the radar-based human pose reconstruction model.

[0010] S4. Input the target 2D-DOA spectrum obtained by radar into the radar-based human posture reconstruction model obtained in step S3 to obtain the target posture reconstruction result, thus realizing radar-based human posture reconstruction.

[0011] The beneficial effects of this invention are as follows: This invention is based on a three-transmitter, four-receiver millimeter-wave radar for human posture perception, which has advantages such as small radar array size, compact system volume, simple structure, and insensitivity to illumination. In terms of posture reconstruction methods, 2D-DOA is used for human posture representation, which has the advantages of intuitive features and comprehensive human posture information representation, effectively improving posture reconstruction accuracy. Regarding the posture reconstruction model, considering the temporal continuity of human posture, human posture is represented through multi-frame 2D-DOA spectral features. A temporal feature deep extraction module is constructed to extract multi-frame spectral information, realizing temporal feature extraction of human posture. Simultaneously, considering the motion changes of human posture at different time scales, multiple observation windows are constructed through multi-scale spatiotemporal feature decomposition to achieve multi-scale spatiotemporal feature extraction. Finally, after fusing the multi-scale spatiotemporal features, the posture reconstruction module realizes the mapping from features to posture results, achieving high-precision human posture reconstruction. The method of this invention has the following advantages:

[0012] 1. The proposed human posture reconstruction method based on MIMO millimeter-wave radar realizes human perception through a three-transmitter, four-receiver small array radar, which effectively reduces the system size and gives it the advantages of small size, light weight and simple structure.

[0013] 2. The proposed human posture reconstruction method based on MIMO millimeter-wave radar cleverly utilizes the 2D-DOA spectrum generated by the radar to represent human posture, which can more effectively represent human posture information.

[0014] 3. The proposed human posture reconstruction model based on radar 2D-DOA spectrograms extracts radar spectrogram features at multiple spatiotemporal scales to reconstruct human postures, which can effectively improve the model's ability to extract features from radar spectrograms and extract spatiotemporal features of human postures, thereby further improving the model's human posture reconstruction effect.

[0015] 4. This invention can be used in indoor human-computer interaction, motion monitoring and medical rehabilitation assessment, providing effective support for contactless intelligent interaction, sports health and medical rehabilitation assessment. Attached Figure Description

[0016] Figure 1 The processing flow of the proposed method is described below.

[0017] Figure 2 A schematic diagram of an indoor human posture reconstruction scenario.

[0018] Figure 3 The real and virtual antenna arrays for a three-transmit, four-receive FMCW millimeter-wave radar;

[0019] (a) is the actual radar antenna array, and (b) is a schematic diagram of the virtual antenna array.

[0020] Figure 4 This describes the process for extracting human pose labels based on images.

[0021] Figure 5 The processing algorithm flow for radar 2D-DOA spectrum estimation.

[0022] Figure 6 A schematic diagram of a frame of raw radar data, TR map, RD map, and 2D-DOA spectrum;

[0023] In the figure, (a) is a schematic diagram of a frame of raw radar data, (b) is the TR map after processing the data, (c) is the RD map after processing the data, and (d) is the 2D-DOA spectrum after processing the data.

[0024] Figure 7 This is an overall block diagram of a human posture reconstruction model based on multi-scale spatiotemporal features.

[0025] Figure 8 Loss curve during model training

[0026] Figure 9 This is the perception front-end and software testing interface for a radar-based human posture reconstruction system.

[0027] Among them, (a) is the sensing front end of the radar-based human posture reconstruction system, and (b) is the software testing interface of the radar-based human posture reconstruction system. Detailed Implementation

[0028] To facilitate understanding of the technical content of this invention by those skilled in the art, the following description, in conjunction with the accompanying drawings, further illustrates the invention.

[0029] This invention establishes a radar signal model of human targets by analyzing the radar echo propagation characteristics of human targets in indoor scenes; it acquires radar echo and attitude image information of human targets using a three-transmitter, four-receiver millimeter-wave radar and an RGB camera, and extracts 2D-DOA spectral features and attitude labels of human targets by combining radar signal processing algorithms and computer vision extraction methods to construct a human attitude dataset; it constructs a multi-scale spatiotemporal feature extraction and human attitude reconstruction model based on radar 2D-DOA spectral features by combining the temporal continuity characteristics of human attitude; and it trains the proposed human attitude reconstruction model using the constructed human attitude dataset to obtain a radar-based human attitude reconstruction model, realizing human attitude reconstruction in close-range indoor home scenes.

[0030] The method flowchart of the present invention is as follows: Figure 1 As shown, it includes the following steps:

[0031] Step 1: Radar echo model analysis of human targets

[0032] Considering the target radar echo model in a near-field indoor home scenario, the scene diagram based on radar-reconstructed human posture is as follows: Figure 2 As shown, the radar and RGB camera are mounted on a tripod, 1.2m above the ground. Firstly, to achieve human target perception within the scene, the system employs a three-transmitter, four-receiver FMCW (Frequency-Modulated Continuous Wave) millimeter-wave radar with a frequency range of 60GHz-64GHz. The radar's real and virtual antenna arrays are shown below. Figure 3 As shown, the time-domain expression of the transmitted signal for this radar is:

[0033]

[0034] Where t represents time, A is the signal amplitude, f0 is the radar's starting frequency, B is the effective bandwidth of the radar's transmitted signal, and T c The period of the radar signal.

[0035] Then, for the human target, considering that it contains multiple scattering points as an extended target, a multi-scattering point model is established, assuming that it contains K scattering points and is located at a distance of d meters from the radar. At this time, the target echo received by the radar is as shown in the following formula.

[0036]

[0037] In the formula, r(t) represents the echo signal received by the radar receiving antenna, K represents the number of scattering points of the target, β represents the radar wave loss in free space, and τ k S represents the electromagnetic wave propagation delay from the transmitter to the k-th reflector and back to the radar receiver.noise (t) represents the electromagnetic background noise in the radar detection scenario.

[0038] After mixing and filtering the signal, the intermediate frequency signal of the human target posture echo is obtained:

[0039]

[0040] In the formula, The phase shift of the radar echo is represented by the above analysis, which yields a radar echo model of a human target in a single virtual channel.

[0041] Furthermore, for radar echoes from MIMO arrays, since the array spacing is half the wavelength, the time delay between different virtual antennas remains consistent; the difference mainly manifests in the phase shift of the echo. Therefore, for MIMO millimeter-wave radar, the intermediate frequency signal expression for the human target echo from different virtual antennas is:

[0042]

[0043] In the formula, yn(t) is the intermediate frequency signal obtained by analyzing the nth channel. Let be the phase shift of the nth channel. By analyzing the phase shifts between different channels, the DOA of the target can be estimated.

[0044] Step 2: Construction of Human Pose Dataset

[0045] In such Figure 2 In the indoor human pose reconstruction scenario shown, radar and RGB cameras are used to synchronously acquire radar echoes and image information of the human target, constructing a human pose dataset. For the synchronization of radar and RGB image data, a heterogeneous sensor data frame synchronization method is adopted, achieving data synchronization between heterogeneous sensors through the synchronization of radar data frames and image data frames. After acquiring radar and image data, to extract the human target's pose labels, YOLOv11-pose is used to extract pose joints from each frame of image data acquired from the RGB camera to obtain the human skeleton. The processing flow is as follows: Figure 4 As shown. To extract the radar 2D-DOA spectrum, a radar signal preprocessing algorithm is used to extract the 2D-DOA spectrum from the raw radar data.

[0046] Step 3: Radar signal preprocessing

[0047] The algorithm flow for radar 2D-DOA spectral estimation is as follows: Figure 5 As shown, the specific process is as follows:

[0048] First, the radar, through y n(t) Sampling is performed to obtain the original radar data matrix. To extract the human target range from the original data matrix, pulse compression is applied over a slow time interval to obtain the target time-range map (TR map), calculated as follows:

[0049]

[0050] In the formula, R(k,n) is the echo signal intensity at the k-th range cell at the n-th time, C(μ,n) represents the complex data in the μ-th row and n-th column of the original radar data matrix, and N is the number of pulse compression points. After calculation using the above formula, the TR map of the reflective object captured by the radar is obtained, which provides the distance between the radar and the human target, and the distance changes with time as the human target moves.

[0051] Next, in order to extract the target Doppler, Doppler spectrum estimation is performed on the TR map of the target in one frame to obtain the range-Doppler map (RD map) of the target in one frame. The calculation formula is shown below.

[0052]

[0053] In the formula, w(μ) represents the window function, R(k,m-μ) represents the value of the k-th range cell of the m-μ-th chirp signal, and N represents the number of pulse compression points. After processing all range cells of the current frame using the above method, the target's RD map is obtained.

[0054] Then, two-dimensional constant false alarm rate (2D-CFAR) detection is performed on the target RD map to obtain RD map cells containing human targets. For RD map cells containing human targets, 2D-DOA spectrum estimation is performed using the minimum mean square distortionless response (MVDR) algorithm. The calculation method of MVDR estimation of 2D-DOA is as follows.

[0055] calculate Figure 3 The steering vector of the MIMO array shown is:

[0056]

[0057] in, It is the horizontal guiding vector. The vertical guide vector is T, which represents the matrix transpose, and I and J correspond to the number of channels in the horizontal and vertical directions, respectively.

[0058] For the RD map cell containing the target, the RD map data of 12 virtual channels composed of three-transmitter, four-receiver radars are extracted, and the autocorrelation matrix is ​​calculated as follows:

[0059] R=E{d(K,M)d H (K,M)},

[0060] Where d(K,M)=[D1(K,M),D2(K,M),...,D 12 [K,M] represents the detected RD map cell data containing all 12 channels of the target, where D... i (K,M) represents the data in the Kth row and Mth column of the RD map of the i-th channel, where H represents the conjugate transpose. Combining the above calculations, a 2D-DOA spectrum estimation is performed, and its power spectrum is:

[0061]

[0062] Where, θ and These represent the horizontal and vertical angles, respectively. After the above processing, a 2D-DOA spectrum is obtained on this RD map unit. By performing the above estimation on all detected RD map units containing human targets, 2D-DOA spectra of all targets are obtained. After superimposing the results of all RD map units, the final 2D-DOA spectrum is obtained, which is used for human pose feature representation. A schematic diagram of one frame of raw radar data for human targets, including the TR map, RD map, and 2D-DOA spectrum, is shown below. Figure 6 As shown.

[0063] Step 4: Construction of Human Pose Reconstruction Model

[0064] Because human posture exhibits temporal continuity, a human posture reconstruction model based on multi-scale spatiotemporal features was constructed to effectively extract radar 2D-DOA spectral features and understand the temporal continuity of human posture. This model comprises a multi-scale spatiotemporal feature decomposition module, a temporal feature depth extraction module, a multi-scale temporal feature fusion module, and a posture reconstruction module. A schematic diagram of the overall model structure is shown below. Figure 7As shown; among them, the multi-scale spatiotemporal feature decomposition completes the decomposition of the input 2D-DOA at different time scales, obtaining 2D-DOA spectrogram sequences with different time steps; the temporal feature depth extraction module completes the temporal feature extraction of the current input 2D-DOA spectrogram sequence, obtaining the spatiotemporal depth information of the current 2D-DOA spectrogram sequence; the multi-scale temporal feature fusion is used to fuse the 2D-DOA spectrogram features extracted at different time scales, obtaining the multi-scale spatiotemporal features of human pose; the pose reconstruction module completes the mapping from the input multi-scale spatiotemporal features of human pose to the output human pose reconstruction result, obtaining the final output human pose.

[0065] Step 5: Model Training and Testing

[0066] To demonstrate the effectiveness of the proposed method, a human pose dataset containing 16,800 samples was constructed in step 2 to train and test the model. During training, the loss function was defined as follows:

[0067]

[0068] Where ||·|| represents the second norm, J represents the number of joints in the human body, and O i This represents the position of the i-th joint point output after fitting. This represents the geometric center of all joints after fitting. This indicates the position of the i-th joint in the label. This indicates the geometric center of all relevant nodes in the label.

[0069] After extracting the pose features of the human target using the data preprocessing method described in step S3, the system trains the human pose reconstruction model based on multi-scale spatiotemporal features constructed in step S4 using the loss function calculation method described above, thus obtaining a radar-based human pose reconstruction model. In model training, the final human pose reconstruction model is obtained through a fixed 1000 training rounds each time. The loss curve of the model training process is shown below. Figure 8 As shown in the figure. Finally, the model is deployed on the computational analysis and display platform to achieve real-time reconstruction of human posture. The computational analysis and display platform consists of a high-performance computer. This platform first acquires radar data of the human target in real time. After radar signal preprocessing as described in step 3, a 2D-DOA spectrum of the human target is obtained. Then, the real-time 2D-DOA spectrum is fed into the trained human posture reconstruction model to complete the real-time reconstruction from radar signal to human posture. The perception front-end and software testing interface of the radar-based human posture reconstruction system are shown in the figure. Figure 9 As shown.

[0070] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A method for reconstructing human posture based on radar 2D-DOA spectrograms, characterized in that, include: S1. For human posture reconstruction in indoor close-range home scenarios, a radar feature representation method for human targets is studied. Human posture features are represented by radar 2D-DOA spectra. The process of obtaining radar 2D-DOA spectra in step S1 is as follows: After mixing and filtering the target echo received by the radar, the intermediate frequency signal of the human target attitude echo is obtained; the target echo received by the radar is represented as: ； in, This indicates the target echo received by the radar; Indicates time; The signal amplitude; This is the radar's starting frequency; The effective bandwidth of the radar transmitted signal; The period of the radar signal; Indicates the number of scattering points of the target; This indicates the loss of radar waves in free space; This indicates that the electromagnetic wave arrives at the first... The electromagnetic wave reflected by a reflector experiences a transmission delay before reaching the radar receiver. This represents the electromagnetic background noise in a radar detection scenario. The intermediate frequency signal is sampled to obtain the original radar data matrix; Pulse compression is performed on the original radar data matrix over a slow time interval to obtain the target time-range spectrum. Doppler spectrum estimation is performed on the target time-range spectrum of a single frame to obtain the target range-Doppler spectrum of a single frame. Two-dimensional constant false alarm rate (CFAR) detection is performed on the target's range-Doppler spectrum to obtain range-Doppler spectrum cells containing human targets. For these cells, 2D-DOA spectral estimation is performed. Specifically, the least mean square distortion-free response algorithm is used for 2D-DOA spectral estimation. The calculation process is as follows: The steering vector of the radar array is calculated as follows: ； in, It is the horizontal guiding vector. It is the guide vector in the vertical direction. Indicates matrix transpose. These correspond to the number of channels in the horizontal and vertical directions, respectively. For the range-Doppler spectral cell containing the target, extract the range-Doppler spectral data and calculate the autocorrelation matrix as follows: ； in, To detect range-Doppler spectral data containing all channels of the target, ; Indicates the first The distance between channels - Doppler spectrum in the first... Okay, number Column data; Indicates conjugate transpose; 2D-DOA spectral estimation was performed, and its power spectrum is as follows: ； in, and These represent the horizontal angle and the vertical angle, respectively. S2. Based on radar 2D-DOA spectrograms and combined with the temporal continuity characteristics of human target posture, construct a human posture reconstruction model based on multi-scale spatiotemporal features. S3. Acquire the 2D-DOA spectrum and pose label of the human target by synchronously collecting data with radar and RGB camera. Construct a human pose dataset based on the 2D-DOA spectrum and pose label of the human target. Train the human pose reconstruction model constructed in step S2 based on the human pose dataset to obtain the radar-based human pose reconstruction model. S4. Input the target 2D-DOA spectrum obtained by radar into the radar-based human posture reconstruction model obtained in step S3 to obtain the target posture reconstruction result, thus realizing radar-based human posture reconstruction.

2. The human posture reconstruction method based on radar 2D-DOA spectrum according to claim 1, characterized in that, The human pose reconstruction model based on multi-scale spatiotemporal features in step S2 includes: a multi-scale spatiotemporal feature decomposition module, a temporal feature depth extraction module, a multi-scale temporal feature fusion module, and a pose reconstruction module. Multi-scale spatiotemporal feature decomposition completes the decomposition of the input 2D-DOA spectrogram at different time scales, obtaining 2D-DOA spectrogram sequences with different time steps. The temporal feature depth extraction module extracts the temporal features of the current input 2D-DOA spectrogram sequence, obtaining the spatiotemporal depth information of the current 2D-DOA spectrogram sequence. Multi-scale temporal feature fusion is used to fuse the 2D-DOA spectrogram features extracted at different time scales, obtaining the multi-scale spatiotemporal features of the human pose. The pose reconstruction module maps the input multi-scale spatiotemporal features of the human pose to the output human pose reconstruction result, obtaining the final output human pose.

3. The human posture reconstruction method based on radar 2D-DOA spectrum according to claim 2, characterized in that, In step S3, the pose labels are extracted from each frame of image data collected by the RGB camera using a computer vision extraction algorithm to obtain the human skeleton.

4. The human posture reconstruction method based on radar 2D-DOA spectrum according to claim 3, characterized in that, The computer vision extraction algorithm is YOLOv11-pose.

5. The human posture reconstruction method based on radar 2D-DOA spectrum according to claim 4, characterized in that, In step S3, during training, the loss function is defined as follows: ； in, This indicates the number of joints in the human body. This represents the first output after fitting. The location of each joint. This represents the geometric center of all joints after fitting. Indicates the first in the label The location of each joint. This indicates the geometric center of all relevant nodes in the label.

Images