A human posture estimation method based on radio frequency heat map data enhancement

Abstract

The present invention discloses a method for human posture estimation based on radio frequency heat map data enhancement. The method first obtains grid data of human posture, simulates radar using physical optics, and obtains human body grid features including radar cross-sectional area by irradiating a human body grid model. Secondly, the human body grid features including radar cross-sectional area are processed to obtain a preliminary simulated radar heat map. Then, the preliminary simulated radar heat map is output through a heat map conversion network based on U‑net, and training is performed. Finally, the synthetic radar heat map is combined with the real radar heat map to construct a hybrid data set, and a human body posture skeleton is obtained through a human body posture estimation network based on radar heat map, and training is performed to complete human body posture estimation. The present invention can accurately generate a simulated radar heat map of a human target, make up for the insufficiency of the current radar-based human body posture estimation data set, and achieve a more accurate estimation of human body posture.

Description

Technical Field

[0001] The present invention relates to the field of wireless intelligent sensing technology, and in particular to a method for estimating human posture by enhancing radio frequency heat map data based on a physical optics method. Background Art

[0002] Human posture estimation has important application value in medical rehabilitation, intelligent monitoring, virtual reality, augmented reality, human-computer interaction and other fields. Traditional visual sensors (such as optical cameras and depth cameras) perform well in posture estimation, but they are easily limited by lighting conditions, occlusions and privacy issues. Radio Frequency (RF) sensing technology, as an emerging non-visual sensing method, can work normally in low-light and complex environments and has the ability to penetrate objects. Therefore, it has great potential in human posture estimation.

[0003] At present, human posture estimation based on radio frequency technologies such as millimeter wave radar has gradually attracted attention. However, the existing radar-based human posture data collection faces the problems of insufficient quantity and difficulty in collection. Due to the complexity of the radar system and the expensive equipment price, it is very difficult to build a large-scale, accurately labeled radar posture dataset, which directly affects the training and performance optimization of the posture estimation model. At the same time, the data generated by the radar is relatively sparse, reflecting limited posture information, and it is difficult to accurately capture the detailed features of the human body, which brings challenges to high-precision posture estimation.

[0004] As a classic method based on electromagnetic wave propagation theory, the Physical Optics (PO) method can accurately simulate the behavior of an object under RF signal irradiation by simulating the interaction between electromagnetic waves and objects. When RF waves come into contact with objects, electromagnetic waves will undergo reflection, refraction, diffraction, and scattering. By calculating these phenomena, the physical optics method can effectively restore the interaction process between RF waves and complex targets such as the human body, thereby generating a more accurate RF heat map of the human body. The RF heat map is input into the neural network for fine-tuning to obtain a heat map that is very similar to a real radar. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a method for human posture estimation based on radio frequency heat map data enhancement. This radio frequency heat map data enhancement method based on physical optics can effectively alleviate the problems of data sparsity and insufficient quantity on the basis of existing radio frequency sensing technology, and improve the accuracy and robustness of human posture estimation.

[0006] The specific technical solutions are as follows:

[0007] A method for estimating human posture based on radio frequency heat map data enhancement comprises the following steps:

[0008] Step S1: using sensors to obtain grid data of human body posture;

[0009] Step S2: using the physical optics method to simulate the radar, constructing a human body mesh model based on the mesh data of S1, and obtaining the human body mesh features including the radar cross-sectional area by illuminating the human body mesh model;

[0010] Step S3: Processing the human body grid features including the radar cross-sectional area obtained in S2 to obtain a preliminary simulated radar heat map, including: a distance-horizontal angle heat map, a distance-elevation angle heat map and a distance-Doppler heat map;

[0011] Step S4: Preliminarily simulate the radar heat map, output the synthetic radar heat map through the U-net-based heat map conversion network map2map, and perform training;

[0012] Step S5: Combine the synthetic radar heat map with the real radar heat map to construct a mixed data set, obtain the human pose skeleton through the human pose estimation network map2pose based on the radar heat map, and perform training to complete the human pose estimation.

[0013] Furthermore, step S2 is implemented by the following sub-steps:

[0014] S21: Simulating plane waves emitted by radar , irradiate the target human mesh model; set the target human mesh model Include triangular face elements, for one illuminated face element According to the physical optics method and the Stratton-Chu formula, the surface scattering field Defined as:

[0015]

[0016] in, are the unit vectors of the incident and viewing directions, respectively; For facet The position vector of The radar reaches the target surface distance; and are the wave number and wave impedance in free space respectively; Respectively represent the face element Surface current and surface magnetic current; is a natural constant, Is an imaginary unit.

[0017] S22: Then the bin The value of the radar cross section (RCS) at the observation point :

[0018]

[0019] in, For facet The incident electric field For facet The distance to the observation point. Repeat the above calculation method to calculate the human body mesh model In The human body mesh features including RCS are obtained by using the triangular face elements.

[0020] Furthermore, step S3 is implemented by the following sub-steps:

[0021] S31: For each bin there is a calculated radar cross section , the coordinates of these face elements in three-dimensional space are:

[0022]

[0023] in are the coordinates of the centroid of each triangular face in three-dimensional Cartesian space.

[0024] S32: Convert each facet position of the grid model to the radar sphere coordinate system. The conversion formula is:

[0025]

[0026] in, are the coordinates in radar space, is the radial distance of the bin from the radar; The elevation angle of the radar; The azimuth of the radar.

[0027] S33: Discretize the entire radar space into several angular units (Bins), corresponding to each pixel of the heat map. Assume that the resolution of the heat map is , the spatial information corresponding to each pixel:

[0028]

[0029] in, is the pitch angle, is the azimuth, For distance, For speed, and is the maximum and minimum pitch angle setting; and are the maximum and minimum values ​​of the azimuth; and The maximum and minimum distances to set; and The maximum and minimum speed settings.

[0030] For the distance-horizontal angle heat map, each pixel Corresponding , accumulate the signal strength of all facets at this angle:

[0031]

[0032] in, It is an indicator function used to determine whether the surface element is within the angle range of the current pixel.

[0033] Similarly, the distance-pitch angle heat map can be calculated as:

[0034]

[0035] For range-Doppler heatmaps, define the human body mesh model from time To exercise to time Facet Centroid The change of radial distance r is ,but Speed , the range-Doppler heat map can be calculated as:

[0036]

[0037] Furthermore, step S4 is implemented by the following sub-steps:

[0038] S41: The three preliminary heat maps obtained in step S3: range-horizontal angle, range-elevation angle and range-Doppler heat map are superimposed in the channel dimension to obtain a shape of Input feature data .

[0039] S42: Construct a map2map network based on U-net. Input the network to get the shape Output The map2map network consists of an encoder and a decoder:

[0040]

[0041]

[0042] in is an intermediate feature; , are learnable parameters for network modules.

[0043] S43: Train a U-net based map2map network and minimize it using the Adam optimizer and With the mean squared error loss between , the map2map network can accurately generate radar heat maps.

[0044] Furthermore, step S41 is implemented by the following sub-steps:

[0045] S411: For ,Depend on The intermediate features are composed of two-dimensional convolution (Conv2d) layers, and each layer uses two-dimensional maximum pooling (MaxPool2d), two-dimensional batch normalization (BatchNorm2d), and the activation function is ReLU. .

[0046] S412: For ,Depend on The network output is composed of two-dimensional inverse convolution (ConvTranspose2d) layers, and each layer uses two-dimensional batch normalization (BatchNorm2d), and the activation function is ReLU. , the shape is .

[0047] Furthermore, step S5 is implemented by the following sub-steps:

[0048] S51: The final composite heat map obtained in step S4 and real radar heat map Mixed ,Will Input to the map2pose network based on Convolutional Neural Networks (CNN), Gated Recurrent Unit (GRU) and Graph Convolutional Network (GCN) to obtain the human body posture skeleton. The specific definition of map2pose is as follows:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] in for The result of splitting by the channel dimension has the shape . is a convolutional neural network, is a gated recurrent unit, is the adaptive fusion module, To adjust the feature shape, and Graph Convolutional Neural Network is a multi-layer perceptron; , , is an intermediate feature; , , and , , , are the learnable parameters of each network module; is the final network output.

[0057] S52: Train the map2pose network and use the Adam optimizer to minimize the label value and the network prediction value The map2pose network can accurately estimate the target human pose by using the mean absolute error loss between .

[0058] In summary, the present invention uses a physical optics method to simulate radar irradiation of a human target; uses a projection method to project a target human three-dimensional grid model with an RCS value into a radar two-dimensional space to obtain a preliminary distance-horizontal angle diagram, a distance-pitch angle diagram, and a distance-Doppler diagram; inputs the preliminary radar two-dimensional heat map into a U-net-based map2map network to obtain an accurate final synthetic radar heat map; hybrid training of the synthetic radar heat map and the real radar heat map is performed on a map2pose network based on the radar heat map to obtain the target human posture data, thereby achieving accurate estimation of the human posture under a small amount of real radar data.

[0059] The beneficial effects of the present invention are:

[0060] (1) The radar heat map synthesis method based on physical optics and map2map network proposed in this paper can accurately generate simulated radar heat maps of human targets, making up for the shortage of current radar-based human posture estimation datasets.

[0061] (2) The map2pose network proposed in this invention achieves a more accurate estimation of human posture by extracting spatiotemporal features of heat map data, adaptively fusing them, and extracting structural features based on graph convolution.

[0062] (3) Compared with the traditional vision-based human posture estimation method, the present invention is based on radio frequency signals, is not affected by adverse factors such as line of sight and light intensity, and can protect the user's privacy to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the overall structure of the system of the present invention;

[0064] Figure 2 is a schematic diagram of the simulated radar position in step S2 of the present invention;

[0065] Figure 3 is a schematic diagram of the map2map network structure in step S2 of the present invention;

[0066] Figure 4 is a schematic diagram of the map2pose network structure in step S3 of the present invention;

[0067] Figure 5 It is a map2map network experiment result diagram in step S2 of the present invention;

[0068] Figure 6 This is a graph of the map2pose network experiment results in step S3 of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be described in detail below according to the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become more clear. The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] like Figure 1 As shown, the human posture estimation method based on RF heat map data enhancement includes the following steps:

[0071] Step S1: Use the sensor to obtain the grid data of the human body posture. The grid data is Each triangle surface element is represented by three vertex coordinates in three-dimensional space and their corresponding normal vector.

[0072] Step S2: Use physical optics to simulate radar, build a human body mesh model from the mesh data of S1, and obtain human body mesh features including radar cross-section by illuminating the human body mesh model. Step S2 is implemented by the following sub-steps:

[0073] S21: Simulating plane waves emitted by radar , irradiate the target human mesh model, such as Figure 2 Specific parameters include: the transmission frequency is The incident direction is horizontal and points to the human body mesh model. The antenna polarization mode is vertical polarization. The observation angle is horizontal with the simulated radar position as the center point. Degree and vertical direction Target human mesh model Include triangular face elements, for one illuminated face element According to the physical optics method and the Stratton-Chu formula, the surface scattering field Defined as:

[0074]

[0075] in, are the unit vectors of the incident and viewing directions, respectively; For facet The position vector of The radar reaches the target surface distance; and are the wave number and wave impedance in free space respectively; Respectively represent the face element Surface current and surface magnetic current; is a natural constant, Is an imaginary unit.

[0076] S22: Then the bin Radar Cross Section (RCS) at the observation point :

[0077]

[0078] in, For facet The incident electric field For facet Repeat the above calculation method to calculate the distance of the human body mesh model S The human body mesh features including RCS are obtained by using the triangular face elements.

[0079] Step S3: Process the human body grid features including the radar cross-sectional area obtained in S2 to obtain a preliminary simulated radar heat map, including: a distance-horizontal angle heat map, a distance-elevation angle heat map, and a distance-Doppler heat map. Step S3 is implemented by the following sub-steps:

[0080] S31: For each bin there is a calculated radar cross section , the coordinates of these face elements in three-dimensional space are:

[0081]

[0082] in are the coordinates of the centroid of each triangular face in three-dimensional Cartesian space.

[0083] S32: Convert each facet position of the grid model to the radar sphere coordinate system. The conversion formula is:

[0084]

[0085] in, are the coordinates in radar space, is the radial distance of the bin from the radar; The elevation angle of the radar; The azimuth of the radar.

[0086] S33: Discretize the entire radar space into several angular units (Bins), corresponding to each pixel of the heat map. Set the resolution of the heat map to , , the spatial information corresponding to each pixel is:

[0087]

[0088] in, is the pitch angle, is the azimuth, For distance, For speed, and is the maximum and minimum pitch angle setting; and are the maximum and minimum values ​​of the azimuth; and The maximum and minimum distances are set; and The maximum and minimum speed settings.

[0089] For the distance-horizontal angle heat map, each pixel Corresponding , accumulate the signal strength of all facets at this angle:

[0090]

[0091] in, It is an indicator function used to determine whether the surface element is within the angle range of the current pixel.

[0092] Similarly, the distance-pitch angle heat map can be calculated as:

[0093]

[0094] For range-Doppler heatmaps, define the human body mesh model from time To exercise to time Facet Centroid The change of radial distance r is ,but Speed , the range-Doppler heat map can be calculated as:

[0095]

[0096] Step S4: Train and use the U-net-based heat map conversion network (map2map) to process the preliminary simulated radar heat map obtained in S3, supplement the scattering and environmental noise that are difficult to simulate by the PO method, and output the final synthetic radar heat map. Step S4 is implemented by the following sub-steps:

[0097] S41: The three preliminary heat maps obtained in step S3: range-horizontal angle, range-elevation angle and range-Doppler heat map are superimposed in the channel dimension to obtain a shape of Input feature data .

[0098] S42: Build a map2map network based on U-net, the structure is as follows Figure 3 As shown, Input the network to get the shape Output The map2map network consists of an encoder and a decoder:

[0099]

[0100]

[0101] in is an intermediate feature; , are learnable parameters for network modules.

[0102] S43: Train a U-net based map2map network and minimize it using the Adam optimizer and The mean square error loss between . Figure 5 As shown, the map2map network can accurately generate radar heat maps.

[0103] Furthermore, step S41 is implemented by the following sub-steps:

[0104] S411: For ,Depend on The convolution kernel size of each layer of 2D convolution is , the padding is ; The channel dimension sizes of the heatmap in each layer of two-dimensional convolution are: ; The width and height dimensions of the heat map are: ; Each layer uses two-dimensional maximum pooling (MaxPool2d), two-dimensional batch normalization (BatchNorm2d), and the activation function is ReLU. The intermediate features obtained Shape .

[0105] S412: For , which consists of 5 layers of two-dimensional inverse convolution (ConvTranspose2d), and the convolution kernel size of each layer of two-dimensional inverse convolution is , the stride is 2, and the channel dimension sizes of the heatmap in each layer of two-dimensional deconvolution are: ; The width and height dimensions of the heat map are: ; Each layer uses two-dimensional batch normalization (BatchNorm2d) and the activation function is ReLU. The final network output , the shape is .

[0106] Step S5: Combine the synthetic radar heat map data obtained in S4 with the real radar data to construct a mixed data set. Train and use the human pose estimation network based on radar heat map (map2pose), the structure of which is as follows: Figure 4 As shown, a high-precision human body posture skeleton is obtained. Step S5 is implemented by the following sub-steps:

[0107] S51: For input features First enter , It consists of multiple convolutional layers. For each convolution layer, the first dimension of the input heat map feature increases, and the second and third dimensions decrease. Each layer uses two-dimensional batch normalization, and the activation function is ReLU. This layer is used to extract multi-scale spatial features in the heat map. Its output is .

[0108] S52: The input consists of multiple layers of gated recurrent units. To capture the time-dependent features in sequence data, the output is .

[0109] S53: enter , adaptively fuse the input features, and the output is .in The definition is as follows:

[0110]

[0111]

[0112] in, is element-wise multiplication, is the normalized exponential function, are learnable parameters.

[0113] S54: , input graph convolutional network and Extract spatiotemporal features in graph dimension:

[0114]

[0115]

[0116] in , is the batch size, is the number of joint points, is the size of the time dimension, is the channel dimension size; To adjust the feature shape, the specific method is: Linearly expand into a one-dimensional sequence, and then according to the target shape Rearrange the combination to get . The definition is as follows:

[0117]

[0118] in, , is the adjacency matrix of the predefined undirected graph G of human joints, when, is the identity matrix, yes The degree matrix of For the Layers can learn parameters, is the activation function, For the Layer input, For the Layer output. After multiple layers of graph convolution, its output is .

[0119] S55: Input into the multi-layer perceptron MLP composed of multiple linear layers to obtain the final human body posture .

[0120] S56: Train the map2pose network and use the Adam optimizer to minimize the mean absolute error loss between the label value and the network prediction value. The results are as follows Figure 6 As shown, the map2pose network can accurately estimate the target human pose.

[0121] attitude It consists of 18 human joints, each of which is represented by a three-dimensional space coordinate. Figure 6 As shown in Figure 2, the map2pose network can accurately estimate the 3D pose of the target human body using 80% synthetic radar heat map data and 20% real radar heat map data.

[0122] The present invention provides a method for estimating human posture based on radio frequency heat map data enhancement, which has high accuracy and stability. The PO method is used to simulate synthetic radar data, and the target human posture can be accurately and robustly estimated when less real radar data is used. In addition, the present invention can be applied to the fields of human-computer interaction, medical care, etc., and has good practicality and broad application prospects.

[0123] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A method for human posture estimation based on radio frequency heat map data enhancement, characterized in that: The following steps are involved: Step S1: using sensors to obtain grid data of human body posture; Step S2: Using physical optics to simulate radar, constructing a human body mesh model based on mesh data, and obtaining human body mesh features including radar cross-sectional area by illuminating the human body mesh model; the specific implementation process is as follows: S21: Simulating the plane wave E emitted by the radar i , irradiate the target human mesh model; suppose the target human mesh model S contains N triangular face elements, for one of the illuminated face elements S n According to the physical optics method and the Stratton-Chu formula, the surface scattering field Defined as: Where i and s are the unit vectors of the incident direction and the observation direction respectively; r n S n The position vector, r n The radar reaches the target surface element S n distance; k and Z are the wave number and wave impedance in free space respectively; Represents the surface element S n The surface current and surface magnetic current; e is a natural constant, j is an imaginary unit; S22: Then the bin S n The value of the radar cross section RCS at the observation point σ n : in, S n The incident electric field; R n S n distance to the observation point; Repeat the above calculation method to calculate N triangular facets in the human body mesh model S to obtain the human body mesh features including RCS; Step S3: Processing the human body mesh features including the radar cross-section to obtain a preliminary simulated radar heat map; Step S4: Preliminarily simulate the radar heat map, output the synthetic radar heat map through the U-net-based heat map conversion network map2map, and perform training; Step S5: Combine the synthetic radar heat map with the real radar heat map to construct a mixed data set, obtain the human pose skeleton through the human pose estimation network map2pose based on the radar heat map, and perform training to complete the human pose estimation.

2. The method for human posture estimation based on radio frequency heat map data enhancement according to claim 1, characterized in that: The specific implementation process of step S3 is as follows: S31: For each bin there is a calculated radar cross section σ n , the coordinates of these face elements in three-dimensional space are: S={a i :i=1,2,3,…,N} (3) where a i =(x i ,y i ,z i ) are the coordinates of the centroid of each triangular face in three-dimensional Cartesian space; S32: Convert each facet position of the grid model to the radar sphere coordinate system. The conversion formula is: in, is the coordinate in radar space, r i is the radial distance of the bin from the radar; θ i The elevation angle of the radar; The azimuth of the radar; S33: Discretize the entire radar space into several angular units, corresponding to each pixel of the heat map; suppose the resolution of the heat map is Q×Q, and the spatial information corresponding to each pixel is: Where Δθ is the pitch angle, is the azimuth, Δr is the distance, Δv is the speed, θ max and θ min is the maximum and minimum value of the pitch angle; and is the maximum and minimum value of the azimuth; r max and r min is the maximum and minimum distance; v max and v min is the maximum and minimum speed; For the distance-horizontal angle heat map, each pixel (m,n) corresponds to Accumulate the signal strength of all facets at this angle: Wherein, I() is an indicator function used to determine whether the face element is within the angle range of the current pixel; Similarly, the distance-pitch angle heat map is calculated as: For the range-Doppler heat map, the range-Doppler heat map is calculated as: Among them, the centroid b of the face element i of the human body mesh model from time t1 to time t2 is defined as i The change of radial distance r is Then b i Speed 3. The method for human posture estimation based on radio frequency heat map data enhancement according to claim 2 is characterized in that: The specific implementation process of step S4 is as follows: S41: Superimpose the three preliminary heat maps: range-horizontal angle, range-elevation angle, and range-Doppler heat map in the channel dimension to obtain an input feature data map of shape (3, Q, Q) coarse ; S42: Construct a map2map network based on U-net to map coarse Input the network to get an output map of shape (3, Q, Q) fine ;The map2map network consists of an encoder Encoder and a decoder Decoder: F0=Encoder(map coarse ,θ e ) (9) map fine =Decoder(F0,θ d ) (10) Where F0 is the intermediate feature; θ e ,θ d are learnable parameters for network modules; S43: Train a U-net based map2map network and use the Adam optimizer to minimize the map coarse With map fine The mean square error loss between .

4. The method for human posture estimation based on radio frequency heat map data enhancement according to claim 3 is characterized in that: The step S41 is implemented by the following sub-steps: S411: For Encoder, it consists of h layers of two-dimensional convolution, and each layer uses two-dimensional maximum pooling, two-dimensional batch normalization, and the activation function is ReLU; S412: For the Decoder, it consists of h layers of two-dimensional deconvolution, and each layer uses two-dimensional batch normalization, the activation function is ReLU, and the final output map is fine .

5. The method for human posture estimation based on radio frequency heat map data enhancement according to claim 4 is characterized in that: The specific implementation process of step S5 is as follows: S51: The final synthesized heat map fine And the real radar heat map real Mixed map mix , map mix Input to the map2pose network based on convolutional neural network CNN, gated recurrent unit GRU and graph convolutional network GCN to obtain the human body posture skeleton; the specific definition of map2pose is as follows: F3=AdaptiveFusion(F 2，ra ,F 2,re ,F 2,rd ,the a ) (13) F4=GCN temporal (F3,θ gt ) (14) F5=Reshape(F4) (15) F6=GCN spatial (F5,θ gs ) (16) F pose =MLP(F6,θ m ) (17) where map i i∈{ra,re,rd} is a map mix The result of segmentation by channel dimension has a shape of (1, Q, Q); CNN i i∈{ra,re,rd} is a convolutional neural network, GRU i i∈{ra,re,rd} is a gated recurrent unit, AdaptiveFusion is an adaptive fusion module, Reshape is to adjust the feature shape, GCN temporal and GCN spatial MLP is a multi-layer perceptron; F is a graph convolutional neural network. j，i , i∈{ra,re,rd}, j∈{1,2,3,4,5,6} are intermediate features; and θ a ,θ gt ,θ gs ,θ m is the learnable parameter of each network module; F pose is the final output; S52: Train the map2pose network and use the Adam optimizer to minimize the label value and the predicted value F pose The mean absolute error loss between .

6. The method for human posture estimation based on radio frequency heat map data enhancement according to claim 5, characterized in that: In step S51, the adaptive fusion module AdaptiveFusion is defined as follows: α i =Softmax(w i ·Concat(F 2，ra ,F 2,re ,F 2，rd )),i∈{ra,re,rd} (19) in, is element multiplication, Softmax() is the normalized exponential function, W i is a learnable parameter; The specific method of adjusting the characteristic shape Reshape is: Linearly expand into a one-dimensional sequence, and then rearrange and combine according to the target shape (B×T×J×C), where B is the batch size, J is the number of joint points, T is the time dimension size, and C is the channel dimension size.

Images