Through-wall human body posture estimation method based on MIMO radar

Abstract

The invention belongs to the technical field of MIMO radars and neural networks, and discloses a through-wall human body posture estimation method based on MIMO radars, which adopts an MIMO radar system to capture signals, and extracts Doppler information by analyzing original radar signals generated by human body reflection in a scene and processing the obtained signals through Fourier transform. A horizontal feature matrix and a vertical feature matrix of a human body posture are constructed in combination with an original signal and a Doppler signal, and then the feature matrixes are input into a Transform network for training and learning so as to realize human body posture estimation. The method can improve the generalization ability of the system under the condition of light data and light network, and has the characteristics of strong robustness, high stability, real-time performance and high efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of MIMO radar and neural network, and specifically relates to a through-wall human body posture estimation method based on MIMO radar. Background Art

[0002] In the past two decades, pose estimation under computer vision has made great progress, and has experienced an evolution from template matching to machine learning, deep learning and multimodal fusion, continuously improving accuracy and scalability. Pose estimation based on RGB images is generally divided into two categories: top-down and bottom-up. Top-down methods perform single-person pose estimation by first detecting each person from the image. Bottom-up methods focus on how to associate detected key points belonging to the same person. In contrast, bottom-up methods are attractive because they provide robustness for early promises and have the potential to decouple runtime complexity from the number of people in the image. However, bottom-up methods do not directly use global contextual cues from other body parts and other people. In fact, previous bottom-up methods cannot retain efficiency improvements because the final parsing requires expensive global inference. However, visual pose estimation methods cannot provide accurate predictions in situations with many occlusions, low light, and darkness.

[0003] In recent years, people have been very interested in attitude estimation based on radio frequency signals. There are two main directions. The first is to use high-frequency radars such as millimeter waves. These devices can accurately image the surface of the human body in low light or dark conditions, such as automotive radars, but cannot penetrate walls. The second type uses a lower frequency. This low-frequency radio frequency device can penetrate wall obstructions. For example, WiFi, but because the frequency of WiFi is relatively high, this limits its ability to penetrate walls and can only be used in scenes that are easy to penetrate materials. Some people have proposed using a lower frequency through-the-wall radar (1Ghz) to complete 2D attitude estimation in the case of penetrating walls. In addition, most of the existing through-the-wall radar-based attitude estimation is implemented in non-free space, and it is necessary to detect the human body in the reflection area of ​​the radar. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a through-wall human posture estimation method based on MIMO radar. The method uses MIMO through-wall radar for posture recognition. It does not need to detect the reflection area of ​​the human body on the radar, but directly predicts the 2D posture of multiple people from the radar image in an end-to-end manner. This prediction method simplifies the recognition difficulty and improves the recognition accuracy.

[0005] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0006] The present invention is a through-wall human body posture estimation method based on MIMO radar. The through-wall human body posture estimation method is implemented by a human body posture estimation network. The human body posture estimation network includes a MIMO radar system, a backbone network and a Transformer network. Specifically, the through-wall human body posture estimation method specifically includes the following steps:

[0007] Step 1: Initialize the MIMO radar system. Place the radar outside the wall to obtain radio frequency signals, i.e., wall-penetrating radar data. Initialize the camera and synchronize the tag data obtained by the camera to the MIMO radar system.

[0008] Step 2: preprocess the collected radio frequency signals to construct a feature matrix of each frame of human posture data, wherein the feature matrix of each frame of human posture data includes a horizontal original radar signal matrix, a vertical original radar signal matrix, a horizontal Doppler information matrix, and a vertical Doppler information matrix, which are spliced ​​according to the channel dimension to obtain horizontal radar data and vertical radar data, i.e., a feature matrix of each frame of human posture data;

[0009] Step 3: Input the feature matrix of each frame of the horizontal radar data and the vertical radar data, i.e., the human body posture data, into the backbone network for feature extraction to obtain the horizontal feature matrix p of the human body posture. h and the vertical feature matrix p v ;

[0010] Step 4: The horizontal feature matrix p of the human body posture obtained in step 3 h and the vertical feature matrix p v After addition, the data is input into the Transformer network to complete the network training.

[0011] Step 5: After training, the system can predict the untrained human posture data with better results.

[0012] A further improvement of the present invention is that: the backbone network in the human posture estimation network includes multiple convolution modules, batch normalization layers, nonlinear activation functions ReLU and position encoders, each of the convolution modules is composed of a volume and a convolution layer of size (7,7), the convolution layer has a stride (2,2) and padding (3,3), and the number of input channels is 2, in each residual block, the first layer belongs to the features of the horizontal original radar signal matrix and the vertical original radar signal matrix, starting from the second layer, the input features of each residual block are the output features of the previous residual block, in each residual block, the input features first pass through multiple convolution layers and batch normalization layers, and then through a jump connection with the input features, and through the nonlinear activation function ReLU, to extract the horizontal feature matrix p of the human posture. h and the vertical feature matrix p v , is the horizontal feature matrix p for each human postureh and the vertical feature matrix p v Add position encoder.

[0013] A further improvement of the present invention is that the Transformer network in the human posture estimation network is composed of a plurality of encoder and decoder layers stacked together, which is very effective in processing global context information and spatial relationships. Each encoder and decoder layer is composed of a deformable attention mechanism and a feedforward neural network. The deformable attention mechanism allows each attention head to adaptively select key points at different positions according to the input features, and the human posture estimation network supports processing multi-feature layer inputs. The use of multi-layer inputs enables the model to better capture multi-scale information in the input image. The use of deformable attention in the encoder and decoder can flexibly adjust the computational complexity and receptive field size.

[0014] A further improvement of the present invention is that in step 1, obtaining the radio frequency signal, i.e., through-wall radar data, through the MIMO radar system specifically includes the following steps:

[0015] Step 1.1: The radar transmits an FMCW signal, which passes through the wall and reflects off the human body. The reflected signal received by the radar is passed through a low noise amplifier (LNA) and a mixer to obtain an intermediate frequency signal; this signal contains the distance information of the target at its frequency.

[0016] Step 1.2: Assume that the MIMO radar has M transmitting antennas and N receiving antennas. The sampling of the intermediate frequency signal by mn transceivers is expressed as a complex discrete function with respect to time t:

[0017]

[0018] Among them, R m,n is the distance of the intermediate frequency signal from the mth transmitting antenna to the target point and back to the nth receiving antenna, λ is the wavelength of the center frequency of the FMCW signal, A(t) is the amplitude at time t; j is the imaginary unit, and e is the natural exponent;

[0019] Step 1.3: The sampling signals of all antenna transceiver pairs are called a frame, that is, a three-dimensional matrix with dimensions (M, N, T) is obtained, that is, the radio frequency signal.

[0020] A further improvement of the present invention is that in step 1, the MINO radar is composed of a 2-D planar array, including 4 transmitting antennas and 16 receiving antennas, the receiving antennas are placed horizontally, and each adjacent two receiving antennas are spaced 10 cm apart, and the transmitting antennas are placed vertically, and each adjacent two transmitting antennas are spaced 20 cm apart.

[0021] The further improvement of the present invention is that: Step 2, preprocessing the collected radio frequency signal to construct a feature matrix of each frame of human posture data, specifically comprising the following steps:

[0022] Step 2.1: Perform a three-dimensional discrete Fourier transform on the three-dimensional matrix of a frame to obtain the three-dimensional power matrix P of the detection space as follows:

[0023]

[0024] in θ,γ represent the pitch angle, azimuth angle and distance respectively, k represents the slope, c represents the signal propagation speed, s m ,s n Respectively represent the spacing between the transmitting antenna and the receiving antenna;

[0025] Step 2.2, obtaining a horizontal original radar signal matrix, a vertical original radar signal matrix, a horizontal Doppler information matrix, and a vertical Doppler information matrix by projection;

[0026] Step 2.3, repeating steps 2.1 to 2.2, processing each frame of the RF signal collected by the radar, and obtaining the original radar data of all frames, i.e., the horizontal original radar signal matrix, the vertical original radar signal matrix, and the Doppler data, i.e., the horizontal Doppler information matrix and the vertical Doppler information matrix;

[0027] Step 2.4, respectively concatenate the horizontal original radar signal matrix and the horizontal Doppler information matrix, the vertical original radar signal matrix and the vertical Doppler information matrix of each frame according to the channel dimension to obtain the feature matrix of the horizontal radar data and the vertical radar data, i.e., the human body posture data of each frame:

[0028] raw′ h =[raw h |dop h ]

[0029] raw v ′=[raw v |dop v ]

[0030] Among them, raw h and dop h Represent the horizontal raw radar signal matrix and the horizontal Doppler information matrix, raw v and dop v Respectively represent the vertical original radar signal matrix and the vertical Doppler information matrix, raw′ h is the horizontal radar data, raw v ′ is the vertical radar data.

[0031] A further improvement of the present invention is that in step 3, the feature matrix of each frame of the horizontal radar data and the vertical radar data, i.e., the human body posture data, is input into the backbone network for feature extraction to obtain the horizontal feature matrix p of the human body posture. h and the vertical feature matrix p v , specifically including the following steps:

[0032] Step 3.1: Get the horizontal radar data raw′ h And vertical radar data raw v 'Enter the backbone network;

[0033] Step 3.2: In the backbone network, horizontal radar data raw′ h And vertical radar data raw v 'Through a convolutional layer, followed by batch normalization and ReLU function processing, local spatial features are extracted;

[0034] Step 3.3: After passing through the backbone network, the horizontal feature matrix p is output h and the vertical feature matrix p v ,These features contain information from low level to high level. These multi-feature matrices are stored, representing the features extracted by different convolutional layers for subsequent processing and fusion.

[0035] Step 3.4: The horizontal feature matrix p output by the backbone network h and the vertical feature matrix p v Collect and enter the position encoding module to finally obtain the horizontal feature matrix p of the human body posture h and the vertical feature matrix p v .

[0036] A further improvement of the present invention is that: Step 4 converts the horizontal feature matrix p of the human body posture into h and the vertical feature matrix p v After addition, the data are input into the Transformer network to complete the network training. The training steps specifically include the following steps:

[0037] Step 4.1, horizontal feature matrix p h and the vertical feature matrix p v First, it is sent to the encoder of the Transformer network. The encoder integrates information from different spaces and scales through a variable multi-head attention mechanism. In the encoder, the horizontal feature matrix p h and the vertical feature matrix p v are aggregated and fused, allowing the model to understand the relationship between features from different levels. Each level feature matrix p h and the vertical feature matrix p vAfter the linear layer and dropout operation of the encoder, the encoded features are obtained to ensure the stability and generalization ability of the features;

[0038] Step 4.2: The encoded features obtained in step 4.1 are sent to the decoder. The decoder matches these features with the target of posture estimation. The encoded features are passed to the posture estimation network through embedding query. The Transformer network finds the key points of the corresponding human body through the learned embedding vector.

[0039] Step 4.3: The decoder output is associated with the key point positions in the pose estimation, and the coordinate prediction of each joint is generated through the fully connected layer. Using back propagation and loss function, the Transformer network continuously updates the parameters to predict the 2D key point positions of the human body.

[0040] Step 4.4: Calculate the loss function by comparing the predicted key points with the true labels; the losses used include position loss and classification loss. The back propagation process minimizes the loss and optimizes the model parameters.

[0041] Step 4.5: Continuously iterate steps 4.1 to 4.4, train multiple batches of data, update the model parameters in each iteration, and gradually improve the prediction accuracy of the network.

[0042] The beneficial effects of the present invention are:

[0043] The present invention uses MIMO through-wall radar for gesture recognition. The radar has a low operating frequency and can penetrate walls and detect targets behind them. The gesture estimation based on this through-wall radar can provide more help in scenarios where obstacles exist;

[0044] The present invention uses a multi-feature matrix to construct the human body posture, so that the information of the human body posture can be fully extracted, which also provides certain help for the system to learn the human body posture features more perfectly;

[0045] The present invention uses an end-to-end posture estimation network based on Transformer to output a set of 2D posture candidate sets and a set of true and false classification sets of people. Then, the human posture is determined by analyzing the prediction results of the classification set. It is not necessary to detect the reflection area of ​​the human body on the radar, but to directly predict the 2D posture of multiple people from the radar image in an end-to-end manner. This prediction method simplifies the recognition difficulty and improves the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the posture estimation method in an embodiment of the present invention.

[0047] Figure 2It is a flowchart of constructing multi-feature matrix posture estimation in an embodiment of the present invention.

[0048] Figure 3 This is the working scenario in the embodiment of the present invention.

[0049] Figure 4 It is a network structure diagram of the present invention.

[0050] Figure 5 It is a backbone network structure diagram of the present invention. DETAILED DESCRIPTION

[0051] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0052] The present invention proposes a through-wall human posture estimation method based on MIMO radar. The method first performs data preprocessing on the original signal collected by the radar, preliminarily extracts the posture information before inputting it into the network, and processes each frame of data into a matrix so that the information of the human posture can be retained. The feature matrix construction enables the posture recognition system to further obtain feature information, which provides the possibility for the posture recognition system to be better put into practical applications.

[0053] like Figure 1-5 As shown, the present invention is a through-wall human posture estimation method based on MIMO radar, and the through-wall human posture estimation method is implemented by a human posture estimation network, and the human posture estimation network includes a MIMO radar system, a backbone network and a Transformer network. Specifically, the through-wall human posture estimation method specifically includes the following steps:

[0054] Step 1: Initialize the MIMO radar system. Place the radar outside the wall to obtain radio frequency signals, i.e., through-wall radar data. Initialize the camera and synchronize the tag data obtained by the camera to the MIMO radar system.

[0055] The MINO radar consists of a 2-D planar array, including 4 transmitting antennas and 16 receiving antennas. The receiving antennas are placed horizontally, with a 10 cm interval between each adjacent receiving antenna, and the transmitting antennas are placed vertically, with a 20 cm interval between each adjacent transmitting antenna. This array is equivalent to a 64-antenna array of the transceiver, which can detect a larger spatial range.

[0056] The MIMO radar consists of an FMCW signal source, a power amplifier (PA), a mixer, a low noise amplifier (LNA), a RF switch, an antenna array, and a field programmable gate array (FPGA). The signal source outputs an FMCW signal with a bandwidth of 1 GHz (1-2 GHz) and a period of 500 μs. This signal brings high distance resolution and the ability to penetrate brick walls to the system.

[0057] Assuming that obtaining the RF signal, i.e., through-wall radar data, through the MIMO radar system specifically includes the following steps:

[0058] Step 1.1: The radar transmits an FMCW signal, which passes through the wall and reflects off the human body. The reflected signal received by the radar is passed through a low noise amplifier (LNA) and a mixer to obtain an intermediate frequency signal; this signal contains the distance information of the target at its frequency.

[0059] Step 1.2: Assume that the MIMO radar has M transmitting antennas and N receiving antennas. The sampling of the intermediate frequency signal by mn transceivers is expressed as a complex discrete function with respect to time t:

[0060]

[0061] Among them, R m,n is the distance of the intermediate frequency signal from the mth transmitting antenna to the target point and back to the nth receiving antenna, λ is the wavelength of the center frequency of the FMCW signal, A(t) is the amplitude at time t; j is the imaginary unit, and e is the natural exponent;

[0062] Step 1.3: The sampling signals of all antenna transceiver pairs are called a frame, that is, a three-dimensional matrix with dimensions (M, N, T) is obtained, that is, the radio frequency signal.

[0063] Step 2: preprocess the collected radio frequency signal to construct a feature matrix of each frame of human posture data, wherein the feature matrix of each frame of human posture data includes a horizontal original radar signal matrix, a vertical original radar signal matrix, a horizontal Doppler information matrix, and a vertical Doppler information matrix, which are spliced ​​according to the channel dimension to obtain horizontal radar data and vertical radar data, i.e., a feature matrix of each frame of human posture data, specifically comprising the following steps:

[0064] Step 2.1: Perform a three-dimensional discrete Fourier transform on the three-dimensional matrix of a frame to obtain the three-dimensional power matrix P of the detection space as follows:

[0065]

[0066] in θ,γ represent the pitch angle, azimuth angle and distance respectively, k represents the slope, c represents the signal propagation speed, s m ,s nRespectively represent the spacing between the transmitting antenna and the receiving antenna;

[0067] Step 2.2, obtaining a horizontal original radar signal matrix, a vertical original radar signal matrix, a horizontal Doppler information matrix, and a vertical Doppler information matrix by projection;

[0068] Step 2.3, repeating steps 2.1 to 2.2, processing each frame of the RF signal collected by the radar, and obtaining the original radar data of all frames, i.e., the horizontal original radar signal matrix, the vertical original radar signal matrix, and the Doppler data, i.e., the horizontal Doppler information matrix and the vertical Doppler information matrix;

[0069] Step 2.4, respectively concatenate the horizontal original radar signal matrix and the horizontal Doppler information matrix, the vertical original radar signal matrix and the vertical Doppler information matrix of each frame according to the channel dimension to obtain the feature matrix of the horizontal radar data and the vertical radar data, i.e., the human body posture data of each frame:

[0070] raw′ h =[raw h |dop h ]

[0071] raw v ′=[raw v |dop v ]

[0072] Among them, raw h and dop h Respectively represent the horizontal raw radar signal matrix and the horizontal Doppler information matrix, raw v and dop v Respectively represent the vertical original radar signal matrix and the vertical Doppler information matrix, raw′ h is the horizontal radar data, raw v ′ is the vertical radar data.

[0073] Step 3: Input the feature matrix of each frame of the horizontal radar data and the vertical radar data, i.e., the human body posture data, into the backbone network for feature extraction to obtain the horizontal feature matrix p of the human body posture. h and the vertical feature matrix p v .

[0074] like Figure 5As shown, the backbone network in the present invention includes multiple convolution modules, batch normalization layers, nonlinear activation functions ReLU and position encoders, each of which is composed of a volume and a convolution layer of size (7,7), the convolution layer has a stride (2,2) and padding (3,3), and the number of input channels is 2. In each residual block, the first layer is characterized by a horizontal original radar signal matrix and a vertical original radar signal matrix. Starting from the second layer, the input feature of each residual block is the output feature of the previous residual block. In each residual block, the input feature first passes through multiple convolution layers and batch normalization layers, and then is connected with the input feature through a jump connection and passes through a nonlinear activation function ReLU to extract the horizontal feature matrix p of the human posture. h and the vertical feature matrix p v , is the horizontal feature matrix p for each human posture h and the vertical feature matrix p v Add position encoder.

[0075] The backbone network freezes some layers, and the frozen layers will no longer update their weights, thereby reducing the amount of training calculations and improving efficiency. In addition, the backbone network can return multi-layer features, such as the second, third, and fourth layers, for more refined multi-scale feature extraction.

[0076] In addition, the backbone network also incorporates a position encoding module, which is used to embed spatial position information in the input feature map to help the model better understand the geometric relationship of the input data. Position encoding is achieved by adding coordinate-related encoding information to the feature map, thereby enhancing the model's perception of spatial information.

[0077] In each residual block, the input features first pass through multiple convolutional layers and batch normalization layers, then are added to the input features through skip connections, and pass through the nonlinear activation function ReLU. The output formula of the residual block can be expressed as:

[0078] output = ReLU (input) + F (input)

[0079] Among them, input represents the input of the residual block, and F represents the nonlinear transformation in the residual block.

[0080] After extracting the features, in order to better preserve the spatial information, position encoding is added to each extracted feature map. Position encoding is used to provide position information to the network to make up for the problem that the convolution operation does not capture enough spatial information. The addition of position encoding can be expressed by the following formula:

[0081] Feature encoded=Feature+Positional Encoding

[0082] Among them, Feature is the feature map extracted by Backbone, and Positional Encoding is the position information encoding matrix that matches the shape of the feature map.

[0083] Through the backbone network structure and position encoding, a multi-scale, high-level feature representation of the RF signal with position information can be obtained, thereby providing richer and more effective features for subsequent posture estimation.

[0084] The horizontal radar data and the vertical radar data, i.e., the feature matrix of each frame of human posture data, are input into the backbone network for feature extraction to obtain the horizontal feature matrix p of the human posture. h and the vertical feature matrix p v , specifically including the following steps:

[0085] Step 3.1: Get the horizontal radar data raw′ h And vertical radar data raw v 'Enter the backbone network;

[0086] Step 3.2: In the backbone network, horizontal radar data raw′ h And vertical radar data raw v 'Through a convolutional layer, followed by batch normalization and ReLU function processing, local spatial features are extracted;

[0087] Step 3.3: After passing through the backbone network, the horizontal feature matrix p is output h and the vertical feature matrix p v , these features contain information from low level to high level. These multi-feature matrices are stored, representing the features extracted by different convolutional layers, for subsequent processing and fusion.

[0088] Step 3.4: The horizontal feature matrix p output by the backbone network h and the vertical feature matrix p v After being collected, it enters the position encoding module, which generates corresponding position information for the feature matrix of each layer to describe the position of the feature in the original space. The position information is added to the feature matrix through a specific encoding method to help the subsequent model better understand the spatial relationship between each feature, and finally obtain the horizontal feature matrix p of the human body posture. h and the vertical feature matrix p v .

[0089] Step 4: The horizontal feature matrix p of the human body posture obtained in step 3 hand the vertical feature matrix p v After addition, the data are input into the Transformer network to complete the network training.

[0090] The Transformer network is composed of multiple encoder and decoder layers stacked together. It is very effective in processing global context information and spatial relationships. Each encoder and decoder layer consists of a deformable attention mechanism and a feedforward neural network. The deformable attention mechanism allows each attention head to adaptively select key points at different positions based on the input features, and the human pose estimation network supports processing multi-feature layer inputs. Using multiple layers of input enables the model to better capture multi-scale information in the input image. Using deformable attention in the encoder and decoder can flexibly adjust the computational complexity and receptive field size.

[0091] The feature matrix is ​​input into the encoder module after position encoding. The multi-head attention mechanism can calculate multiple different attention weights from the input features and fuse the information to capture dependencies of different scales and directions. Its output is processed through residual connection and layer normalization, and then enters the forward propagation network to further extract deep feature information. The operation of the encoder can be expressed by the following formula:

[0092] Encoded Feature=LayerNorm(MultiHeadAttention(Feature)+Feature)+FFN(Feature)

[0093] By stacking multiple encoder layers, the global information of features is fully modeled, which is especially important for the complex representation of human poses.

[0094] The decoder module follows the standard Transformer decoder structure. The decoder layer first extracts the encoded features from the encoder through the self-attention mechanism and combines the learnable query embeddings to generate target candidate queries. Specifically, the d position embeddings are first converted into N target candidate queries. These query embeddings help the decoder module focus on the key positions in the image. The decoder focuses on the input features extracted by the encoder through the cross-attention mechanism to extract the most relevant features for each candidate query. After multiple layers of decoders are stacked, the spatial position information and posture prediction of the target candidate are gradually refined.

[0095] Finally, the output of the decoder is decoded by a feedforward network (FFN) to generate the pose coordinates and classification label information of the person. The pose coordinates contain the positions of the joints, and the classification labels are used to identify different people. The decoder module can be expressed by the formula:

[0096]

[0097] Among them, Query is the target candidate query generated by the decoder, and Encoded Feature is the feature matrix output by the encoder.

[0098] Through the stacking and coordination of encoders and decoders, the Transformer network can effectively learn the posture characteristics of each person and the global spatial relationship, and generate posture estimation results for multiple people. Combined with the feature extraction and position encoding in step 3, the complete network can perform in-depth analysis of the complex input of the RF signal and output high-precision human posture estimation results.

[0099] The training steps specifically include the following steps:

[0100] Step 4.1, horizontal feature matrix p h and the vertical feature matrix p v First, it is sent to the encoder of the Transformer network. The encoder integrates information from different spaces and scales through a variable multi-head attention mechanism. In the encoder, the horizontal feature matrix p h and the vertical feature matrix p v are aggregated and fused, allowing the model to understand the relationship between features from different levels. Each level feature matrix p h and the vertical feature matrix p v After the linear layer and dropout operation of the encoder, the encoded features are obtained to ensure the stability and generalization ability of the features;

[0101] Step 4.2: The encoded features obtained in step 4.1 are sent to the decoder. The decoder matches these features with the target of posture estimation. The encoded features are passed to the posture estimation network through embedding query. The Transformer network finds the key points of the corresponding human body through the learned embedding vector.

[0102] Step 4.3: The decoder output is associated with the key point positions in the pose estimation, and the coordinate prediction of each joint is generated through the fully connected layer. Using back propagation and loss function, the Transformer network continuously updates the parameters to predict the 2D key point positions of the human body.

[0103] Step 4.4: Calculate the loss function by comparing the predicted key points with the true labels; the losses used include position loss and classification loss. The back propagation process minimizes the loss and optimizes the model parameters.

[0104] Step 4.5: Continuously iterate steps 4.1 to 4.4, train multiple batches of data, update the model parameters in each iteration, and gradually improve the prediction accuracy of the network.

[0105] Step 5: After training, the system can predict the untrained human posture data with better results.

[0106] The trained system can directly recognize new posture data without additional training. The model has the ability to learn and predict posture features directly from untrained data, thus enabling the recognition of new postures.

[0107] In order to verify the present invention, simulation and experiments are proposed, which are specifically described as follows:

[0108] 1. Dataset

[0109] like Figure 3 As shown in the figure, there is a brick wall with a thickness of 28cm and a free space of 3m*8m, and the radar is placed outside the wall. It includes scenes of one person and two people walking freely in the space.

[0110] 2. Metrics

[0111] (1) Object Keypoint Similarity OKS: OKS is used to measure the average keypoint localization quality of all predefined object keypoints.

[0112] (2) Average Precision AP a (Average Precision): AP a is the proportion of key points that satisfy OKS ≥ a. It is calculated based on each key point of the entire dataset.

[0113] 3. Comparison of algorithms

[0114] (1)RF-Pose (Zhao M, Li T, Abu Alsheikh M, et al. Through-wall human poseestimation using radio signals [C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition. 2018:7356-7365.)

[0115] (2)PRSNet (Z.Zheng et al., "Recovering Human Pose and Shape FromThrough-the-Wall Radar Images," in IEEE Transactions on Geoscience and RemoteSensing, vol.60, pp.1-15, 2022, Art no.5112015, doi:10.1109 / TGRS.2022.3162333.)

[0116] 4. Experimental results

[0117] As shown in Table 1, according to the method described in the present invention, the collected data is processed into a multi-feature matrix and input into the deep learning network for training. After the system training is completed, the human body posture estimation of the untrained data is performed. In addition, according to the human body posture estimation method described in Part 3, the data set is processed, and a comparative analysis of the posture estimation results of the new data is carried out.

[0118] Table 1 AP of untrained data

[0119] AP AP AP AP AP AP AVE RF-Pose 83.6 83.3 82.2 81.7 79.5 82.1 PRSNet 89.2 88.3 87.3 86.1 84.4 87.1 The present invention 96.6 93.9 91.4 88.6 84.9 91.1

[0120] The experimental results show that the average accuracy of the posture estimation of the untrained data by the present invention is higher than that of other methods after the system is completed, which proves the true effectiveness of the present invention.

[0121] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A through-wall human posture estimation method based on MIMO radar, characterized by: The through-wall human body posture estimation method is implemented by a human body posture estimation network, and the human body posture estimation network includes a MINO radar system, a backbone network and a Transformer network. Specifically, the through-wall human body posture estimation method specifically includes the following steps: Step 1: Initialize the MIMO radar system. Place the radar outside the wall to obtain radio frequency signals, i.e., wall-penetrating radar data. Initialize the camera and synchronize the tag data obtained by the camera to the MIMO radar system. Step 2: preprocess the collected radio frequency signals to construct a feature matrix of each frame of human posture data, wherein the feature matrix of each frame of human posture data includes a horizontal original radar signal matrix, a vertical original radar signal matrix, a horizontal Doppler information matrix, and a vertical Doppler information matrix, which are spliced ​​according to the channel dimension to obtain horizontal radar data and vertical radar data, i.e., a feature matrix of each frame of human posture data; Step 3: Input the feature matrix of each frame of the horizontal radar data and the vertical radar data, i.e., the human body posture data, into the backbone network for feature extraction to obtain the horizontal feature matrix and the vertical feature matrix of the human body posture; Step 4: Add the horizontal feature matrix and the vertical feature matrix of the human body posture obtained in step 3 and input them into the Transformer network to complete the network training; Step 5: After training, the system can predict the untrained human posture data with better results.

2. The method for estimating a human body posture through a wall based on MIMO radar according to claim 1, characterized in that: The backbone network in the human posture estimation network includes multiple convolution modules, batch normalization layers, nonlinear activation functions ReLU and position encoders. Each convolution module consists of a volume and a convolution layer of size (7,7). The convolution layer has a stride of (2,2) and padding of (3,3), and the number of input channels is 2. In each residual block, the first layer is characterized by a horizontal original radar signal matrix and a vertical original radar signal matrix. Starting from the second layer, the input feature of each residual block is the output feature of the previous residual block. In each residual block, the input feature first passes through multiple convolution layers and batch normalization layers, and then is connected with the input feature through a jump connection and passes through a nonlinear activation function ReLU to extract the horizontal feature matrix and the vertical feature matrix of the human posture, and a position encoder is added to the horizontal feature matrix and the vertical feature matrix of each human posture.

3. The method for estimating human body posture through walls based on MIMO radar according to claim 1, characterized in that: The Transformer network in the human pose estimation network is composed of multiple encoder and decoder layers stacked together, each of which is composed of a deformable attention mechanism and a feedforward neural network. The deformable attention mechanism allows each attention head to adaptively select key points at different positions according to input features, and the human pose estimation network supports processing multi-feature layer inputs.

4. A through-wall human body posture estimation method based on MIMO radar according to any one of claims 1 to 3, characterized in that: Step 2: pre-process the collected radio frequency signal to construct a feature matrix of each frame of human posture data, which specifically includes the following steps: Step 2.1: Perform a three-dimensional discrete Fourier transform on the three-dimensional matrix of a frame to obtain the three-dimensional power matrix P of the detection space as follows: in They represent the pitch angle, azimuth angle and distance respectively, k represents the slope, c represents the signal propagation speed, s m ,s n Respectively represent the spacing between the transmitting antenna and the receiving antenna; Step 2.2, obtaining a horizontal original radar signal matrix, a vertical original radar signal matrix, a horizontal Doppler information matrix, and a vertical Doppler information matrix by projection; Step 2.3, repeating steps 2.1 to 2.2, processing each frame of the RF signal collected by the radar, and obtaining the original radar data of all frames, i.e., the horizontal original radar signal matrix, the vertical original radar signal matrix, and the Doppler data, i.e., the horizontal Doppler information matrix and the vertical Doppler information matrix; Step 2.4, respectively concatenate the horizontal original radar signal matrix and the horizontal Doppler information matrix, the vertical original radar signal matrix and the vertical Doppler information matrix of each frame according to the channel dimension to obtain the feature matrix of the horizontal radar data and the vertical radar data, i.e., the human body posture data of each frame: raw ′ h =[raw h |dop h ] raw′ v =[raw v |dop v ] Among them, raw h and dop h Represent the horizontal raw radar signal matrix and the horizontal Doppler information matrix, raw v and dop v Respectively represent the vertical raw radar signal matrix and the vertical Doppler information matrix, raw ′ h is the horizontal radar data, raw v ′ It is the vertical radar data.

5. The method for estimating human body posture through walls based on MIMO radar according to claim 4, characterized in that: In step 3, the horizontal radar data and the vertical radar data, i.e., the feature matrix of each frame of human posture data, are input into the backbone network for feature extraction to obtain the horizontal feature matrix p of the human posture. h and the vertical feature matrix p v , specifically including the following steps: Step 3.1: Get the horizontal radar data raw ′ h And vertical radar data raw v ′ Enter the backbone network; Step 3.2: In the backbone network, the horizontal radar data raw ′ h And vertical radar data raw v ′ Through a convolutional layer, followed by batch normalization and ReLU function processing, local spatial features are extracted; Step 3.3: After passing through the backbone network, the horizontal feature matrix p is output h and the vertical feature matrix p v ; Step 3.4: The horizontal feature matrix p output by the backbone network h and the vertical feature matrix p v Collect and enter the position encoding module to finally obtain the horizontal feature matrix p of the human body posture h and the vertical feature matrix p v .

6. The method for estimating human body posture through walls based on MIMO radar according to claim 5, characterized in that: Step 4: The horizontal feature matrix p of the human body posture h and the vertical feature matrix p v After addition, the data are input into the Transformer network to complete the network training. The training steps specifically include the following steps: Step 4.1, horizontal feature matrix p h and the vertical feature matrix p v It is first fed into the encoder of the Transformer network, where the horizontal feature matrix p h and the vertical feature matrix p v Aggregated and fused, each horizontal feature matrix p h and the vertical feature matrix p v After the linear layer and dropout operation of the encoder, the encoded features are obtained; Step 4.2: The encoded features obtained in step 4.1 are sent to the decoder. The decoder matches these features with the target of posture estimation. The encoded features are passed to the posture estimation network through embedding query. The Transformer network finds the key points of the corresponding human body through the learned embedding vector. Step 4.3: The decoder output is associated with the key point positions in the pose estimation, and the coordinate prediction of each joint is generated through the fully connected layer. Using back propagation and loss function, the Transformer network continuously updates the parameters to predict the 2D key point positions of the human body. Step 4.4, calculate the loss function by comparing the predicted key points with the true labels; Step 4.5: Continuously iterate steps 4.1 to 4.4, train multiple batches of data, update the model parameters in each iteration, and gradually improve the prediction accuracy of the network.

7. The method for estimating human body posture through walls based on MIMO radar according to claim 1, characterized in that: In step 1, obtaining the radio frequency signal, i.e., through-wall radar data, through the MIMO radar system specifically includes the following steps: Step 1.1, the radar transmits an FMCW signal, which passes through the wall and is reflected by the human body. The reflected signal received by the radar is passed through a low noise amplifier (LNA) and a mixer to obtain an intermediate frequency signal; Step 1.2: Assume that the MIMO radar has M transmitting antennas and N receiving antennas. The sampling of the intermediate frequency signal by mn transceivers is expressed as a complex discrete function with respect to time t: Among them, R m,n is the distance of the intermediate frequency signal from the mth transmitting antenna to the target point and back to the nth receiving antenna, λ is the wavelength of the center frequency of the FMCW signal, A(t) is the amplitude at time t; j is the imaginary unit, and e is the natural exponent; Step 1.3: The sampling signals of all antenna transceiver pairs are called a frame, that is, a three-dimensional matrix with dimensions (M, N, T) is obtained, that is, the radio frequency signal.

8. The method for estimating a human body posture through a wall based on MIMO radar according to claim 1, characterized in that: In step 1, the MINO radar consists of a 2-D planar array, including 4 transmitting antennas and 16 receiving antennas. The receiving antennas are placed horizontally, with a spacing of 10 cm between each adjacent receiving antenna, and the transmitting antennas are placed vertically, with a spacing of 20 cm between each adjacent transmitting antenna.

Images