Multi-person human behavior recognition method based on MIMO through-wall radar feature fusion

Abstract

This invention belongs to the fields of MIMO through-wall radar and computer vision, and discloses a method for multi-person human behavior recognition based on MIMO through-wall radar feature fusion. The method performs preliminary feature extraction on behavior action information, including raw features and Doppler features. The collected radio frequency signals are preprocessed to construct a three-dimensional spatial power matrix and a Doppler behavior matrix. The feature matrices of multiple frames are combined to construct a behavior feature model and input it into the FS-Convnext network model to complete the training.

Description

Technical Field

[0001] This invention belongs to the fields of MIMO through-wall radar and computer vision, specifically relating to a method for recognizing human behavior in multiple people based on MIMO through-wall radar feature fusion. Background Technology

[0002] Human behavior recognition aims to automatically detect, analyze, identify, and classify human movements, postures, or behaviors from sensor data, video sequences, or other sensing devices, so that computing systems can provide assistance to users. It has wide applications in areas such as monitoring daily living activities, video analytics, and autonomous driving. The rapid development of artificial intelligence (AI) technology has had a positive impact on human behavior recognition.

[0003] Almost all objects reflect signals, causing radio frequency signals to carry a lot of information unrelated to people or tasks. For multi-position human behavior recognition systems, most work and research focuses on comparing the performance of single-position models with that of fused position models. Multi-person action recognition often requires integrating information from multiple sensors, such as RGB cameras, depth cameras, and inertial sensors. In previous work, some studies have not separated the raw data by position, but instead concatenated all positions and fed them into the network for training; others have focused more on kernel optimization of convolutional neural networks and multimodal sensor separation.

[0004] However, in scenarios involving multiple people passing through walls, the area corresponding to each person in the radar radio frequency signal may be very small and unevenly distributed. Most of the radio frequency signal is reflected from objects in the environment, so it may be strongly biased by background noise. It is not immediately clear which part of the reflected signal corresponds to a person in the scene, and it is impossible to learn useful information about people. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention provides a multi-person human behavior recognition method based on MIMO through-wall radar feature fusion. This method performs preliminary feature extraction on behavioral action information, including original features and Doppler features, segments different users in the scene, and superimposes multi-frame information matrices to achieve real-time behavior recognition. This provides the possibility for better application of behavior recognition systems in practical applications, improving recognition accuracy in complex environments such as congestion and noise interference, and enhancing the robustness of the system.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is a method for recognizing multiple human behaviors based on MIMO through-wall radar feature fusion. The method is implemented using a MIMO through-wall radar system and specifically includes the following steps:

[0008] Step 1: Initialize the MIMO through-wall radar system, configure the information sampling parameters, and acquire the radio frequency signals collected by the MIMO through-wall radar. The information sampling parameters include: power amplifier, low noise amplifier, antenna array, mixer, signal source programmable gate array, etc.

[0009] Step 2: Preprocess the radio frequency signals acquired in Step 1. Combine the sampling signals of each transmit and receive pair antenna in Step 1 into a frame, perform a three-dimensional discrete Fourier transform on it, and construct a three-dimensional spatial power matrix.

[0010] Step 3: Initialize the three-dimensional spatial power matrix constructed in Step 2 to construct the original feature matrix, perform dynamic background removal on the constructed original feature matrix, and construct the Doppler behavior matrix;

[0011] Step 4: Separate the original feature matrix and the Doppler behavior matrix and obtain the user feature matrix block. Combine the feature matrices of multiple frames of data to construct a behavior feature model.

[0012] Step 5: Input the behavioral feature model obtained in Step 4 into the FS-Convnext network model. Through different convolutional layers and residual blocks in the FS-Convnext network model, feature extraction of different behaviors is achieved, and the training of the MIMO through-wall radar system is completed.

[0013] Step 6: After training, the MIMO through-wall radar system can recognize the behavior of untrained personnel without needing to retrain using the FS-Convnext network model.

[0014] A further improvement of the present invention is that, in step 3, dynamic background removal is performed on the original feature matrix to construct a Doppler behavior matrix, specifically as follows:

[0015] If the original data matrix is ​​data, and the horizontal and vertical background data are b respectively... h and b v Then we have:

[0016]

[0017]

[0018] Where, raw h The raw horizontal feature matrix is ​​the original horizontal feature matrix. v The original vertical feature matrix, dop h The doppler horizontal characteristic matrix, dopv Let N be the Doppler vertical feature matrix, and N be the total number of samples.

[0019] A further improvement of the present invention is that step 4 separates the original feature matrix and the Doppler behavior matrix and obtains the user feature matrix block, and combines the feature matrices of multiple frames of data to construct a behavior feature model, specifically including the following steps:

[0020] Step 4.1: Project the 3D behavior of the person captured by the camera onto the radar heat map to determine the reflection area of ​​the k-th person. Let the coordinate set of the k-th target be... Then the projected coordinates of each key point on the horizontal radar heat map Projected coordinates on vertical radar thermal map for:

[0021]

[0022] Among them, s d For the range resolution of radar detection, s θ For horizontal angular resolution, s γ Vertical angular resolution;

[0023] Step 4.2: Obtain the projected coordinates of each key point on the horizontal radar heatmap. Projected coordinates on vertical radar thermal map Projected coordinates on the horizontal radar thermal map, serving as the boundary of the region of interest. Projected coordinates on vertical radar thermal map The regions of interest in the horizontal and vertical radar heatmaps are as follows:

[0024]

[0025] The captured sensing area constitutes one frame of data;

[0026] Step 4.3: For each detected target person, create a mask for its region of interest and multiply it with the original data matrix data to shield against interference from other people, object reflections, and multipath clutter.

[0027] Step 4.4: Finally, combine the feature matrices of multiple frames of data to construct a behavioral feature model, using the original behavioral feature model as the basis. For example:

[0028] A further improvement of this invention is that the MIMO through-wall radar system includes a MIMO through-wall radar and an FS-Convnext network model. The FS-Convnext network is composed of multiple stacked ConvNext Blocks, which undergo convolution operations, downsampling, and normalization layer by layer, and finally perform classification prediction through an attention mechanism and fully connected layers. The channel dimension of each ConvNextBlock gradually increases, while the spatial dimension of the feature map gradually decreases.

[0029] The FS-Convnext network employs a 3D convolutional layer (Conv3D) and a normalization layer (Layer Norm) for initial feature extraction, followed by three downsampling layers and three residual blocks (ConvNext Block) to process the image dimensions to capture more complex behavioral features, a global average pooling layer to compress the spatial dimensions of the feature map, and a fully connected layer (Linear FC) to output classification results.

[0030] The ConvNext Block is the core part of the entire network, responsible for multi-layer convolution and feature processing: such as Figure 3 As shown on the right, the main components include a 7x7x7 depthwise convolutional layer (Conv3D), a layer normalization layer (Layer Norm), a 1x1x1 convolutional layer (Conv3D), a GELU activation function for non-linear transformation of the output, a layer scaling layer, and a drop path layer to control layer weights and perform regularization to prevent overfitting. Finally, a skip connection is used to directly add the input features to the output, preserving feature propagation while preventing gradient vanishing.

[0031] Downsampling is used to reduce the resolution of feature maps and reduce computational cost. The downsampling layer consists of a normalization layer and a convolutional layer with a kernel size of 2x2x2 to achieve downsampling and reduce the spatial dimension of the feature map.

[0032] A further improvement of the present invention is that the training of the FS-Convnext network model specifically includes the following steps:

[0033] Step 5.1: The FS-Convnext network model is trained on the data in the original horizontal feature matrix, the original vertical feature matrix, the Doppler horizontal feature matrix, and the Doppler vertical feature matrix, respectively. The final output feature is:

[0034] feature=X+GELU(LayerNorm(Conv3d 1×1 (Conv 7×7 (X)))

[0035] Where X represents the input feature;

[0036] Step 5.2: Concatenate the trained behavioral feature data according to the weights to form the final feature data. k :

[0037]

[0038] Where α1+α2+α3+α4=1;

[0039] Step 5.3: Use softmax to extract the behavioral features of the target person, and use cross-entropy loss to measure the difference between the predicted results and the actual behavioral labels. The loss is calculated as follows:

[0040]

[0041] Where N represents the total number of samples, t k The label represents the actual behavior sample of the k-th target person. The parameter information is updated, and the training is repeated for multiple rounds.

[0042] A further improvement of the present invention is that, in step 1, the MIMO through-wall radar includes a transceiver with 4 transmitting antennas and 16 receiving antennas. The MIMO through-wall radar transmits frequency-modulated continuous waves with a range of 1-2 GHz, which penetrate obstacles and illuminate the surface of the human body, thereby generating a reflected signal. Assuming P... t It is the transmission power, f c The radar center frequency is given by k, and the slope of the linear frequency modulated pulse is given by T. p Let the sweep period of the linear frequency modulated pulse be denoted as , then the discrete function of the transmitted signal at time t is:

[0043]

[0044] After receiving the reflected signal, the radar obtains the intermediate frequency (IF) signal through a low-noise amplifier and a mixer. The transceiver with 4 transmitting antennas and 16 receiving antennas samples the IF signal as a discrete function with respect to time t, which is:

[0045]

[0046] Where m and n represent the indices of the transmitting and receiving antennas, respectively, A(t) represents the amplitude at time t, λ represents the wavelength of the center frequency, and R... m，n This represents the distance traveled from the m-th transmitting antenna to the target point and then to be received by the n-th receiving antenna.

[0047] A further improvement of the present invention is that step 2 specifically involves: for radio frequency signals, a frequency-domain-based feature extraction method is used to transform the time-domain complex signal to the frequency domain using Fourier transform, observing the status of each frequency component of the signal, and extracting features in the frequency domain; the sampled signals of each transmit / receive pair antenna are grouped into a frame, and a three-dimensional discrete Fourier transform is performed on it to obtain the three-dimensional spatial power matrix P in space.

[0048]

[0049] in, θ and γ represent the pitch angle, azimuth angle, and distance, respectively; k represents the slope; c represents the signal propagation speed; and s represents the speed of signal propagation. m and s n These represent the spacing between the transmitting and receiving antennas, respectively. This matrix not only captures the spatial distribution characteristics of the signal but also lays a solid foundation for further analysis and feature extraction.

[0050] The beneficial effects of this invention are:

[0051] This invention is based on radio frequency signals rather than traditional optical images, which provides important protection for the system's security, especially in applications where privacy needs to be protected.

[0052] This invention designs a branch structure suitable for tracking and segmenting different people in multi-person scenarios. By effectively eliminating interference from dynamic backgrounds, it significantly improves the robustness of the system in recognizing the behavior of different individuals in complex environments.

[0053] This invention constructs a behavior matrix that integrates multiple features by projecting the original and Doppler behavioral features horizontally and vertically. This not only enriches the feature dimensions but also improves the accuracy of identifying the behavior of different people.

[0054] While ensuring recognition performance, this invention significantly reduces computational overhead, optimizes computational efficiency, and broadens the scope of applications by adopting a lightweight network architecture. Attached Figure Description

[0055] Figure 1 This is a flowchart of the multi-person human behavior recognition method of the present invention.

[0056] Figure 2 This is a flowchart of the behavior and action recognition process of multiple feature matrices in this invention.

[0057] Figure 3 This invention relates to behavioral actions in multi-person scenarios.

[0058] Figure 4 These are the four behavioral actions of this invention. Detailed Implementation

[0059] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0060] like Figure 1-3 As shown, this invention is a multi-person human behavior recognition method based on MIMO through-wall radar feature fusion. The method is characterized by being implemented through a MIMO through-wall radar system, which includes a MIMO through-wall radar and an FS-Convnext network model. Figure 3 As shown, the FS-Convnext network consists of multiple stacked ConvNext Blocks, which undergo convolution, downsampling, and normalization operations layer by layer. Finally, classification prediction is performed through an attention mechanism and fully connected layers. The channel dimension of each ConvNext Block gradually increases, while the spatial dimension of the feature map gradually decreases.

[0061] The FS-Convnext network employs a 3D convolutional layer (Conv3D) and a normalization layer (Layer Norm) for initial feature extraction, followed by three downsampling layers and three residual blocks (ConvNext Block) to process the image dimensions to capture more complex behavioral features, a global average pooling layer to compress the spatial dimensions of the feature map, and a fully connected layer (Linear FC) to output classification results.

[0062] The ConvNext Block is the core part of the entire network, responsible for multi-layer convolution and feature processing: such as Figure 3 As shown on the right, the main components include a 7x7x7 depthwise convolutional layer (Conv3D), a layer normalization layer (Layer Norm), a 1x1x1 convolutional layer (Conv3D), a GELU activation function for non-linear transformation of the output, a layer scaling layer, and a drop path layer to control layer weights and perform regularization to prevent overfitting. Finally, a skip connection is used to directly add the input features to the output, preserving feature propagation while preventing gradient vanishing.

[0063] Downsampling is used to reduce the resolution of feature maps and reduce computational cost. The downsampling layer consists of a normalization layer and a convolutional layer with a kernel size of 2x2x2 to achieve downsampling and reduce the spatial dimension of the feature map.

[0064] This invention divides the entire network into several small modules, such as convolutional layers, normalization layers, and activation functions. This not only simplifies the network adjustment and optimization process but also improves the network's flexibility. In particular, the introduction of large-kernel convolutions and deeper network structures greatly enhances the network's ability to capture complex human behavioral and action information, thus ensuring good recognition results in a wide range of application scenarios. Layer normalization and the Gaussian Error Linear Unit (GELU) activation function are employed. Layer normalization effectively improves the stability during training by standardizing the input of each layer, reducing the complexity of model training and its sensitivity to hyperparameter adjustments. The GELU activation function, on the other hand, improves the model's training efficiency and performance while maintaining network nonlinearity.

[0065]

[0066] The multi-person human behavior recognition method of the present invention specifically includes the following steps:

[0067] Step 1: Initialize the MIMO through-wall radar system, configure the information sampling parameters, and acquire the radio frequency signals collected by the MIMO through-wall radar. The information sampling parameters include: power amplifier, low noise amplifier, antenna array, mixer, signal source programmable gate array, etc.

[0068] This invention employs a self-designed MIMO through-wall radar, which includes a transceiver with four transmitting antennas and sixteen receiving antennas. The MIMO through-wall radar transmits frequency-modulated continuous waves with a range of 1-2 GHz, penetrating obstacles and illuminating the human body surface, thereby generating a reflected signal. Assuming P... t It is the transmission power, f c The radar center frequency is given by k, and the slope of the linear frequency modulated pulse is given by T. p Let the sweep period of the linear frequency modulated pulse be denoted as , then the discrete function of the transmitted signal at time t is:

[0069]

[0070] After receiving the reflected signal, the radar obtains the intermediate frequency (IF) signal through a low-noise amplifier and a mixer. The transceiver with 4 transmitting antennas and 16 receiving antennas samples the IF signal as a discrete function with respect to time t, which is:

[0071]

[0072] Where m and n represent the indices of the transmitting and receiving antennas, respectively, A(t) represents the amplitude at time t, λ represents the wavelength of the center frequency, and R... m，n This represents the distance traveled from the m-th transmitting antenna to the target point and then to be received by the n-th receiving antenna.

[0073] Step 2: Preprocess the radio frequency signals acquired in Step 1. Combine the sampling signals of each transmit and receive pair antenna in Step 1 into a frame, perform a three-dimensional discrete Fourier transform on it, and construct a three-dimensional spatial power matrix.

[0074] For radio frequency signals, a frequency-domain-based feature extraction method is used. The complex time-domain signal is transformed to the frequency domain using Fourier transform, allowing observation of the signal's frequency components and feature extraction within the frequency domain. The sampled signals from each transmit / receive pair antenna are grouped into a frame, and a three-dimensional discrete Fourier transform is performed on it to obtain the three-dimensional spatial power matrix P.

[0075]

[0076] in, θ and γ represent the pitch angle, azimuth angle, and distance, respectively; k represents the slope; c represents the signal propagation speed; and s represents the speed of signal propagation. m and s n These represent the spacing between the transmitting and receiving antennas, respectively. This matrix not only captures the spatial distribution characteristics of the signal but also lays a solid foundation for further analysis and feature extraction.

[0077] Step 3: Initialize the three-dimensional spatial power matrix constructed in Step 2 to construct the original feature matrix, perform dynamic background removal on the constructed original feature matrix, and construct the Doppler behavior matrix.

[0078] Background subtraction was employed, calculating the average value of reflected signals from multiple consecutive frames to obtain dynamic background data. Subtracting this average background data from subsequent frames generated a static radar heatmap, effectively separating static elements from dynamic targets. Furthermore, to accurately capture dynamic information about humans in the scene, such as velocity, Doppler static removal was introduced. By calculating the Doppler velocities of spatial points in multiple frames and removing frames with zero velocity, a dynamic radar heatmap was obtained, focusing on moving objects within the scene. Projecting the obtained static and dynamic radar heatmaps horizontally and vertically, respectively, yielded static horizontal / vertical and dynamic horizontal / vertical radar heatmaps. These heatmaps not only demonstrate the distribution of static and dynamic targets in the scene but also subdivide their characteristics in the horizontal and vertical directions, providing a rich source of information for further feature analysis and recognition.

[0079] Specifically, dynamic background cleanup is performed on the original feature matrix to construct the Doppler behavior matrix as follows:

[0080] If the original data matrix is ​​data, and the horizontal and vertical background data are b respectively... h and b v Then we have:

[0081]

[0082]

[0083] Where, raw h The raw horizontal feature matrix is ​​the original horizontal feature matrix. v The original vertical feature matrix, dop h The doppler horizontal characteristic matrix, dop v Let N be the Doppler vertical feature matrix, and N be the total number of samples.

[0084] Step 4: Separate the original feature matrix and the Doppler behavior matrix and obtain the user feature matrix block. Combine the feature matrices of multiple frames of data to construct a behavior feature model.

[0085] This invention constructs a branching structure to achieve target separation, tracking, and behavioral feature extraction, thereby achieving behavior recognition. The branching structure cleverly utilizes reverse engineering techniques to accurately calculate the reflection area of ​​the target in horizontal and vertical radar heatmaps. Furthermore, it employs masking techniques to precisely extract the region of interest (ROI) for each individual target, effectively eliminating interference from other non-target areas in the scene. The branching structure uses a unique method to correlate the dynamic behavior of the target personnel with their specific location in the scene, improving the accuracy and efficiency of recognition. In addition, the application of a feature fusion layer allows for the effective combination of features output from the two branch networks, enhancing not only the accuracy of target personnel behavior recognition but also improving the system's ability to recognize multiple people's behavior in complex scenes.

[0086] Specifically, the steps include the following:

[0087] Step 4.1: Project the 3D behavior of the person captured by the camera onto the radar heat map to determine the reflection area of ​​the k-th person. Let the coordinate set of the k-th target be... Then the projected coordinates of each key point on the horizontal radar heat map Projected coordinates on vertical radar thermal map for:

[0088]

[0089] Among them, s dFor the range resolution of radar detection, s θ For horizontal angular resolution, s γ Vertical angular resolution;

[0090] Step 4.2: Obtain the projected coordinates of each key point on the horizontal radar heatmap. Projected coordinates on vertical radar thermal map Projected coordinates on the horizontal radar thermal map, serving as the boundary of the region of interest. Projected coordinates on vertical radar thermal map The regions of interest in the horizontal and vertical radar heatmaps are as follows:

[0091]

[0092] The captured sensing area constitutes one frame of data;

[0093] Step 4.3: For each detected target person, create a mask for its region of interest and multiply it with the original data matrix data to shield against interference from other people, object reflections, and multipath clutter.

[0094] Step 4.4: Finally, combine the feature matrices of multiple frames of data to construct a behavioral feature model, using the original behavioral feature model as the basis. For example:

[0095]

[0096] Step 5: Input the behavioral feature model obtained in Step 4 into the FS-Convnext network model. Through different convolutional layers and residual blocks in the FS-Convnext network model, feature extraction for different behaviors is achieved, completing the training of the MIMO through-wall radar system. Simultaneous radar and camera data are used, and the behaviors extracted from the images are used as labels for training. Figure 3 As shown, the network estimates personnel behavior based on radar personnel behavior sequences.

[0097] The training of the FS-Convnext network model specifically includes the following steps:

[0098] Step 5.1: The FS-Convnext network model is trained on the data in the original horizontal feature matrix, the original vertical feature matrix, the Doppler horizontal feature matrix, and the Doppler vertical feature matrix, respectively. The final output feature is:

[0099] feature=X+GELU(LayerNorm(Conv3d 1×1 (Conv 7×7 (X)))

[0100] Where X represents the input feature;

[0101] Step 5.2: Concatenate the trained behavioral feature data according to the weights to form the final feature data. k :

[0102]

[0103] Where α1+α2+α3+α4=1;

[0104] Step 5.3: Use softmax to extract the behavioral features of the target person, and use cross-entropy loss to measure the difference between the predicted results and the actual behavioral labels. The loss is calculated as follows:

[0105]

[0106] Where N represents the total number of samples, t k The label represents the actual behavior sample of the k-th target person. The parameter information is updated, and the training is repeated for multiple rounds.

[0107] Step 6: After training, the MIMO through-wall radar system can recognize the actions of untrained individuals without needing to retrain the FS-Convnext network model. After training, the MIMO through-wall radar system demonstrated robustness in accurately recognizing the actions of new users. This signifies that the model has excellent generalization performance, capable of capturing and understanding common characteristics of human movements. It reduces reliance on continuous training and also lowers the maintenance costs of the model in practical applications.

[0108] To verify the present invention, simulation and comparative experiments are provided.

[0109] 1. Dataset

[0110] As shown in Table 4, four types of actions were designed: stand, walk, sit, and lie. In the experiment, the MIMO radar was placed outside the wall, and the target personnel performed actions in the scene inside the wall. Different personnel randomly displayed different behaviors, and the range of movement of the actions was controlled within 5 meters.

[0111] 2. Measurement Indicators

[0112] Accuracy: The proportion of data correctly classified by the model out of the total data; it is the percentage of correctly classified samples N. correct The ratio of the model to the total number of samples N is used to calculate the model's average recognition accuracy by calculating the accuracy of different behaviors, which serves as a metric.

[0113] 3. Comparison Algorithm

[0114] (1)Resnet_50(TP.Chen et al.,"Through-Wall Human Motion RecognitionBased on Transfer Learning and Ensemble Learning,"in IEEE Geoscience andRemote Sensing Letters,vol.19,pp.1-5,2022)

[0115] (2)3D—OPEN(Z.Wang,C.Yao,J.Ren,M.Feng and X.Jiang,"Human ActivityRecognition Using 3D Orthogonally-projected EfficientNet on Radar Time-Range-Doppler Signature,"2022IEEE 2nd International Conference on SoftwareEngineering and Artificial Intelligence(SEAI),Xiamen,China,2022,pp.26-30)

[0116] (3)RegNet(P.Chen,S.Guo,C.Jiang and G.Cui,"Through-Wall Human MotionRecognition Based on Semi-Manual Network,"2021IEEE International Workshop onElectromagnetics:Applications and Student Innovation Competition(iWEM),Guangzhou,China,2021,pp.1-3)

[0117] (4)Inception-v3(J.Wu,L.Wang,L.Wang,J.Guo and G.Wu,"Learning ActorRelation Graphs for Group Activity Recognition,"2019IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR))

[0118] 4. Experimental Results

[0119] As shown in Table 1, the collected data is processed into a multi-feature matrix according to the description of this invention and input into the network for training. After the system is trained, behavior recognition is performed on untrained users. In addition, the dataset is divided into... Figure 3 The behavior recognition method described herein is used to process and compare the results of new user recognition.

[0120] Table 1. Recognition accuracy and average accuracy of new user behavior in scenarios with different numbers of users.

[0121]

[0122] Experimental results show that the accuracy and average behavior accuracy of the present invention in recognizing untrained personnel after the system is completed are higher than other methods, proving the true effectiveness of the present invention. It overcomes the challenges encountered by traditional behavior recognition in handling multi-target and dynamic background situations, and significantly improves the accuracy and robustness of recognition.

[0123] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for recognizing human behavior in multiple people based on MIMO through-wall radar feature fusion, characterized in that: The multi-person human behavior recognition method is implemented through a MIMO through-wall radar system, specifically including the following steps: Step 1: Initialize the MIMO through-wall radar system, configure the information sampling parameters, and acquire the radio frequency signals collected by the MIMO through-wall radar. Step 2: Preprocess the radio frequency signals acquired in Step 1. Combine the sampling signals of each transmit and receive pair antenna in Step 1 into a frame, perform a three-dimensional discrete Fourier transform on it, and construct a three-dimensional spatial power matrix. Step 3: Initialize the three-dimensional spatial power matrix constructed in Step 2 to construct the original feature matrix, perform dynamic background removal on the constructed original feature matrix, and construct the Doppler behavior matrix; Step 4: Separate the original feature matrix and the Doppler behavior matrix and obtain the user feature matrix block. Combine the feature matrices of multiple frames of data to construct a behavior feature model. Step 5: Input the behavioral feature model obtained in Step 4 into the FS-Convnext network model. Through different convolutional layers and residual blocks in the FS-Convnext network model, feature extraction of different behaviors is achieved, and the training of the MIMO through-wall radar system is completed. Step 6: After training, the MIMO through-wall radar system can recognize the actions of untrained personnel without needing to retrain the FS-Convnext network model. In step 3, dynamic background cleanup is performed on the original feature matrix to construct the Doppler behavior matrix, specifically as follows: The original data matrix is ​​data, and the horizontal and vertical background data are b respectively. h and b v Then we have: Among them, dop h The doppler horizontal characteristic matrix, dop v is the Doppler vertical feature matrix, and N is the total number of samples; Step 4 involves separating the original feature matrix and the Doppler behavior matrix and obtaining user feature matrix blocks. The behavior feature model is constructed by combining feature matrices from multiple frames of data. This step specifically includes the following steps: Step 4.1: Project the 3D behavior of the person captured by the camera onto the radar heat map to determine the reflection area of ​​the k-th person. Let the coordinate set of the k-th target be... Then the projected coordinates of each key point on the horizontal radar heat map Projected coordinates on vertical radar thermal map for: Among them, s d For the range resolution of radar detection, s θ For horizontal angular resolution, s γ Vertical angular resolution; Step 4.2: Obtain the projected coordinates of each key point on the horizontal radar heatmap. Projected coordinates on vertical radar thermal map Projected coordinates on the horizontal radar thermal map, serving as the boundary of the region of interest. Projected coordinates on vertical radar thermal map The regions of interest in the horizontal and vertical radar heatmaps are as follows: The captured sensing area constitutes one frame of data; Step 4.3: Finally, combine the feature matrices of multiple frames of data to construct a behavioral feature model. The training of the FS-Convnext network model specifically includes the following steps: Step 5.1: The FS-Convnext network model is trained on the data in the original behavior horizontal feature matrix, the original behavior vertical feature matrix, the Doppler horizontal feature matrix, and the Doppler vertical feature matrix, respectively. The final output feature is: feature=X+GELU(LayerNorm(Conv3d 1×1 (Conv 7×7 (X))) Where X represents the input feature; Step 5.2: Concatenate the trained behavioral feature data according to the weights to form the final feature data. k : Where α1+α2+α3+α4=1; Step 5.3: Use softmax to extract the behavioral features of the target person, and use cross-entropy loss to measure the difference between the predicted results and the actual behavioral labels. The loss is calculated as follows: Where N represents the total number of samples, t k The label represents the actual behavior sample of the k-th target person. The parameter information is updated, and the training is repeated for multiple rounds.

2. The method for multi-person human behavior recognition based on MIMO through-wall radar feature fusion according to claim 1, characterized in that: The MIMO through-wall radar system includes MIMO through-wall radar and an FS-Convnext network model. The FS-Convnext network model includes a 3D convolutional layer, a normalization layer, multiple downsampling layers, multiple residual blocks, and a global average pooling layer. The residual block includes a depthwise convolutional layer, a normalization layer, a convolutional layer, a GELU activation function that performs a non-linear transformation on the output, a scaling layer, and a path dropout layer that controls the layer weights and performs regularization. Finally, the residual blocks are connected to directly add the input features to the output. The downsampling layer is used to reduce the resolution of the feature map and reduce the computational cost. The downsampling layer contains a normalization layer and a convolutional layer to achieve downsampling and reduce the spatial dimension of the feature map.

3. The method for multi-person human behavior recognition based on MIMO through-wall radar feature fusion according to claim 1, characterized in that: In step 1, the MIMO through-wall radar includes a transceiver with 4 transmitting antennas and 16 receiving antennas. The MIMO through-wall radar transmits frequency-modulated continuous waves with a range of 1-2 GHz, which penetrate obstacles and illuminate the surface of the human body, thereby generating a reflected signal. t It is the transmission power, f c The radar center frequency is given by k, and the slope of the linear frequency modulated pulse is given by T. p Let the sweep period of the linear frequency modulated pulse be denoted as , then the discrete function of the transmitted signal at time t is: After receiving the reflected signal, the radar obtains the intermediate frequency (IF) signal through a low-noise amplifier and a mixer. The transceiver with 4 transmitting antennas and 16 receiving antennas samples the IF signal as a discrete function with respect to time t, which is: Where m and n represent the indices of the transmitting and receiving antennas, respectively, A(t) represents the amplitude at time t, λ represents the wavelength of the center frequency, and R... m,n This represents the distance traveled from the m-th transmitting antenna to the target point and then to be received by the n-th receiving antenna.

4. The method for multi-person human behavior recognition based on MIMO through-wall radar feature fusion according to claim 3, characterized in that: Step 2 specifically involves: for radio frequency signals, a frequency-domain-based feature extraction method is used to transform the complex time-domain signal to the frequency domain using Fourier transform, observing the status of each frequency component of the signal, and extracting features in the frequency domain; the sampled signals from each transmit / receive pair antenna are grouped into a frame, and a three-dimensional discrete Fourier transform is performed on it to obtain the three-dimensional spatial power matrix P in space. in, These represent the pitch angle, azimuth angle, and distance, respectively; k represents the slope; c represents the signal propagation speed; and s represents the speed of signal propagation. m and s n These represent the spacing between the transmitting antenna and the receiving antenna, respectively.

Images