Multi-person human body behavior identification method based on MIMO through-the-wall radar feature fusion

Abstract

The invention belongs to the field of MIMO through-the-wall radar and computer vision, and discloses a multi-person human body behavior recognition method based on MIMO through-the-wall radar feature fusion, which comprises the following steps: carrying out preliminary feature extraction on behavior action information, including original features and Doppler features, preprocessing collected radio frequency signals, constructing a three-dimensional space power matrix, and carrying out feature fusion on the three-dimensional space power matrix; constructing a Doppler behavior matrix, combining a multi-frame data feature matrix, constructing a behavior feature model, and inputting the behavior feature model into an FS-Convnext network model to complete training.

Description

Technical Field

[0001] The invention belongs to the field of MIMO through-wall radar and computer vision, and specifically relates to a multi-person human behavior recognition method based on MIMO through-wall radar feature fusion. Background Art

[0002] Human behavior recognition aims to automatically detect, analyze, identify and classify human actions, gestures or behaviors from sensor data, video sequences or other sensing devices, so that the computing system can provide assistance to users. It has a wide range of applications in daily life activity monitoring, video analysis, autonomous driving, etc. The rapid development of artificial intelligence (AI) technology has had a positive impact on human behavior recognition.

[0003] Almost all objects can reflect signals, resulting in RF signals carrying a lot of information that is irrelevant to people or tasks. For multi-position human action recognition systems, most work and research focuses more on the comparison of single-position model performance and fusion position model performance. Multi-person action recognition often requires the integration of information from multiple sensors, such as RGB cameras, depth cameras, inertial sensors, etc. In previous work, some studies did not separate the raw data by position, but connected all positions and input them into the network for training; others focused more on the kernel optimization of convolutional neural networks and multimodal sensor separation.

[0004] However, in scenarios with multiple people through walls, the area corresponding to each person in the radar RF signal may be small and unevenly distributed. Most RF signals are reflected from objects in the environment and may therefore be strongly biased by background noise. It is not immediately clear which part of the reflected signal corresponds to the person in the scene, and useful information about the person cannot be learned. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multi-person human behavior recognition method based on MIMO through-wall radar feature fusion. The method performs preliminary feature extraction on behavior action information, including original features and Doppler features, divides different users in the scene, and superimposes multi-frame information matrices to achieve real-time behavior recognition, which provides the possibility for the behavior recognition system to be better put into practical applications, and is used to improve the recognition accuracy in complex environments such as crowded, noisy and other scenes, and enhance the robustness of the system.

[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0007] The present invention is a multi-person human behavior recognition method based on MIMO through-wall radar feature fusion, which is implemented by a MIMO through-wall radar system and specifically includes the following steps:

[0008] Step 1: Initialize the MIMO through-wall radar system, configure the parameters of information sampling, and obtain the radio frequency signal collected by the MIMO through-wall radar. The parameters of information sampling include: power amplifier, low noise amplifier, antenna array, mixer, signal source programmable gate array, etc.

[0009] Step 2: pre-process the radio frequency signal collected in step 1, form a frame of the sampling signal of each transmitting and receiving antenna in step 1, perform a three-dimensional discrete Fourier transform on it, and construct a three-dimensional spatial power matrix;

[0010] Step 3, initializing the three-dimensional space power matrix constructed in step 2 to construct an original feature matrix, performing dynamic background removal on the constructed original feature matrix, and constructing a Doppler behavior matrix;

[0011] Step 4: Separate the original feature matrix and the Doppler behavior matrix and obtain the user feature matrix block, and combine the feature matrices of multiple frames of data to build a behavior feature model;

[0012] Step 5: Input the behavior feature model obtained in step 4 into the FS-Convnext network model, and extract the features of different behaviors through different convolutional layers and residual blocks in the FS-Convnext network model to complete the training of the MIMO through-wall radar system.

[0013] Step 6: After training, the MIMO through-the-wall radar system can recognize the actions of untrained personnel without the need to retrain using the FS-Convnext network model.

[0014] The further improvement of the present invention is that in step 3, dynamic background removal is performed on the original feature matrix to construct a Doppler behavior matrix, specifically:

[0015] If the original data matrix is ​​data, the horizontal and vertical background data are b respectively. h and b v , then:

[0016]

[0017]

[0018] Among them, raw h is the original horizontal feature matrix, raw v is the original vertical feature matrix, dop h is the Doppler level characteristic matrix, dopv is the Doppler vertical characteristic matrix, and N is the total number of samples.

[0019] A further improvement of the present invention is that: the step 4 separates the original feature matrix and the Doppler behavior matrix and obtains the user feature matrix block, and combines the feature matrices of multiple frames of data to construct a behavior feature model, which specifically includes the following steps:

[0020] Step 4.1: Project the 3D behavior of the current person captured by the camera onto the radar heat map to determine the reflection area of ​​the kth person. Let the coordinate set of the kth target be Then the projection coordinates of each key point on the horizontal radar heat map are and the projection coordinates on the vertical radar heat map for:

[0021]

[0022] Among them, s d is the range resolution of radar detection, s θ is the horizontal angle resolution, s γ is the vertical angle resolution;

[0023] Step 4.2: Take the projection coordinates of each key point on the horizontal radar heat map and projection coordinates on vertical radar heatmap Projected coordinates on the horizontal radar heat map as the area of ​​interest boundary and the projection coordinates on the vertical radar heat map The areas of interest in the horizontal radar heat map and the vertical radar heat map are:

[0024]

[0025] The intercepted sensing area is a frame of data;

[0026] Step 4.3: For each detected target person, make its region of interest into a mask and multiply it with the original data matrix data to shield the interference of other people, object reflections and multipath clutter;

[0027] Step 4.4: Finally, combine the multi-frame data feature matrix to construct a behavior feature model. For example:

[0028] A further improvement of the present invention is that the MIMO through-the-wall radar system includes a MINO through-the-wall radar and a FS-Convnext network model, and the FS-Convnext network is composed of a plurality of ConvNext Block stacks, which perform convolution operations, downsampling and normalization processing layer by layer, and finally perform classification prediction through an attention mechanism and a fully connected layer. The channel dimension of each ConvNextBlock gradually increases, and the spatial dimension of the feature map gradually decreases.

[0029] FS-Convnext network: It uses a 3D convolution layer (Conv3D) and a normalization layer (Layer Norm) to initialize feature extraction, followed by three downsampling layers (Downsample) and three residual blocks (ConvNext Block) to process the image dimensions to capture more complex behavioral features, a global average pooling layer (GlobalAverage Pooling) to compress the spatial dimensions of the feature map, and a fully connected layer (Linear FC) to classify the output through a linear fully connected layer.

[0030] ConvNext Block is the core part of the entire network, responsible for multi-layer convolution and feature processing: Figure 3 As shown on the right, it mainly includes a depthwise Conv3D with a convolution kernel size of 7x7x7, a normalization layer (Layer Norm), a convolution layer (Conv3D) with a convolution kernel size of 1x1x1, a GELU activation function to perform nonlinear transformation on the output, a scaling layer (Layer Scale) and a path drop layer (Drop Path) to control the layer weights and perform regularization to prevent overfitting. Finally, a residual connection (skip connection) is used to add the input features directly to the output, retaining feature transfer while preventing gradient disappearance.

[0031] Downsample is used to reduce the resolution of the feature map and reduce the amount of calculation. Downsampling layer: It contains a normalization layer and a convolution layer with a convolution kernel size of 2x2x2 to achieve downsampling and reduce the spatial dimension of the feature map.

[0032] A further improvement of the present invention is that the FS-Convnext network model training specifically includes the following steps:

[0033] Step 5.1, the FS-Convnext network model trains the data in the original horizontal feature matrix, the original vertical feature matrix, the Doppler horizontal feature matrix, and the Doppler vertical feature matrix respectively, and the final output feature is:

[0034] feature=X+GELU(LayerNorm(Conv3d 1×1 (Conv 7×7 (X)))

[0035] Among them, X is the input feature;

[0036] Step 5.2: Combine the weights to form the final trained behavior feature data k :

[0037]

[0038] Among them, α1+α2+α3+α4=1;

[0039] Step 5.3: Use softmax to extract the behavioral features of the target person, and use cross entropy loss to measure the difference between the predicted result and the true behavior label. The loss is calculated as follows:

[0040]

[0041] Where N is the total number of samples, t k Represent the real behavior sample label of the k-th target person, update the parameter information, and train for multiple rounds.

[0042] A further improvement of the present invention is that in step 1, the MIMO through-the-wall radar includes a transceiver having 4 transmitting antennas and 16 receiving antennas, and the MIMO through-the-wall radar transmits a frequency modulated continuous wave with a range of 1-2 GHz, penetrates obstacles and irradiates the surface of the human body, thereby generating a reflected signal. Assume that P t is the transmit power, f c is the radar center frequency, k is the slope of the linear frequency modulation pulse, T p represents the sweep period of the linear frequency modulation pulse, then the discrete function of the transmitted signal at time t is:

[0043]

[0044] After receiving the reflected signal, the radar obtains the intermediate frequency signal through the low noise amplifier and mixer. The discrete function of the sampling of the intermediate frequency signal by the transceiver with 4 transmitting antennas and 16 receiving antennas with respect to time t is:

[0045]

[0046] Where m and n represent the indexes of the transmitting antenna and the receiving antenna respectively, A(t) represents the amplitude at time t, λ represents the wavelength of the center frequency, and R m，n It represents the distance from the mth transmitting antenna through the target point and received by the nth receiving antenna.

[0047] The further improvement of the present invention is that: the step 2 is specifically: for the radio frequency signal, a feature extraction method based on the frequency domain is used to transform the complex signal in the time domain into the frequency domain through the Fourier transform method, the status of each frequency component of the signal is observed, and the feature is extracted in the frequency domain; the sampling signal of each transmitting and receiving antenna is composed into a frame, and a three-dimensional discrete Fourier transform is performed on it to obtain a three-dimensional space power matrix P in the space:

[0048]

[0049] in, θ, γ represent the pitch angle, azimuth angle and distance respectively, k represents the slope, c represents the signal propagation speed, s m and n Represent the intervals between the transmitting antenna and the receiving antenna respectively. This matrix not only captures the spatial distribution characteristics of the signal, but also lays a solid foundation for further analysis and feature extraction.

[0050] The beneficial effects of the present invention are:

[0051] The present invention is based on radio frequency signals rather than traditional optical images, which provides important protection for the security of the system, especially in applications where privacy protection is required.

[0052] The present invention designs a branch structure suitable for tracking and segmenting different persons in multi-person scenes, which significantly improves the robustness of the system for behavior recognition of different individuals in complex environments by effectively eliminating the interference of dynamic background.

[0053] The present invention constructs a behavior matrix integrating multiple features by horizontally and vertically projecting the original and Doppler behavior features, which not only enriches the feature dimensions but also improves the accuracy of identifying different personnel behaviors.

[0054] While ensuring recognition performance, the present invention significantly reduces computing overhead by adopting a lightweight network architecture, optimizes computing efficiency, and broadens the scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of the method for identifying multi-person human behavior of the present invention.

[0056] Figure 2 It is a flow chart of multiple feature matrix behavior action recognition of the present invention.

[0057] Figure 3 It is a behavioral action in the multi-person scenario of the present invention.

[0058] Figure 4 These are four behavioral actions of the present invention. DETAILED DESCRIPTION

[0059] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0060] like Figure 1-3 As shown, the present invention is a multi-person human behavior recognition method based on MIMO through-wall radar feature fusion, characterized in that: the multi-person human behavior recognition method is implemented by a MIMO through-wall radar system, and the MIMO through-wall radar system includes a MINO through-wall radar and a FS-Convnext network model. Figure 3 As shown in the figure, the FS-Convnext network consists of multiple ConvNext Block stacks, which perform convolution operations, downsampling and normalization layer by layer, and finally perform classification prediction through the attention mechanism and the fully connected layer. The channel dimension of each ConvNext Block gradually increases, and the spatial dimension of the feature map gradually decreases.

[0061] FS-Convnext network: It uses a 3D convolution layer (Conv3D) and a normalization layer (Layer Norm) to initialize feature extraction, followed by three downsampling layers (Downsample) and three residual blocks (ConvNext Block) to process the image dimensions to capture more complex behavioral features, a global average pooling layer (GlobalAverage Pooling) to compress the spatial dimensions of the feature map, and a fully connected layer (Linear FC) to classify the output through a linear fully connected layer.

[0062] ConvNext Block is the core part of the entire network, responsible for multi-layer convolution and feature processing: Figure 3 As shown on the right, it mainly includes a depthwise Conv3D with a convolution kernel size of 7x7x7, a normalization layer (Layer Norm), a convolution layer (Conv3D) with a convolution kernel size of 1x1x1, a GELU activation function to perform nonlinear transformation on the output, a scaling layer (Layer Scale) and a path drop layer (Drop Path) to control the layer weights and perform regularization to prevent overfitting. Finally, a residual connection (skip connection) is used to add the input features directly to the output, retaining feature transfer while preventing gradient disappearance.

[0063] Downsample is used to reduce the resolution of the feature map and reduce the amount of calculation. Downsampling layer: It contains a normalization layer and a convolution layer with a convolution kernel size of 2x2x2 to achieve downsampling and reduce the spatial dimension of the feature map.

[0064] The present invention divides the entire network into several small modules, such as convolutional layers, normalization layers, activation functions, etc., which not only simplifies the adjustment and optimization process of the network, but also improves the flexibility of the network. In particular, the introduction of large-core-size convolutions and deeper network structures greatly enhances the ability of the network to capture complex human behavior action information, thereby ensuring that good recognition effects can be obtained in a wide range of application scenarios. Layer normalization and Gaussian error linear unit (GELU) activation functions are adopted. The application of layer normalization can effectively improve the stability during the training process, and by standardizing the input of each layer, the complexity of model training and the sensitivity to hyperparameter adjustment are reduced. The GELU activation function can improve the training efficiency and performance of the model while maintaining the nonlinearity of the network.

[0065]

[0066] The multi-person human behavior recognition method of the present invention specifically comprises the following steps:

[0067] Step 1: Initialize the MIMO through-wall radar system, configure the parameters of information sampling, and obtain the RF signal collected by the MIMO through-wall radar. The parameters of information sampling include: power amplifier, low-noise amplifier, antenna array, mixer, signal source programmable gate array, etc.

[0068] The present invention adopts a self-designed MIMO through-wall radar, which includes a transceiver with 4 transmitting antennas and 16 receiving antennas. The MIMO through-wall radar transmits a frequency modulated continuous wave with a range of 1-2 GHz, penetrates obstacles and irradiates the surface of the human body, thereby generating a reflected signal. Assuming P t is the transmit power, f c is the radar center frequency, k is the slope of the linear frequency modulation pulse, T p represents the sweep period of the linear frequency modulation pulse, then the discrete function of the transmitted signal at time t is:

[0069]

[0070] After receiving the reflected signal, the radar obtains the intermediate frequency signal through the low noise amplifier and mixer. The discrete function of the sampling of the intermediate frequency signal by the transceiver with 4 transmitting antennas and 16 receiving antennas with respect to time t is:

[0071]

[0072] Where m and n represent the indexes of the transmitting antenna and the receiving antenna respectively, A(t) represents the amplitude at time t, λ represents the wavelength of the center frequency, and R m，n It represents the distance from the mth transmitting antenna through the target point and received by the nth receiving antenna.

[0073] Step 2: pre-process the radio frequency signal collected in step 1, form a frame of the sampling signal of each transmitting and receiving antenna in step 1, perform a three-dimensional discrete Fourier transform on it, and construct a three-dimensional spatial power matrix.

[0074] For RF signals, a feature extraction method based on frequency domain is used. The complex time domain signal is transformed into the frequency domain through the Fourier transform method, the status of each frequency component of the signal is observed, and the features are extracted in the frequency domain; the sampling signal of each transmitting and receiving antenna is composed into a frame, and a three-dimensional discrete Fourier transform is performed on it to obtain the three-dimensional space power matrix P in space:

[0075]

[0076] in, θ, γ represent the pitch angle, azimuth angle and distance respectively, k represents the slope, c represents the signal propagation speed, s m and n Represent the spacing between the transmitting antenna and the receiving antenna respectively. This matrix not only captures the spatial distribution characteristics of the signal, but also lays a solid foundation for further analysis and feature extraction.

[0077] Step 3: Initialize the three-dimensional space power matrix constructed in step 2 to construct an original feature matrix, perform dynamic background removal on the constructed original feature matrix, and construct a Doppler behavior matrix.

[0078] Background subtraction technology is adopted to calculate the average value of the reflected signals of multiple consecutive frames of scenes to obtain dynamic background data. The static heat map of the radar is generated by subtracting the subsequent frame data from this average background data. This step effectively separates the static part of the scene from the dynamic targets. In addition, in order to accurately capture the dynamic information of the human body in the scene, such as speed, the Doppler static removal technology is introduced. By calculating the Doppler velocity of spatial points in multiple frames of data and eliminating frames with zero velocity, the dynamic heat map of the radar can be obtained. This process focuses on the moving objects in the scene. The radar static and dynamic heat maps obtained above are projected in the horizontal and vertical directions respectively to obtain the radar static horizontal / vertical heat map and radar dynamic horizontal / vertical heat map. These heat maps not only show the distribution of static and dynamic targets in the scene, but also subdivide their features in the horizontal and vertical directions, providing a rich source of information for further feature analysis and recognition.

[0079] Specifically, the original feature matrix is ​​dynamically cleaned up to construct the Doppler behavior matrix:

[0080] If the original data matrix is ​​data, the horizontal and vertical background data are b respectively. h and b v , then:

[0081]

[0082]

[0083] Among them, raw h is the original horizontal feature matrix, raw v is the original vertical feature matrix, dop h is the Doppler level characteristic matrix, dop v is the Doppler vertical characteristic matrix, and N is the total number of samples.

[0084] Step 4: Separate the original feature matrix and the Doppler behavior matrix and obtain the user feature matrix block, and combine the feature matrices of multiple frames of data to build a behavior feature model.

[0085] The present invention constructs a branch structure to achieve the separation, tracking and behavioral feature extraction of target personnel, thereby achieving the purpose of behavior recognition. The branch structure cleverly uses reverse engineering technology to accurately calculate the reflection area of ​​the target in the horizontal and vertical radar heat map, and further uses mask technology to accurately extract the region of interest (ROI) of each target individual, effectively eliminating interference from other non-target areas in the scene. The branch structure associates the dynamic behavior of the target person with his specific position in the scene through a unique method, thereby improving the accuracy and efficiency of recognition. In addition, the application of the feature fusion layer enables the features output by the two branch networks to be effectively combined, which not only enhances the accuracy of target person behavior recognition, but also improves the system's ability to recognize multiple people's behavior in complex scenes.

[0086] The specific steps include:

[0087] Step 4.1: Project the 3D behavior of the current person captured by the camera onto the radar heat map to determine the reflection area of ​​the kth person. Let the coordinate set of the kth target be Then the projection coordinates of each key point on the horizontal radar heat map are and the projection coordinates on the vertical radar heat map for:

[0088]

[0089] Among them, s dis the range resolution of radar detection, s θ is the horizontal angle resolution, s γ is the vertical angle resolution;

[0090] Step 4.2: Take the projection coordinates of each key point on the horizontal radar heat map and the projection coordinates on the vertical radar heat map Projected coordinates on the horizontal radar heat map as the area of ​​interest boundary and the projection coordinates on the vertical radar heat map The areas of interest in the horizontal radar heat map and the vertical radar heat map are:

[0091]

[0092] The intercepted sensing area is a frame of data;

[0093] Step 4.3: For each detected target person, make its region of interest into a mask and multiply it with the original data matrix data to shield the interference of other people, object reflections and multipath clutter;

[0094] Step 4.4: Finally, combine the multi-frame data feature matrix to construct a behavior feature model. For example:

[0095]

[0096] Step 5: Input the behavior feature model obtained in step 4 into the FS-Convnext network model. Through different convolutional layers and residual blocks in the FS-Convnext network model, the features of different behaviors are extracted to complete the MIMO through-wall radar system training. Synchronize radar and camera data and use the behaviors extracted from the image as labels for training. Figure 3 As shown, the personnel behavior predicted by the radar personnel behavior sequence estimation network.

[0097] The FS-Convnext network model training specifically includes the following steps:

[0098] Step 5.1, the FS-Convnext network model trains the data in the original horizontal feature matrix, the original vertical feature matrix, the Doppler horizontal feature matrix, and the Doppler vertical feature matrix respectively, and the final output feature is:

[0099] feature=X+GELU(LayerNorm(Conv3d 1×1 (Conv 7×7 (X)))

[0100] Among them, X is the input feature;

[0101] Step 5.2: Combine the weights to form the final trained behavior feature data k :

[0102]

[0103] Among them, α1+α2+α3+α4=1;

[0104] Step 5.3: Use softmax to extract the behavioral features of the target person, and use cross entropy loss to measure the difference between the predicted result and the true behavior label. The loss is calculated as follows:

[0105]

[0106] Where N is the total number of samples, t k Represent the real behavior sample label of the k-th target person, update the parameter information, and train for multiple rounds.

[0107] Step 6: After training, the MIMO through-the-wall radar system can recognize the actions of untrained people without the need to retrain the FS-Convnext network model. After training, the MIMO through-the-wall radar system demonstrated its robustness in accurately recognizing the actions of new users. This indicates that the model has excellent generalization performance and can capture and understand the common characteristics of human actions. This reduces the reliance on continuous training and also reduces the maintenance cost of the model in practical applications.

[0108] In order to verify the present invention, simulation and comparative experiments of the present invention are provided.

[0109] 1. Dataset

[0110] As shown in Table 4, four kinds of behavioral actions are designed, namely stand, walk, sit and lie. In the experiment, the MIMO radar is placed outside the wall, and the target person performs actions in the scene inside the wall. Different people randomly display different behaviors, and the movement range of the behavioral actions is controlled within 5 meters.

[0111] 2. Metrics

[0112] Recognition accuracy: The proportion of data that the model correctly judges to the total data, which is the number of samples N that are correctly classified correct The ratio to the total number of samples N is used to calculate the average recognition accuracy of the model by calculating the accuracy of different behaviors as a measurement indicator.

[0113] 3. Comparison of algorithms

[0114] (1)Resnet_50(TP.Chen et al.,"Through-Wall Human Motion RecognitionBased on Transfer Learning and Ensemble Learning,"in IEEE Geoscience andRemote Sensing Letters,vol.19,pp.1-5,2022)

[0115] (2)3D—OPEN(Z.Wang,C.Yao,J.Ren,M.Feng and X.Jiang,"Human ActivityRecognition Using 3D Orthogonally-projected EfficientNet on Radar Time-Range-Doppler Signature,"2022IEEE 2nd International Conference on SoftwareEngineering and Artificial Intelligence(SEAI),Xiamen,China,2022,pp.26-30)

[0116] (3)RegNet(P.Chen,S.Guo,C.Jiang and G.Cui,"Through-Wall Human MotionRecognition Based on Semi-Manual Network,"2021IEEE International Workshop onElectromagnetics:Applications and Student Innovation Competition(iWEM),Guangzhou,China,2021,pp.1-3)

[0117] (4)Inception-v3(J.Wu,L.Wang,L.Wang,J.Guo and G.Wu,"Learning ActorRelation Graphs for Group Activity Recognition,"2019IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR))

[0118] 4. Experimental results

[0119] As shown in Table 1, the collected data is processed into a multi-feature matrix according to the description of the present invention and input into the network for training. After the system training is completed, the behavior recognition of untrained users is performed. In addition, the data set is processed according to Figure 3 The behavior recognition method described in is used to compare the recognition results of new users.

[0120] Table 1 Recognition accuracy and average accuracy of new users’ behaviors in different number of people scenarios

[0121]

[0122] The experimental results show that after the system is completed, the behavior recognition accuracy and average behavior accuracy of the present invention for untrained personnel are higher than those of other methods, which proves the authenticity and effectiveness of the present invention, overcomes the challenges encountered by traditional behavior recognition when dealing with multiple targets, dynamic backgrounds, etc., and significantly improves the recognition accuracy and robustness.

[0123] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A multi-person human behavior recognition method based on MIMO through-wall radar feature fusion, characterized by: The multi-person human behavior recognition method is implemented by a MIMO through-wall radar system, and specifically comprises the following steps: Step 1: Initialize the MIMO through-wall radar system, configure the parameters of information sampling, and obtain the RF signal collected by the MIMO through-wall radar; Step 2: pre-process the radio frequency signal collected in step 1, form a frame of the sampling signal of each transmitting and receiving antenna in step 1, perform a three-dimensional discrete Fourier transform on it, and construct a three-dimensional spatial power matrix; Step 3, initializing the three-dimensional space power matrix constructed in step 2 to construct an original feature matrix, performing dynamic background removal on the constructed original feature matrix, and constructing a Doppler behavior matrix; Step 4: Separate the original feature matrix and the Doppler behavior matrix and obtain the user feature matrix block, and combine the feature matrices of multiple frames of data to build a behavior feature model; Step 5: Input the behavior feature model obtained in step 4 into the FS-Convnext network model, and extract the features of different behaviors through different convolutional layers and residual blocks in the FS-Convnext network model to complete the training of the MIMO through-wall radar system. Step 6: After training, the MIMO through-the-wall radar system can recognize the actions of untrained personnel without the need to retrain using the FS-Convnext network model.

2. The multi-person human behavior recognition method based on MIMO through-wall radar feature fusion according to claim 1 is characterized in that: In step 3, the original feature matrix is ​​dynamically cleaned to construct the Doppler behavior matrix, specifically: If the original data matrix is ​​data, the horizontal and vertical background data are b respectively. h and b v , then: Among them, raw h is the original horizontal feature matrix, raw v is the original vertical feature matrix, dop h is the Doppler level characteristic matrix, dop v is the Doppler vertical characteristic matrix, and N is the total number of samples.

3. The multi-person human behavior recognition method based on MIMO through-wall radar feature fusion according to claim 1 is characterized in that: The step 4 separates the original feature matrix and the Doppler behavior matrix and obtains the user feature matrix block, and combines the feature matrices of multiple frames of data to build a behavior feature model, which specifically includes the following steps: Step 4.1: Project the 3D behavior of the current person captured by the camera onto the radar heat map to determine the reflection area of ​​the kth person. Let the coordinate set of the kth target be Then the projection coordinates of each key point on the horizontal radar heat map are and the projection coordinates on the vertical radar heat map for: Among them, s d is the range resolution of radar detection, s θ is the horizontal angle resolution, s γ is the vertical angle resolution; Step 4.2: Take the projection coordinates of each key point on the horizontal radar heat map and the projection coordinates on the vertical radar heat map Projected coordinates on the horizontal radar heat map as the area of ​​interest boundary and the projection coordinates on the vertical radar heat map The areas of interest in the horizontal radar heat map and the vertical radar heat map are: The intercepted sensing area is a frame of data; Step 4.3, for each detected target person, make its region of interest into a mask and multiply it with the original data matrix data, object reflection and multipath clutter interference; Step 4.4: Finally, combine the multi-frame data feature matrix to construct a behavior feature model. For example:

4. The multi-person human behavior recognition method based on MIMO through-wall radar feature fusion according to claim 1 is characterized in that: The MIMO through-wall radar system includes a MIMO through-wall radar and a FS-Convnext network model. The FS-Convnext network model includes a 3D convolution layer, a normalization layer, multiple downsampling layers, multiple residual blocks, a global average pooling layer, and the residual block includes a deep convolution layer, a normalization layer, a convolution layer, a GELU activation function for performing nonlinear transformation on the output, a scaling layer, a path discarding layer for controlling the layer weights and performing regularization. Finally, the residual block is used to connect and the input features are directly added to the output. The downsampling layer is used to reduce the resolution of the feature map and reduce the amount of calculation. The downsampling layer includes a normalization layer and a convolution layer to achieve downsampling and reduce the spatial dimension of the feature map.

5. The multi-person human behavior recognition method based on MIMO through-wall radar feature fusion according to claim 1 is characterized in that: The FS-Convnext network model training specifically includes the following steps: Step 5.1, the FS-Convnext network model trains the data in the original horizontal feature matrix, the original vertical feature matrix, the Doppler horizontal feature matrix, and the Doppler vertical feature matrix respectively, and the final output feature is: feature=X+GELU(LayerNorm(Conv3d 1×1 (Conv 7×7 (X))) Among them, X is the input feature; Step 5.2: Combine the weights to form the final trained behavior feature data k : Among them, α1+α2+α3+α4=1; Step 5.3: Use softmax to extract the behavioral features of the target person, and use cross entropy loss to measure the difference between the predicted result and the true behavior label. The loss is calculated as follows: Where N is the total number of samples, t k Represent the real behavior sample label of the k-th target person, update the parameter information, and train for multiple rounds.

6. The multi-person human behavior recognition method based on MIMO through-wall radar feature fusion according to any one of claims 1 to 5, characterized in that: In step 1, the MIMO through-the-wall radar includes a transceiver with 4 transmitting antennas and 16 receiving antennas. The MIMO through-the-wall radar transmits a frequency modulated continuous wave with a range of 1-2 GHz, penetrates obstacles and irradiates the surface of the human body, thereby generating a reflected signal. Assume that P t is the transmit power, f c is the radar center frequency, k is the slope of the linear frequency modulation pulse, T p represents the sweep period of the linear frequency modulation pulse, then the discrete function of the transmitted signal at time t is: After receiving the reflected signal, the radar obtains the intermediate frequency signal through the low noise amplifier and mixer. The discrete function of the sampling of the intermediate frequency signal by the transceiver with 4 transmitting antennas and 16 receiving antennas with respect to time t is: Where m and n represent the indexes of the transmitting antenna and the receiving antenna respectively, A(t) represents the amplitude at time t, λ represents the wavelength of the center frequency, and R m，n It represents the distance from the mth transmitting antenna through the target point and received by the nth receiving antenna.

7. The multi-person human behavior recognition method based on MIMO through-wall radar feature fusion according to claim 6 is characterized by: The step 2 is specifically as follows: for the RF signal, a feature extraction method based on the frequency domain is used to transform the complex time domain signal into the frequency domain through the Fourier transform method, the status of each frequency component of the signal is observed, and the feature is extracted in the frequency domain; the sampling signal of each transmitting and receiving antenna is composed into a frame, and a three-dimensional discrete Fourier transform is performed on it to obtain a three-dimensional space power matrix P in the space: in, They represent the pitch angle, azimuth angle and distance respectively, k represents the slope, c represents the signal propagation speed, s m and n Represent the spacing between the transmitting antenna and the receiving antenna respectively.

Images