A multi-person human body behavior recognition method based on MIMO through-wall radar cross-modal multi-feature fusion

Abstract

The application belongs to the field of MIMO through-wall radar and deep learning, and discloses a multi-person human body behavior recognition method based on MIMO through-wall radar cross-modal multi-feature fusion, which comprises the following steps: initializing and configuring a MIMO through-wall radar system, generating a multi-channel radar heat map sequence containing distance-Doppler information and distance-angle information, constructing a dual-view radar feature extraction and fusion network, and outputting radar feature representation; constructing a Transformer network containing an encoder and a decoder, modeling the radar feature representation, introducing a learnable query vector to obtain an output result, constructing an overall matching relationship and minimizing an overall matching cost function, and realizing optimal alignment; constructing a skeleton point teacher network and a radar student network, and completing model training. The application realizes accurate recognition of multi-person human body behaviors behind a wall, and provides a feasible scheme for deployment of a radar behavior recognition system in actual application.

Description

Technical Field

[0001] This invention belongs to the fields of MIMO through-wall radar and deep learning, specifically relating to a method for multi-person human behavior recognition based on cross-modal multi-feature fusion of MIMO through-wall radar. Background Technology

[0002] Human behavior recognition refers to the automatic analysis and classification of human movements, postures, and behavioral states using sensor data, video sequences, or other sensing methods, thereby providing computing systems with the ability to understand human activities. This technology has broad application prospects in areas such as daily activity monitoring, intelligent security, video content analysis, and autonomous driving. In recent years, the rapid development of artificial intelligence technology, especially the introduction of deep learning methods, has significantly improved the feature modeling and classification performance of human behavior recognition.

[0003] However, radar-based behavior recognition still faces many challenges in multi-person scenarios with wall penetration. The proportion of human targets in radar echo signals is usually small, and the uneven spatial distribution of different people makes human-related signals easily obscured by environmental reflections and background noise. Since radar signals are reflected by almost all objects, the echoes often contain a large amount of information unrelated to human behavior, making it difficult to accurately determine the effective feature regions corresponding to the human body, thus limiting the model's ability to learn human behavior features. For multi-location or multi-person human behavior recognition, existing research mainly focuses on comparing the performance of single-location and multi-location models, or relies on the fusion of multimodal sensor information, such as vision, depth, or inertial sensors. In addition, some methods train by directly stitching together multi-location data, or focus on optimizing convolutional network structures, but these still have certain limitations in complex multi-person scenarios with wall penetration. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion. This method first extracts spatiotemporal features from millimeter-wave radar heatmaps from both horizontal and vertical perspectives. Then, it introduces a multi-scale interaction mechanism within each perspective to enhance the collaborative expression capability between shallow detail information and deep semantic information. Furthermore, it adaptively weights and fuses the dual-perspective features at corresponding scales through a cross-perspective dynamic fusion mechanism, and combines this with a multi-scale aggregation strategy to form a unified radar feature representation. Finally, it utilizes a query-based decoding network to distinguish and model multiple human targets in the scene, and combines prior knowledge of human structure to enhance the semantic expression capability of behavior, thereby achieving accurate recognition of multi-person human behavior behind a wall. This provides a feasible solution for the deployment of radar behavior recognition systems in practical applications.

[0005] To achieve the above objectives, this application employs the following technical solution:

[0006] This application presents a method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion, comprising the following steps:

[0007] Step 1: MIMO Radar Data Acquisition and Preprocessing: Initialize and configure the MIMO through-wall radar system, acquire human motion echo signals, and perform multidimensional Fourier transform preprocessing on the acquired human motion echo signals to generate a multi-channel radar heatmap sequence containing range-Doppler information and range-angle information; the multi-channel radar heatmap sequence includes four independent heatmaps: horizontal range-Doppler heatmap, vertical range-Doppler heatmap, horizontal range-angle heatmap, and vertical range-angle heatmap;

[0008] Step 2: Construct a dual-view radar feature extraction and fusion network: The horizontal range-Doppler heatmap and horizontal range-angle heatmap from the horizontal view are concatenated along the channel dimension to form the input horizontal radar heatmap for the horizontal processing branch. The vertical range-Doppler heatmap and vertical range-angle heatmap from the vertical view are concatenated to form the vertical radar heatmap. Both the horizontal and vertical radar heatmaps are input into the dual-view radar feature extraction and fusion network for training. This yields multi-scale features from both the horizontal and vertical views, and performs multi-scale interaction, dynamic fusion, and aggregation to output a compact radar feature representation. ;

[0009] Step 3: Transformer-based feature decoding and querying: Construct a Transformer network containing an encoder and a decoder to process the radar feature representation output from Step 2. To perform modeling, learnable query vectors are introduced. Automatically search for feature regions related to human targets in horizontal and vertical radar heatmaps, and use an attention mechanism to guide learnable query vectors to adaptively aggregate radar feature representations. After being processed by the decoder, each learnable query vector is obtained. The one-to-one corresponding output results, the predicted output results constitute a set of predicted instances, including human behavior categories and human instance features, and serve as the basis for constructing the subsequent matching cost function;

[0010] Step 4: By constructing an overall matching relationship between the predicted instance set and the real target set provided by the training samples, and minimizing the overall matching cost function, the optimal alignment between the instances in the predicted instance set and the target personnel in the real target set is achieved, thereby eliminating the impact of prediction order uncertainty on the recognition results.

[0011] Step 5: After obtaining the optimal alignment relationship between the predicted instance set and the real target set in Step 4, a cross-modal structural prior distillation strategy is introduced to construct a skeleton point teacher network and a radar student network. For successfully matched instance pairs, the target-level radar instance features output by the decoder in Step 3 are constrained to align to the human skeleton semantic space through the structural consistency loss function to complete the model training.

[0012] Step 6: In the inference phase, remove the skeleton point teacher network and only use the trained radar student network. Input the real-time collected radar data into the radar student network to perform real-time behavior classification of multiple targets.

[0013] A further improvement of this application is that, in step 2, the dual-view radar feature extraction and fusion network includes a dual-branch 3D convolutional backbone network, a multi-scale interaction module within a single view, a cross-view dynamic fusion module, and a multi-scale aggregation module.

[0014] The horizontal processing branch receives horizontal radar heatmaps from a horizontal perspective, and the vertical processing branch receives vertical radar heatmaps from a vertical perspective. The horizontal and vertical radar heatmaps are then concatenated along the channel dimension. After concatenation, the horizontal range-Doppler heatmap and the horizontal range-angle heatmap from the horizontal perspective are concatenated along the channel dimension to form the input horizontal radar heatmap for the horizontal processing branch. The vertical range-Doppler heatmap and the vertical range-angle heatmap from the vertical perspective are then concatenated to form a vertical radar heatmap sequence. The horizontal and vertical radar heatmap sequences are then input into a dual-view radar feature extraction and fusion network for training, and spatiotemporal feature encoding is performed to extract shallow feature maps and deep semantic feature maps.

[0015] After the 3D convolutional backbone network outputs shallow feature maps and deep semantic feature maps, the multi-scale interaction module in a single view performs unified mapping and interactive enhancement on the shallow feature maps and deep semantic feature maps, forming enhanced multi-scale features in the horizontal view and multi-scale features in the vertical view.

[0016] At the corresponding scale, the multi-scale features of the horizontal view and the multi-scale features of the vertical view are input into the cross-view dynamic fusion module. The multi-scale features of the horizontal view and the multi-scale features of the vertical view are weighted and fused to obtain the fused cross-view fusion features.

[0017] A multi-scale aggregation module is used to unify and integrate the fused cross-view features, outputting a compact radar feature representation. This provides input for subsequent human target decoding and behavior classification based on Transformer.

[0018] A further improvement of this application is that, in step 3, the Transformer network includes an encoder and a decoder, and the encoder utilizes a multi-head self-attention mechanism (MSA) to represent radar features. Sequence modeling is performed, and the decoder uses a cross-attention mechanism to learn the query vector and radar feature representation. The interaction and attention calculation formula is as follows:

[0019]

[0020] in, This is the query vector in the decoder. The radar feature key-value pairs output by the encoder. This is the scaling factor.

[0021] A further improvement in this application is that step 3 specifically includes the following steps:

[0022] Step 3.1, Position Encoding: Introducing Sine Positional Encoding for radar feature representation. Given the coordinates (x, y, t), generate a position vector. And superimposed on radar feature representation Above, to preserve spatial structural information;

[0023] Step 3.2, Encoder Processing: The encoder consists of L encoder layers. Each encoder layer comprises a multi-head self-attention mechanism (MSA) and a feedforward neural network (FFN). The MSA is calculated as follows:

[0024]

[0025]

[0026] in, The projection matrix is ​​learnable;

[0027] To adapt to the input format of the Transformer encoder, the radar feature representation is... Expand into a feature sequence and introduce the corresponding position vector. The initial input to the encoder is obtained by adding elements one by one: For the first The output of a layer encoder can be expressed as:

[0028]

[0029]

[0030] Finally, after After layer encoder, the encoder output radar feature is obtained. ;

[0031] Step 3.3, Decoder Query: Initialization Learnable query vectors Learnable query vectors The decoder first performs self-attention interaction, and then interacts with the radar features output by the encoder through a cross-attention mechanism. Interacting allows for learning query vectors. The system automatically searches for feature regions related to human targets in multi-channel radar heatmap sequences. After processing by a decoder, it obtains the corresponding learnable query vectors. The one-to-one prediction output results include human behavior category results and human instance feature results, which constitute the prediction instance set and serve as the basis for constructing the subsequent matching cost function.

[0032] A further improvement in this application is that step 4 specifically involves: constructing a matching cost function based on the human behavior category results and the human instance feature results; and achieving optimal alignment between instances in the predicted instance set and target individuals in the real target set by minimizing the overall matching cost function. The matching cost function is expressed as:

[0033]

[0034] in, For the first A real goal For the first Learnable query vectors The prediction results For the first The prediction belongs to the true category. The probability, For the first Normalized skeletal keypoint coordinates corresponding to each real target For the first The human instance features corresponding to each prediction result and These represent the weight coefficients of the classification cost term and the regression cost term, respectively;

[0035] By solving the problem of minimizing the matching cost function, the globally optimal instance correspondence between the predicted instance set and the true target set is obtained:

[0036]

[0037] in, Indicates by The set of all possible permutations of elements is used to achieve one-to-one alignment between the instances in the predicted instance set and the target personnel in the real target set.

[0038] A further improvement in this application is that, in step 5, in order to enhance the semantic expression of radar feature representation, a skeletal point teacher network is introduced for knowledge distillation. The skeletal point teacher network includes a classroom network, student network projection, and a dual gating mechanism, specifically including the following steps:

[0039] Step 5.1: Extracting Skeletal Semantic Features from the Teacher Network: A skeletal architecture is adopted. The input is a sequence of key points of the human skeleton collected synchronously with radar data. The temporal self-attention and joint self-attention modules in the skeletal point teacher network are used to extract skeletal semantic features. ;

[0040] Step 5.2, Student Network Projection: A projector is added to the output of the radar student network. The projector consists of a linear layer, layer normalization, activation function, and another linear layer, mapping the radar feature representation to the skeletal features output by the skeletal point teacher network. Radar features are obtained from the feature space in which they are located. ;

[0041] Step 5.3: Introduce a dual gating mechanism, i.e., set a confidence threshold. and distance threshold Only when the prediction confidence of the skeletal point teacher network is higher than the confidence threshold. Furthermore, the distance between the radar-predicted location and the actual location is lower than the distance threshold. Distillation losses are only calculated at this time:

[0042]

[0043] in, This indicates the number of matching feature pairs that participate in distillation after being screened through a dual gating mechanism. and The first and second matches were successful. Radar features and skeletal features.

[0044] The beneficial effects of this application are:

[0045] This application is based on millimeter-wave MIMO through-wall radar echo signals for human behavior recognition, without relying on visible light images. It effectively protects personal privacy while realizing human perception behind walls, and is suitable for privacy-sensitive scenarios.

[0046] This application introduces a query-based multi-person human feature decoding method, which realizes automatic differentiation and one-to-one modeling of different human targets, avoids matching ambiguity caused by the uncertainty of target order in multi-person human behavior recognition, and improves the recognition robustness in complex multi-person scenarios.

[0047] This application combines prior information about human anatomy to constrain and model radar temporal features, making the learned behavioral features more consistent with human movement patterns, effectively suppressing radar noise and non-human interference, and improving the accuracy of behavior recognition.

[0048] This application introduces a cross-modal knowledge distillation mechanism, using human skeletal features as teacher signals to guide the learning of radar features, achieving semantic alignment between different modalities. This method can effectively transfer human structural information to radar feature representations, thereby improving the expressive power and discriminative performance of radar behavior recognition models.

[0049] This application can directly recognize the behavior of people who did not participate in the training after the model is trained, without the need to retrain the network, and has good generalization ability and practical application value. Attached Figure Description

[0050] Figure 1 This is the flowchart of the method used in this application.

[0051] Figure 2 This is a flowchart of the cross-modal multi-feature fusion behavior and action recognition process of this application.

[0052] Figure 3 This is a schematic diagram of the horizontal and vertical processing branches of the dual-branch 3D convolutional backbone network in this application.

[0053] Figure 4 This is a schematic diagram of the Intra-View Multi-Scale Interaction (IVMSI) module in this application.

[0054] Figure 5 This is a schematic diagram of the Cross-View Dynamic Fusion (CVDF) module in this application.

[0055] Figure 6 These are the actions and behaviors in a multi-person scenario in this application. Detailed Implementation

[0056] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the present invention. That is, in some embodiments of the present invention, these practical details are not essential. In addition, for the sake of simplicity, some conventional structures and components will be shown in the drawings in a simple schematic manner.

[0057] like Figure 1 and Figure 2 As shown, this application discloses a method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion. The method specifically includes the following steps:

[0058] Step 1: MIMO Radar Data Acquisition and Preprocessing: Initialize and configure the MIMO through-wall radar system, acquire human motion echo signals, and perform multidimensional Fourier transform preprocessing on the acquired human motion echo signals to generate a multi-channel radar heat map sequence containing range-Doppler information and range-angle information; the multi-channel radar heat map sequence includes four independent heat maps: horizontal range-Doppler heat map, vertical range-Doppler heat map, horizontal range-angle heat map, and vertical range-angle heat map.

[0059] Step 2: Construct a dual-view radar feature extraction and fusion network: The horizontal range-Doppler heatmap and horizontal range-angle heatmap from the horizontal view are concatenated along the channel dimension to form the input horizontal radar heatmap for the horizontal processing branch. The vertical range-Doppler heatmap and vertical range-angle heatmap from the vertical view are concatenated to form the vertical radar heatmap. Both the horizontal and vertical radar heatmaps are input into the dual-view radar feature extraction and fusion network for training. This yields multi-scale features from both the horizontal and vertical views, and performs multi-scale interaction, dynamic fusion, and aggregation to output a compact radar feature representation. ;

[0060] Step 3: Transformer-based feature decoding and querying: Construct a Transformer network containing an encoder and a decoder to process the radar feature representation output from Step 2. To perform modeling, a set of learnable query vectors is introduced that are independent of the number of human targets in the scene. Automatically search for feature regions related to human targets in horizontal and vertical radar heatmaps, and use an attention mechanism to guide learnable query vectors to adaptively aggregate radar feature representations. After being processed by the decoder, each learnable query vector is obtained. The one-to-one output results constitute the real target set, including human behavior categories and human instance features, and serve as the basis for constructing the subsequent matching cost function.

[0061] In step 3, the Transformer network includes an encoder and a decoder. The encoder utilizes a multi-head self-attention (MSA) mechanism to represent radar features. Sequence modeling is performed, and the decoder uses a cross-attention mechanism to learn the query vector and radar feature representation. The interaction and attention calculation formula is as follows:

[0062]

[0063] in, This is the query vector in the decoder. The radar feature key-value pairs output by the encoder. As a scaling factor, this mechanism allows the query vector to automatically focus on the human target area in the radar heatmap without relying on external sensors for physical cropping.

[0064] Step 3 specifically includes the following steps:

[0065] Step 3.1, Position Encoding: Introducing Sine Positional Encoding for radar feature representation. Given the coordinates (x, y, t), generate a position vector. And superimposed on radar feature representation Above, to preserve spatial structural information;

[0066] Step 3.2, Encoder Processing: The encoder consists of L encoder layers. Each encoder layer comprises a multi-head self-attention mechanism (MSA) and a feedforward neural network (FFN). The MSA is calculated as follows:

[0067]

[0068]

[0069] in, The projection matrix is ​​learnable;

[0070] To adapt to the input format of the Transformer encoder, the radar feature representation is... Expand into a feature sequence and introduce the corresponding position vector. The initial input to the encoder is obtained by adding elements one by one: For the first The output of a layer encoder can be expressed as:

[0071]

[0072]

[0073] Finally, after After layer encoder, the encoder output radar feature is obtained. ;

[0074] Step 3.3, Decoder Query: Initialization Learnable query vectors Learnable query vectors The decoder first performs self-attention interaction, and then interacts with the radar features output by the encoder through a cross-attention mechanism. Interacting allows for learning query vectors. The system automatically searches for feature regions related to human targets in multi-channel radar heatmap sequences. After processing by a decoder, it obtains the corresponding learnable query vectors. The prediction outputs, which correspond one-to-one, include human behavior category results and human instance feature results, forming a predicted instance set, which serves as the basis for constructing the subsequent matching cost function. Human behavior categories refer to each learnable query vector. The classification output in the corresponding prediction results after decoder processing; the human instance feature results are the learnable query vectors. The corresponding instance features are output after being processed by the decoder.

[0075] Step 4 introduces a target instance association method based on a globally optimal allocation strategy. During the model training phase, an overall matching relationship is constructed between the predicted instance set and the real target set provided by the training samples, and the overall matching cost function is minimized to achieve optimal alignment between instances in the predicted instance set and target personnel in the real target set, thereby eliminating the impact of prediction order uncertainty on the recognition results. Specifically, Step 4 involves constructing a matching cost function based on human behavior category results and human instance feature results, and using an instance-level matching module based on a globally optimal allocation strategy to establish an overall correspondence between the predicted instance set and the real target set. The instance-level matching module achieves optimal alignment between instances in the predicted instance set and target personnel in the real target set by minimizing the overall matching cost function. The matching cost function comprehensively considers the differences in classification prediction and human structure regression, and is expressed as follows:

[0076]

[0077] in, For the first A real goal For the first Learnable query vectors The prediction results For the first The prediction belongs to the true category. The probability, For the first Normalized skeletal keypoint coordinates corresponding to each real target For the first The human instance features corresponding to each prediction result and These represent the weight coefficients of the classification cost term and the regression cost term, respectively. This step resolves the matching ambiguity problem caused by the disorder of targets in multi-person scenarios.

[0078] By solving the problem of minimizing the matching cost function, the globally optimal instance correspondence between the predicted instance set and the true target set is obtained:

[0079]

[0080] in, Indicates by The set of all possible permutations of elements is used to achieve one-to-one alignment between the instances in the predicted instance set and the target personnel in the real target set.

[0081] Step 5: Introduce a cross-modal structural prior distillation strategy to construct a skeletal point teacher network and a radar student network, and constrain the radar feature representation through a structural consistency loss function. Align the model with the semantic space of the human skeleton to complete model training;

[0082] In step 5, to enhance the semantic representation of radar features, a skeletal point teacher network is introduced for knowledge distillation. The skeletal point teacher network includes a classroom network, student network projection, and a dual gating mechanism, specifically including the following steps:

[0083] Step 5.1: Extracting Skeletal Semantic Features from the Teacher Network: A skeletal architecture is adopted. The input is a sequence of key points of the human skeleton collected synchronously with radar data. The temporal self-attention and joint self-attention modules in the skeletal point teacher network are used to extract skeletal semantic features. For the sequence of human skeletal key points input by the teacher network, human motion relationships are modeled from both the temporal and joint topological dimensions, and skeletal semantic features are extracted. These skeletal semantic features serve as the teacher-side representation for subsequent cross-modal structural prior distillation, and are subject to structural consistency constraints with the features projected onto the radar student network. These features are implemented internally within the teacher network and are used to support subsequent feature alignment and distillation loss calculation.

[0084] Step 5.2, Student Network Projection: A projector is added to the output of the radar student network. The projector consists of a linear layer, layer normalization, activation function, and another linear layer, mapping the radar feature representation to the skeletal features output by the skeletal point teacher network. Radar features are obtained from the feature space in which they are located. The radar network includes a dual-view radar feature extraction and fusion network and a Transformer encoding and decoding network. Specifically, adding a projection head to the output of the radar student network means adding a projection mapping after the human target features output by the student network to align with the skeletal semantic features extracted by the teacher network.

[0085] Step 5.3: Introduce a dual gating mechanism, i.e., set a confidence threshold. and distance threshold Only when the prediction confidence of the skeletal point teacher network is higher than the confidence threshold. Furthermore, the distance between the radar-predicted location and the actual location is lower than the distance threshold. Distillation losses are calculated only at this time to filter out low-quality teaching samples:

[0086]

[0087] in, This indicates the number of matching feature pairs that participate in distillation after being screened through a dual gating mechanism. and The first and second matches were successful. The loss function is applied to both radar features and skeletal features. It forces the radar network to learn the movement patterns of the human skeleton, enhancing its ability to discriminate complex movements.

[0088] Step 6: In the inference phase, remove the skeleton point teacher network and only use the trained radar student network. Input the real-time collected radar data into the radar student network to perform real-time behavior classification of multiple targets.

[0089] like Figure 3 As shown, the dual-view radar feature extraction and fusion network includes a dual-branch 3D convolutional backbone network, a multi-scale interaction module within a single view, a cross-view dynamic fusion module, and a multi-scale aggregation module.

[0090] The horizontal processing branch receives horizontal radar heatmaps from the horizontal viewpoint, and the vertical processing branch receives vertical radar heatmaps from the vertical viewpoint. The horizontal and vertical radar heatmaps are then concatenated along the channel dimension. After concatenation, the horizontal range-Doppler heatmap and the horizontal range-angle heatmap from the horizontal viewpoint are concatenated along the channel dimension to form the input horizontal radar heatmap for the horizontal processing branch. The vertical range-Doppler heatmap and the vertical range-angle heatmap from the vertical viewpoint are then concatenated to form a vertical radar heatmap sequence. The horizontal and vertical radar heatmap sequences are then input into a dual-view radar feature extraction and fusion network for training, and spatiotemporal feature encoding is performed to extract shallow feature maps and deep semantic feature maps to extract multi-scale features from the horizontal and vertical viewpoints.

[0091] After the 3D convolutional backbone network outputs shallow feature maps and deep semantic feature maps, a multi-scale interaction module within a single viewpoint performs unified mapping and interactive enhancement on the shallow feature maps and deep semantic feature maps, forming enhanced multi-scale features from both the horizontal and vertical views. The multi-scale interaction module within a single viewpoint is as follows: Figure 4 As shown.

[0092] At the corresponding scale, multi-scale features from both the horizontal and vertical perspectives are input into the cross-view dynamic fusion module. An adaptive weight allocation mechanism is used to weight and fuse these features, highlighting perspective features with higher information content and stronger discriminative power, resulting in the fused cross-view features. The structure of the cross-view dynamic fusion module is as follows: Figure 5 As shown.

[0093] A multi-scale aggregation module is used to unify and integrate the fused cross-view features, outputting a compact radar feature representation. This provides input for subsequent human target decoding and behavior classification based on Transformer.

[0094] To verify this application, the following comparative experiment is provided:

[0095] 1. Dataset like Figure 3 As shown, six behavioral actions were designed: walking, sitting, squatting, lying down, crawling, and attacking. In the experiment, a MIMO radar was placed outside a 26cm thick brick wall, and the target personnel were allowed to demonstrate their behavior within a 15-meter range behind the wall. Different personnel randomly demonstrated different behaviors.

[0096] 2. Measurement Indicators

[0097] (1) Accuracy represents the proportion of the number of samples correctly predicted by the model to the total number of test samples, and is used to measure the overall classification performance of the model.

[0098] (2) Precision reflects the proportion of samples that the model predicts as a certain behavior category, but which actually belong to that category. It is used to evaluate the reliability of the model's prediction results.

[0099] (3) Recall represents the proportion of samples correctly identified by the model in a certain behavior category, and is used to measure the model's ability to detect samples in that category.

[0100] (4) F1-score is a comprehensive indicator of precision and recall, which can achieve a balance between the two and more comprehensively reflect the overall performance of the model in multi-class behavior recognition tasks.

[0101] 3. Comparison Algorithm (1): Ding M, Dongye G, Lv P, et al. Fml-vit: A lightweight visiontransformer algorithm for human activity recognition using fmcw radar[J]. IEEE Sensors Journal, 2024. (2): Wang Y, Wang Y, Xu C, et al. SelaFD: Seamless Adaptation of VisionTransformer Fine-tuning for Radar-based Human Activity Recognition[C] / / ICASSP2025-2025 IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP). IEEE, 2025: 1-5. (3): Luo F, Bodanese E, Khan S, et al. Spectro-temporal modeling for human activity recognition using a radar sensor network[J]. IEEE Transactions onGeoscience and Remote Sensing, 2023, 61: 1-13.

[0105] 4. Experimental Results

[0106] As shown in Table 1, the collected data is processed into multi-channel radar heatmaps as described in this application and input into a dual-branch feature coding network for training. After the system is trained, behavior recognition is performed on untrained users. Furthermore, the dataset is divided into... Figure 6 The behavior recognition method described herein is used to process and compare the results of new user recognition.

[0107] Table 1. Recognition accuracy and average accuracy of new user behavior in scenarios with different numbers of users.

[0108]

[0109] Table 2. Master Method Ablation Experiment

[0110]

[0111] Experimental results show that after the system is completed, the accuracy, precision, recall, and F1-score of the present invention for recognizing the behavior of untrained personnel are all higher than other methods, proving the true effectiveness of the present invention.

[0112] This application utilizes MIMO radar to perform non-contact radio wave transmission and detection of scenes behind walls, acquiring echo signals generated by human movement. Fourier transforms are applied to the original signals to generate a multi-channel radar heatmap sequence containing range and Doppler information. A dual-view radar feature extraction and fusion network is employed to model the multi-channel temporal heatmap, extracting multi-level spatiotemporal features from both horizontal and vertical perspectives. Within each perspective, a multi-scale interaction mechanism within a single perspective is introduced to enhance the collaborative expression between shallow detail information and deep semantic information. Furthermore, at corresponding scales, a cross-view dynamic fusion mechanism is used to adaptively weight and fuse the dual-view features, combining this with a multi-scale aggregation strategy to form a unified radar feature representation. Subsequently, a query-based multi-person feature decoding module is introduced to distinguish and model different human targets in the scene, achieving automatic matching of multi-person human features. Based on this, prior information about human structure is used to constrain the radar features. By introducing the structural relationships of key human points, the radar features are guided to learn a temporal representation that conforms to the laws of human movement, enhancing the model's ability to discriminate complex actions and multi-person interaction scenes. Finally, the corresponding human behavior categories are output through a behavior classification network to achieve multi-person human behavior recognition behind a wall. This invention fully utilizes the dual-view, multi-scale spatiotemporal features of millimeter-wave radar and human structural information, exhibiting high recognition accuracy and good robustness and real-time performance even in complex occlusion environments.

[0113] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion, characterized in that: The multi-person human behavior recognition method specifically includes the following steps: Step 1: MIMO Radar Data Acquisition and Preprocessing: Initialize and configure the MIMO through-wall radar system, acquire human motion echo signals, and perform multidimensional Fourier transform preprocessing on the acquired human motion echo signals to generate a multi-channel radar heatmap sequence containing range-Doppler information and range-angle information; the multi-channel radar heatmap sequence includes four independent heatmaps: horizontal range-Doppler heatmap, vertical range-Doppler heatmap, horizontal range-angle heatmap, and vertical range-angle heatmap; Step 2: Construct a dual-view radar feature extraction and fusion network: The horizontal range-Doppler heatmap and horizontal range-angle heatmap from the horizontal view are concatenated along the channel dimension to form the input horizontal radar heatmap for the horizontal processing branch. The vertical range-Doppler heatmap and vertical range-angle heatmap from the vertical view are concatenated to form the vertical radar heatmap. Both the horizontal and vertical radar heatmaps are input into the dual-view radar feature extraction and fusion network for training. This yields multi-scale features from both the horizontal and vertical views, and performs multi-scale interaction, dynamic fusion, and aggregation to output a compact radar feature representation. ; Step 3: Transformer-based feature decoding and querying: Construct a Transformer network containing an encoder and a decoder to process the radar feature representation output from Step 2. To perform modeling, learnable query vectors are introduced. Automatically search for feature regions related to human targets in horizontal and vertical radar heatmaps, and use an attention mechanism to guide learnable query vectors to adaptively aggregate radar feature representations. After being processed by the decoder, each learnable query vector is obtained. The prediction output results correspond one-to-one, and the prediction output results constitute a set of prediction instances, including human behavior categories and human instance features, and serve as the basis for constructing the subsequent matching cost function. Step 4: By constructing an overall matching relationship between the predicted instance set and the real target set provided by the training samples, and minimizing the overall matching cost function, the optimal alignment between the instances in the predicted instance set and the real target personnel in the real target set is achieved, thereby eliminating the impact of prediction order uncertainty on the recognition results. Step 5: After obtaining the optimal alignment relationship between the predicted instance set and the real target set in Step 4, a cross-modal structural prior distillation strategy is introduced to construct a skeleton point teacher network and a radar student network. For successfully matched instance pairs, the target-level radar instance features output by the decoder in Step 3 are constrained to align to the human skeleton semantic space through the structural consistency loss function to complete the model training. Step 6: In the inference phase, remove the skeleton point teacher network and only use the trained radar student network. Input the real-time collected radar data into the radar student network to perform real-time behavior classification of multiple targets.

2. The method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion as described in claim 1, characterized in that: In step 2, the dual-view radar feature extraction and fusion network includes a dual-branch 3D convolutional backbone network, a multi-scale interaction module within a single view, a cross-view dynamic fusion module, and a multi-scale aggregation module. The horizontal processing branch receives horizontal radar heatmaps from a horizontal perspective, and the vertical processing branch receives vertical radar heatmaps from a vertical perspective. The horizontal and vertical radar heatmaps are then concatenated along the channel dimension. After concatenation, the horizontal range-Doppler heatmap and the horizontal range-angle heatmap from the horizontal perspective are concatenated along the channel dimension to form the input horizontal radar heatmap for the horizontal processing branch. The vertical range-Doppler heatmap and the vertical range-angle heatmap from the vertical perspective are then concatenated to form a vertical radar heatmap sequence. The horizontal and vertical radar heatmap sequences are then input into a dual-view radar feature extraction and fusion network for training, and spatiotemporal feature encoding is performed to extract shallow feature maps and deep semantic feature maps. After the 3D convolutional backbone network outputs shallow feature maps and deep semantic feature maps, the multi-scale interaction module in a single view performs unified mapping and interactive enhancement on the shallow feature maps and deep semantic feature maps, forming enhanced multi-scale features in the horizontal view and multi-scale features in the vertical view. At the corresponding scale, the multi-scale features of the horizontal view and the multi-scale features of the vertical view are input into the cross-view dynamic fusion module. The multi-scale features of the horizontal view and the multi-scale features of the vertical view are weighted and fused to obtain the fused cross-view fusion features. A multi-scale aggregation module is used to unify and integrate the fused cross-view features, outputting a compact radar feature representation. This provides input for subsequent human target decoding and behavior classification based on Transformer.

3. The method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion as described in claim 1, characterized in that: In step 3, the Transformer network includes an encoder and a decoder. The encoder utilizes a multi-head self-attention (MSA) mechanism to represent radar features. Sequence modeling is performed, and the decoder uses a cross-attention mechanism to learn the query vector and radar feature representation. The interaction and attention calculation formula is as follows: in, This is the query vector in the decoder. The radar feature key-value pairs output by the encoder. This is the scaling factor.

4. The method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion as described in claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3.1, Position Encoding: Introducing Sine Positional Encoding for radar feature representation. Given the coordinates (x, y, t), generate a position vector. And superimposed on radar feature representation Above, to preserve spatial structural information; Step 3.2, Encoder Processing: The encoder consists of L encoder layers. Each encoder layer comprises a multi-head self-attention mechanism (MSA) and a feedforward neural network (FFN). The MSA is calculated as follows: in, The projection matrix is ​​learnable; To adapt to the input format of the Transformer encoder, the radar feature representation is... Expand into a feature sequence and introduce the corresponding position vector. The initial input to the encoder is obtained by adding elements one by one: For the first The layer encoder, whose output is represented as: Finally, after After layer encoder, the encoder output radar feature is obtained. ; Step 3.3, Decoder Query: Initialization Learnable query vectors Learnable query vectors The decoder first performs self-attention interaction, and then interacts with the radar features output by the encoder through a cross-attention mechanism. Interacting allows for learning query vectors. The system automatically searches for feature regions related to human targets in multi-channel radar heatmap sequences. After processing by a decoder, it obtains the corresponding learnable query vectors. The one-to-one prediction output results include human behavior category results and human instance feature results, which constitute the prediction instance set and serve as the basis for constructing the subsequent matching cost function.

5. A method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion as described in claim 4, characterized in that: Step 4 specifically involves: constructing a matching cost function based on the human behavior category results and human instance feature results; and achieving optimal alignment between instances in the predicted instance set and target individuals in the real target set by minimizing the overall matching cost function. The matching cost function is expressed as follows: in, For the first A real goal For the first Learnable query vectors The prediction results For the first The prediction belongs to the true category. The probability, For the first Normalized skeletal keypoint coordinates corresponding to each real target For the first The human instance features corresponding to each prediction result and These represent the weight coefficients of the classification cost term and the regression cost term, respectively; By solving the problem of minimizing the matching cost function, the globally optimal instance correspondence between the predicted instance set and the true target set is obtained: in, Indicates by The set of all possible permutations of elements is used to achieve one-to-one alignment between the instances in the predicted instance set and the target personnel in the real target set.

6. The method for multi-person human behavior recognition based on MIMO through-wall radar cross-modal multi-feature fusion according to claim 1, characterized in that: In step 5, to enhance the semantic representation of radar features, a skeletal point teacher network is introduced for knowledge distillation. The skeletal point teacher network includes a classroom network, student network projection, and a dual gating mechanism, specifically including the following steps: Step 5.1: Extracting Skeletal Semantic Features from the Teacher Network: A skeletal architecture is adopted. The input is a sequence of key points of the human skeleton collected synchronously with radar data. The temporal self-attention and joint self-attention modules in the skeletal point teacher network are used to extract skeletal semantic features. ; Step 5.2, Student Network Projection: A projector is added to the output of the radar student network. The projector consists of a linear layer, layer normalization, activation function, and another linear layer, mapping the radar feature representation to the skeletal features output by the skeletal point teacher network. Radar features are obtained from the feature space in which they are located. ; Step 5.3: Introduce a dual gating mechanism, i.e., set a confidence threshold. and distance threshold Only when the prediction confidence of the skeletal point teacher network is higher than the confidence threshold. Furthermore, the distance between the radar-predicted location and the actual location is lower than the distance threshold. Distillation losses are only calculated at this time: in, This indicates the number of matching feature pairs that participate in distillation after being screened through a dual gating mechanism. and The first and second matches were successful. Radar features and skeletal features.

Images