Secret-related space multi-mode illegal behavior identification method based on millimeter wave radar

Abstract

The invention discloses a secret-related space multi-mode illegal behavior identification method based on a millimeter wave radar, relates to the technical field of intelligent security and protection, and aims to solve the problems of high missing and false alarm rate, privacy leakage and poor environmental adaptability of a traditional security and protection means. According to the method, hardware-level synchronization is realized through a millimeter-wave radar and a distributed microphone array, and a low-density point cloud sequence and multi-channel mixed audio in a monitoring area are acquired; reconstructing the low-density point cloud into a high-density point cloud capable of depicting fine human body postures by using a point cloud super-resolution model based on a reversible neural network, and extracting pure target voice through blind source separation and adaptive beam forming in combination with radar target azimuth information; then, respectively extracting point cloud behavior features and voice audio features, and performing deep fusion through a cross-modal collaborative attention mechanism to obtain joint features; and finally, inputting into a multi-layer perceptron classifier, outputting illegal behavior probability distribution through a Softmax function, and judging a result.

Description

Technical Field

[0001] This invention relates to the field of intelligent security technology, and in particular to a method for identifying multimodal violations in classified spaces based on millimeter-wave radar. Background Technology

[0002] Voice interaction and behavioral monitoring of personnel within classified units have become a core component of intelligent security systems, directly impacting the security of sensitive information and placing extremely high demands on the accuracy, privacy protection, and environmental adaptability of identification solutions. However, existing technologies for identifying violations in classified areas still suffer from numerous key bottlenecks, making it difficult to meet the requirements of high-level security protection.

[0003] Traditional surveillance solutions often rely on optical imaging video surveillance systems. On the one hand, these systems indiscriminately collect biometric information such as faces and body postures, which can easily lead to serious privacy leaks. On the other hand, these systems are highly dependent on ambient lighting conditions, and can easily create blind spots in nighttime, enclosed low-light, and other scenarios, resulting in interruptions in monitoring continuity.

[0004] Meanwhile, existing identification systems generally employ single-modal perception technology: either analyzing behavior solely through point cloud data or relying solely on audio signals to recognize speech content. This fails to establish a correlation between behavior and speech, only able to judge surface-level actions but unable to discern underlying intent to leak information, resulting in persistently high false positive and false negative rates in complex scenarios. Furthermore, while some solutions attempt to combine multiple modalities, they suffer from shortcomings in data synchronization and deep feature fusion, failing to fully leverage the synergistic advantages of multi-source information. This further limits the accuracy and robustness of violation identification, necessitating targeted breakthroughs. Summary of the Invention

[0005] The purpose of this invention is to provide a method for identifying multimodal violations in classified spaces based on millimeter-wave radar. This method eliminates the need for imaging acquisition, effectively protects privacy, significantly improves the accuracy of violation identification in classified areas and enhances adaptability to complex environments, and is suitable for intelligent security in high-level classified locations.

[0006] To achieve the above objectives, this invention provides a method for identifying multimodal violations in classified spaces based on millimeter-wave radar, the steps of which are as follows: S1. Synchronous acquisition of multimodal raw data: A sensing system composed of millimeter-wave radar and distributed microphone array is configured to acquire low-density point cloud sequences and multi-channel mixed audio signals within the monitoring area; S2. High-density point cloud sequence reconstruction: The low-density point cloud sequence acquired in step S1 is input into a pre-trained point cloud super-resolution model. The point cloud super-resolution model constructs a mapping relationship from low-density point cloud to high-density point cloud, and outputs a high-density point cloud sequence that can depict detailed human poses. S3. Target speech enhancement and separation: The multi-channel mixed audio signal collected in step S1 is processed, and combined with the real-time target azimuth information provided by the millimeter-wave radar in step S1, interference and noise are suppressed, and the clean target speech associated with the radar-monitored target is extracted. S4. Multimodal feature extraction: Extract features from the high-density point cloud sequence output in step S2 and the clean target speech output in step S3 to obtain behavioral feature vectors representing human action postures and audio feature vectors representing speech content. S5. Cross-modal spatiotemporal collaborative attention fusion: The behavioral feature vector and audio feature vector obtained in step S4 are fused using a cross-modal collaborative attention mechanism. After feature association enhancement and normalization, a joint feature vector of fused behavior and speech information is obtained. S6. Identification and Classification of Violations: Input the joint feature vector output from step S5 into the classifier, calculate the probability distribution of each violation category through the classifier, and determine the final violation type in the monitoring area based on the principle of maximum probability.

[0007] Preferably, in step S1, the low-density point cloud sequence is denoted as... ,in T The total number of time frames, and the point cloud in each frame. Include N A three-dimensional spatial point ( x , y , z The multi-channel mixed audio signal is denoted as... ,in C Number of microphone channels L The length of the audio signal.

[0008] Preferably, in step S2, the point cloud super-resolution model is a point cloud super-resolution flow model constructed based on a reversible neural network. The point cloud super-resolution flow model is constructed using a reversible neural network. Establish a deterministic bijective mapping from the latent space to the data space, and transform simple prior distributions. Mapping to high-density point cloud data distribution The training objective is to maximize the log-likelihood of the observed data, and the formula for calculating the log-likelihood is: (1) in, The output is a high-density point cloud sequence. For expectation operator, This is a function for calculating determinants. Reversible neural network For high-density point cloud sequences The partial derivatives of .

[0009] Preferably, the specific processing procedure of step S3 includes: S3.1. Blind source separation is performed on the multi-channel mixed audio signal. Multiple sound source signals are initially separated in the short-time Fourier transform domain using a time-frequency masking method. The specific formula is as follows: (2) (3) in, It is the short-time Fourier transform function. For the time-frequency domain representation of mixed audio, Let be the time-frequency mask for the k-th sound source. This is the time-frequency domain estimation result for the k-th sound source, where t is the time frame index and f is the frequency index. S3.2 Target azimuth angle provided by millimeter-wave radar at time t The optimal weight vector is solved using a linearly constrained minimum variance beamformer. Adaptive beamforming is applied to the initially separated sound source signals to suppress residual interference and noise, resulting in clean target speech. The analytical solution formulas for the optimization problem and the optimal weight vector are as follows: (4) (5) in, For beamforming weights, It is the covariance matrix of noise and interference. It is a guide vector.

[0010] Preferably, the specific extraction process in step S4 includes: S4.1 Constructing point clouds for each frame of the high-density point cloud sequence k Nearest neighbor graph, through graph attention network k The nearest neighbor graph is used to encode spatial features, and then the encoded sequential features are input into a temporal convolutional network for temporal feature extraction to obtain behavioral feature vectors. Among them, nodes in the graph attention network i The output feature calculation relationship is as follows: (6) (7) (8) in, Let W be the set of neighboring nodes of node i, and let W be the shared linear transformation weight matrix. It is a non-linear activation function. Attention coefficient Score for attention; S4.2 First, convert the clean target speech into a Mel-scale spectrogram. Where F is the number of Mel filter banks, To determine the time frame number of the spectrogram, local spectral features are extracted from the Mel-scale spectrogram using a deep convolutional neural network. These local spectral features are then subjected to global average pooling and flattened into a sequence of temporal feature vectors. The temporal feature vector sequence is then input into a bidirectional gated recurrent unit network to extract context-related features, resulting in an audio feature vector. The forward propagation relationship of the bidirectional gated cyclic unit network is as follows: (9) (10) (11) (12) in, Let be the temporal feature vector at time t. Let be the hidden state at time t-1. To update the gate output, To reset the gate output, In the candidate hidden state, Let be the hidden state at time t. , , , , , This is the weight matrix. , , This is the bias vector.

[0011] Preferably, the specific fusion process in step S5 includes: S5.1, Regarding behavioral feature vectors With audio feature vectors Perform linear projections on each vector to generate the corresponding query vector, key vector, and value vector, as follows: (13) (14) Among them, among them, , , The projection weight matrix is ​​the feature vector of the behavior. , , This is the projection weight matrix of the audio feature vectors; S5.2, Query vector based on behavioral feature vectors Key vector of audio feature vector Attention calculation is performed, combining the value vector of the audio feature vector. Obtain behavioral context-enhanced features ; Query vector based on audio feature vector Key vector of behavioral feature vector Attention calculation is performed by combining the value vector of the behavioral feature vector. Obtain audio context enhancement features The specific formula is as follows: (15) (16) in, The dimension of the key vector; S5.3. Perform residual connections and layer normalization on the behavior context enhancement features and the original behavior feature vector, and on the audio context enhancement features and the original audio feature vector, respectively. Then, concatenate the processed feature vectors to obtain the joint feature vector. The specific formula is as follows: (17) (18) (19) in, The behavior feature vector after layer normalization. This is the audio feature vector after layer normalization.

[0012] Preferably, in step S6, the classifier is a multilayer perceptron classifier, including several fully connected layers and Dropout layers. The probability distribution of each violation category is calculated and output using the Softmax function. The formula for calculating the probability distribution is: (20) in, Represents the joint eigenvector The probability of belonging to category c. , This is the classification weight matrix. , Let C be the classification bias vector, and C be the set of all violation categories. The categories of violations are categorized as follows: unauthorized conversations, data theft, unauthorized operations, taking photos, making phone calls, and irrelevant behavior.

[0013] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: (1) Achieving a key breakthrough in data processing. Its innovative research and development of behavior feature enhancement technology based on point cloud super-resolution flow model, relying on a carefully designed reversible neural network architecture, can reconstruct the collected original low-density point cloud sequence into high-density point cloud data with rich details in real time. This high-density point cloud can accurately depict the three-dimensional spatial coordinates and dynamic change trajectory of each joint of the human body, providing high-quality spatial data support for subsequent fine behavior recognition and motion analysis, and solving the problem that traditional low-density point clouds are difficult to depict the fine posture of the human body.

[0014] (2) At the multimodal fusion level, this invention establishes a deep collaborative judgment mechanism for speech and behavior. First, the mixed audio is preliminarily processed by blind source separation technology, and then the interference and noise are suppressed by adaptive beamforming technology to extract directional pure speech features. Then, through an advanced cross-modal feature fusion algorithm, the visual behavior features represented by high-density point cloud and the directional speech features are fused at multiple levels to fully explore the intrinsic correlation between the two in the spatiotemporal dimension, significantly improve the system's environmental adaptability and recognition robustness in complex scenarios, and avoid the limitations of single-modal perception.

[0015] (3) This invention emphasizes the synergistic unity of privacy protection and recognition accuracy. The system adopts a non-imaging monitoring method, fundamentally avoiding the collection of personal biometric information; combined with directional sound source enhancement technology, it can minimize the probability of collecting the voices of unrelated personnel. This design ensures high-precision recognition of violations while building a comprehensive privacy protection system, effectively avoiding the privacy leakage risks of traditional monitoring, and providing a safe and reliable solution for intelligent monitoring of important locations.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the workflow of an embodiment of the method for identifying multimodal violations in classified spaces based on millimeter-wave radar according to the present invention. Figure 2 This is a schematic diagram of the high-density point cloud behavior during high-density point cloud reconstruction in Embodiment S2 of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Example like Figure 1 As shown, a method for identifying multimodal violations in classified spaces based on millimeter-wave radar includes the following steps: S1: Data Acquisition and Synchronization First, domestically produced Doppler radar boards are used to collect human body data, millimeter-wave radar and distributed microphone arrays are deployed, and hardware time synchronization is performed.

[0022] Acquiring low-density radar point cloud sequences: ,in .

[0023] Acquire multi-channel mixed audio signals:

[0024] S2: High-density point cloud reconstruction Will Input pre-trained point cloud super-resolution flow model High-density point cloud sequences are generated through inverse transformation. .like Figure 2 As shown, (a) is a high-density point cloud of the action of taking a picture; (b) is a high-density point cloud of the action of bending over; (c) is a high-density point cloud of the action of turning around; (d) is a high-density point cloud of the action of raising an arm; (e) is a high-density point cloud of the action of making a phone call; and (f) is a high-density point cloud of the action of squatting.

[0025] The training objective of the model is to maximize the log-likelihood.

[0026] S3: Target Speech Enhancement and Separation Blind source separation: Blind source separation is performed on multi-channel mixed audio signals. Multiple sound source signals are initially separated in the short-time Fourier transform domain using a time-frequency masking method. The formula is as follows:

[0027] Adaptive beamforming: utilizing target azimuth provided by radar Solve for the optimal weights of the LCMV beamformer:

[0028] Obtain pure audio .

[0029] Behavioral feature stream input: high-density point cloud sequence .

[0030] Audio feature stream input: The converted Mel language spectrogram.

[0031] S4: Spatiotemporal Feature Extraction Behavioral characteristics: A k-nearest neighbor graph is constructed for each frame of point cloud, and spatial encoding is performed using a graph attention network (GAT). The output of node i is:

[0032] The serialized graph features are input into a temporal convolutional network (TCN), which outputs behavioral features. .

[0033] Audio features: Local features are extracted from the Mel speech spectrogram using a CNN, and then temporal dependencies are captured using a bidirectional GRU network to output audio features. .

[0034] S5: Cross-modal fusion A collaborative attention mechanism is used to compute behavioral context enhancement features and audio context enhancement features:

[0035] After residual connection and layer normalization, joint features are obtained by splicing. .

[0036] S6: Identification and Classification of Violations Will Input an MLP classifier and calculate the class probabilities using the Softmax function:

[0037] The final behavior category is determined based on the highest probability. Violation categories include: unauthorized conversation, data theft, unauthorized operations, taking photos, making phone calls, and irrelevant behavior.

[0038] This invention achieves deep fusion of millimeter-wave radar point clouds and directional audio, completely avoiding the collection of sensitive information such as faces and text, and ensuring inherent privacy and security. It significantly improves data quality by utilizing point cloud super-resolution and speech enhancement technologies, and achieves refined behavior recognition and semantic association by leveraging graph neural networks and cross-modal attention mechanisms. This effectively solves the core pain points of inaccurate perception, modal isolation, and privacy leakage in the monitoring of confidential locations.

[0039] The remaining technical features in the above embodiments can be flexibly selected by those skilled in the art to meet different specific practical needs according to actual circumstances. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims. In the above description, numerous specific details have been set forth to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to implement the present invention. In other instances, to avoid obscuring the present invention, well-known techniques, such as specific construction details, operating conditions, and other technical conditions, have not been specifically described.

[0040] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for identifying multi-modal violation behaviors in a classified space based on millimeter wave radar, characterized in that, The steps are as follows: S1, multi-modal original data synchronous acquisition: configure a perception system composed of a millimeter wave radar and a distributed microphone array, and acquire a low-density point cloud sequence and a multi-channel mixed audio signal in a monitoring area; S2, high-density point cloud sequence reconstruction: input the low-density point cloud sequence acquired in step S1 into a pre-trained point cloud super-resolution model, construct a mapping relationship from low-density point cloud to high-density point cloud through the point cloud super-resolution model, and output a high-density point cloud sequence capable of depicting fine human body posture S3, target speech enhancement and separation: processing the multi-channel mixed audio signal acquired in step S1, combining the real-time orientation information of the target provided by the millimeter wave radar in step S1, suppressing interference and noise, and extracting the pure target speech associated with the radar monitoring target; S4, multi-modal feature extraction: respectively extracting features from the high-density point cloud sequence output in step S2 and the pure target speech output in step S3, obtaining a behavior feature vector representing personnel action posture and an audio feature vector representing speech content; S5, cross-modal spatio-temporal collaborative attention fusion: using a cross-modal collaborative attention mechanism to fuse the behavior feature vector and the audio feature vector obtained in step S4, and after feature correlation enhancement and regularization processing, obtaining a joint feature vector of fused behavior and speech information; S6, rule violation behavior recognition and classification: inputting the joint feature vector output in step S5 into a classifier, calculating the probability distribution of each rule violation behavior class through the classifier, and determining the final rule violation behavior type in the monitoring area according to the maximum probability principle.

2. The method according to claim 1, wherein the method is a method for identifying multi-modal violation behaviors in a classified space based on a millimeter wave radar. In step S1, the low-density point cloud sequence is denoted as wherein T is the total number of time frames, and each frame of point cloud contains N three-dimensional spatial points x , y , z ; the multi-channel mixed audio signal is denoted as wherein C is the number of microphone channels, and L is the length of the audio signal.

3. The method of claim 2, wherein the method is a method for identifying multi-modal violation behaviors in a classified space based on a millimeter wave radar. In step S2, the point cloud super-resolution model is a point cloud super-resolution flow model constructed based on a reversible neural network, and the point cloud super-resolution flow model is trained by using the reversible neural network A deterministic bijective mapping from the latent space to the data space is established, and a simple prior distribution is mapped to a high-density point cloud data distribution The training target is to maximize the log-likelihood of the observed data, and the calculation formula of the log-likelihood is: （1） wherein, is an output high-density point cloud sequence, is a desired operator, is a determinant computation function, is an invertible neural network a high-density point cloud sequence a partial derivative.

4. The method according to claim 3, wherein the method is characterized in that: The specific processing process of step S3 includes: S3.1, blind source separation of multi-channel mixed audio signal, using time-frequency mask method in short-time Fourier transform domain to preliminarily separate multiple sound source signals, the specific formula is: （2） （3） wherein is a short-time Fourier transform function, is a time-frequency domain representation of the mixed audio, is a time-frequency mask of the kth sound source, is a time-frequency domain estimate of the kth sound source, t is a time frame index, and f is a frequency index. S3.2, target azimuth angle provided by the millimeter wave radar at time t , the optimal weight vector is solved by using a linearly constrained minimum variance beamformer , the preliminary separated sound source signals are subjected to adaptive beamforming to suppress residual interference and noise, and pure target speech is obtained ; the optimization problem and the analytical solution formula of the optimal weight vector are respectively （4） （5） wherein is a beamforming weight, is a covariance matrix of noise and interference, is a steering vector.

5. The method of claim 4, wherein the method is a method for identifying multi-modal violation behaviors in a classified space based on a millimeter wave radar. The specific extraction process of step S4 includes: S4.1, constructing each frame of point cloud of the high-density point cloud sequence k a neighbor graph, performing spatial feature encoding on the neighbor graph by a graph attention network k , and inputting the encoded sequence feature into a time convolution network to perform time feature extraction and obtain a behavior feature vector ; wherein a calculation relationship of an output feature of a node i in the graph attention network is （6） （7） （8） wherein, is a set of neighbor nodes of node i, W is a shared linear transformation weight matrix, is a non-linear activation function, is an attention coefficient, is an attention score; S4.2, first convert the pure target speech into a mel-scale spectrogram where F is the number of mel filter banks, is the number of spectrogram time frames, local spectral features are extracted from the mel-scale spectrogram by a deep convolutional neural network, global average pooling is performed on the local spectral features, and the local spectral features are flattened into a time sequence feature vector sequence The time sequence feature vector sequence is then input into a bidirectional gated recurrent unit network to extract context-related features, obtaining an audio feature vector ; wherein the forward propagation relationship of the bidirectional gated recurrent unit network is （9） （10） （11） （12） wherein, is a timing feature vector for time t, is a hidden state for time t-1, is an update gate output, is a reset gate output, is a candidate hidden state, is a hidden state for time t, , , , , , is a weight matrix, , , is a bias vector.

6. The method of claim 5, wherein the method is a method for identifying multi-modal violation behaviors in a classified space based on a millimeter wave radar. The specific fusion process of step S5 includes: S5.1, projecting the behavior feature vector with the audio feature vector respectively, to generate the corresponding query vector, key vector and value vector, denoted as: （13） （14） wherein, wherein, , , is a projection weight matrix of the behavior feature vector, , , is a projection weight matrix of the audio feature vector; S5.2, query vector in terms of behavior feature vectors key vector for audio feature vectors perform attention computation, incorporating value vector for audio feature vectors get audio context enhanced features ; query vector in terms of audio feature vectors key vector for behavior feature vectors perform attention computation, incorporating value vector for behavior feature vectors get behavior context enhanced features , in particular: （15） （16） wherein is the bond vector dimension; S5.3, the behavior context enhanced feature and the original behavior feature vector, the audio context enhanced feature and the original audio feature vector are respectively subjected to residual connection and layer normalization, and the processed feature vectors are spliced to obtain a joint feature vector The specific formula is: （17） （18） （19） wherein, is the layer normalized behavior feature vector, is the layer normalized audio feature vector.

7. The method according to claim 6, wherein: In step S6, the classifier is a multi-layer perception classifier, including several fully connected layers and Dropout layers, and the probability distribution of each rule violation behavior class is calculated and output by using Softmax function, and the calculation formula of the probability distribution is: （20） wherein, denotes a joint feature vector denotes a probability of belonging to a class c, , is a classification weight matrix, , is a classification bias vector, C is a set of all violation classes, is a class label of a behavior; the violation classes include: violation conversation, data theft, boundary operation, photographing, calling, irrelevant behavior.

Images