A millimeter-wave radar human behavior recognition system and method based on global pixel features

Abstract

This invention discloses a millimeter-wave radar human behavior recognition system and method based on global pixel features, belonging to the field of human behavior recognition technology. Its features include: a millimeter-wave radar raw data acquisition module, a millimeter-wave radar raw data transmission module, a millimeter-wave radar raw data preprocessing module, a distance-Doppler feature extraction and classification module based on global pixel features using a time extraction network, and a human behavior information application module, connected sequentially. The millimeter-wave radar raw data acquisition module includes a millimeter-wave radar transmitting unit, a receiving unit, a signal mixing unit, and a power amplification unit. The technical problem this invention aims to solve is to provide a millimeter-wave radar human behavior recognition system and method based on global pixel features, reducing computational load and the number of parameters, improving the recognition accuracy of easily confused behaviors, and achieving effective recognition of complex human behaviors, thereby enhancing the recognition performance of existing models.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition technology, and more specifically, to a millimeter-wave radar human behavior recognition system and method based on global pixel features. Background Technology

[0002] Human Activity Recognition (HAR) is a technology that uses sensors to collect physiological information about human movement and posture, and then uses artificial intelligence algorithms to process and analyze this data to automatically identify human behavior. The core objective of HAR technology is to achieve accurate classification and prediction of different behavioral patterns through the extraction and modeling of human behavior data, thereby providing decision support for intelligent applications.

[0003] HAR (Hardware-Aware Recognition) technology has demonstrated broad application value in various fields in recent years, becoming an important technology driving the development of an intelligent society. In the smart home sector, HAR technology uses sensors to capture real-time activity information of family members, such as monitoring whether children enter dangerous areas and analyzing daily behavior patterns, thereby providing users with personalized health management plans and safety guarantees. In intelligent assisted driving scenarios, HAR technology can monitor the driver's state, such as fatigue or inattention, and identify passenger behavior to enhance driving safety. In the medical and rehabilitation fields, HAR technology can be used to detect dangerous behaviors or abnormal states in patients in real time, such as sudden falls or abnormal body postures, thereby issuing timely alerts to prevent accidents. Furthermore, HAR technology can also remotely and continuously monitor patients' health status, including changes in activity levels, sleep quality, and daily behavior patterns, providing comprehensive data support for medical personnel to help them more efficiently assess and manage patients' health conditions. With the development of artificial intelligence technology, the application of HAR technology in human-computer interaction and virtual reality is rapidly expanding. Through HAR technology, users can use gestures to achieve natural control of smart home devices, game interfaces, or virtual environments, providing a more intuitive way for human-computer interaction. Furthermore, HAR technology can combine motion and facial expressions to infer users' emotional states, giving intelligent assistants or robots more human-like interactive capabilities. In addition, in the education field, HAR systems can monitor students' focus, participation, and emotional changes in real time, helping teachers optimize teaching strategies and improve classroom effectiveness. These applications not only enhance the intelligence of interaction but also bring innovative user experiences to multiple fields.

[0004] Based on the different sensor types and data acquisition methods, HAR technology can be mainly divided into three categories: wearable device-based HAR, video-based HAR, and radio frequency (RF)-based HAR. RF devices include radar and Bluetooth, among others.

[0005] With the advancement and iteration of technologies in related fields, HAR algorithms and systems have entered a phase of rapid development. Currently, research on HAR mainly focuses on classification models based on deep neural networks. Deep neural networks can automatically extract behavioral features, thereby achieving end-to-end behavior recognition and effectively improving recognition accuracy. Convolutional Neural Networks (CNNs) are among the most widely used deep neural networks. CNNs are multi-layered, stacked deep feedforward networks that process input data layer by layer, gradually integrating information and transforming the original data into a high-level feature representation more closely related to the output target, ultimately completing label mapping through a classifier. Compared to CNNs, Recurrent Neural Networks (RNNs) focus more on the temporal features of data and can capture the correlation between temporal features. Therefore, RNNs and their various variants, such as Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs), are widely used in HAR model construction.

[0006] Radar-based HAR technology has received widespread attention in recent years, but many problems still need to be solved. When extracting spatiotemporal features, many existing networks struggle to fully exploit the temporal correlation information in multi-frame range-Doppler images, and the problem of the mixture of temporal and spatial features has not been effectively addressed. In network design, many algorithms prioritize high recognition accuracy at the expense of lightweight model design. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a millimeter-wave radar human behavior recognition system and method based on global pixel features. This method can solve the problem of insufficient extraction of temporal correlation information in multi-frame range-Doppler feature maps. It decomposes the traditional three-dimensional convolution into spatial domain two-dimensional convolution and temporal domain one-dimensional convolution, explicitly distinguishes the two types of features, avoids feature mixing, reduces the amount of computation and the number of parameters, improves the recognition accuracy of easily confused behaviors, and achieves effective recognition of complex human behaviors, thereby improving the recognition performance of existing models.

[0008] The present invention achieves its objective by employing the following technical solution:

[0009] A millimeter-wave radar human behavior recognition system and method based on global pixel features, characterized in that it includes: a millimeter-wave radar raw data acquisition module, a millimeter-wave radar raw data transmission module, a millimeter-wave radar raw data preprocessing module, a time extraction network distance-Doppler feature extraction and classification module based on global pixel features, and a human behavior information application module connected in sequence.

[0010] The millimeter-wave radar raw data acquisition module includes a millimeter-wave radar transmitting unit, a receiving unit, a signal mixing unit, and a power amplification unit;

[0011] The millimeter-wave radar raw data preprocessing module includes a raw data storage unit, a raw data partitioning unit, a chirp matrix reshaping unit, a range-dimensional fast Fourier transform unit, a moving target display unit, a velocity-dimensional fast Fourier transform unit, and a range-Doppler feature map visualization unit.

[0012] The distance-Doppler feature extraction and classification module based on global pixel features includes a spatial downsampling unit, a spatio-Temporal Attention Module Based on Global Pixel Features (STAM-GPF), a spatio-Temporal Feature Separation Module (STFSM), and a fully connected layer classifier.

[0013] As a further limitation of this technical solution, the millimeter-wave radar raw data transmission module supports multiple transmission protocols including Wi-Fi, serial communication, Ethernet, LoRa, and ZigBee.

[0014] As a further limitation of this technical solution, the spatiotemporal attention module based on global pixel features includes a spatial downsampling unit, a self-attention unit, a global mean filtering unit, and a residual linking unit connected in sequence.

[0015] The spatiotemporal feature separation module includes a channel downsampling unit, a spatial feature extraction unit, a temporal feature extraction unit, and a residual connection unit.

[0016] As a further limitation of this technical solution, the fully connected layer classifier of the distance-Doppler feature extraction and classification module of the time extraction network based on global pixel features maps features to specific behavior category probabilities through the normalized exponential function Softmax to identify different behaviors.

[0017] As a further limitation of this technical solution, the human behavior information application module includes smart healthcare, smart transportation, and smart home scenarios.

[0018] A method for human behavior recognition in a millimeter-wave radar system based on global pixel features includes the following steps:

[0019] S1: Millimeter-wave radar raw data acquisition. The millimeter-wave radar in the millimeter-wave radar raw data acquisition module acquires raw data of user behavior. The raw data is in IQ data format and is stored in binary file format.

[0020] S2. Millimeter-wave radar raw data transmission: The raw data acquired by the millimeter-wave radar raw data acquisition module is transmitted to the raw data storage unit of the millimeter-wave radar raw data preprocessing module via an Ethernet cable.

[0021] S3. Millimeter-wave radar raw data preprocessing: The raw data storage unit of the millimeter-wave radar raw data preprocessing module cuts the radar raw data frame by frame with a 25% overlap rate. Each frame contains 128 chirps.

[0022] S4: Construct a distance-Doppler feature extraction and classification module based on global pixel features for temporal extraction network to perform human behavior recognition and classification; input the 6 frames of distance-Doppler feature maps with completed feature preprocessing into the distance-Doppler feature extraction and classification module based on global pixel features in batches, then perform temporal modeling through the spatiotemporal attention module based on global pixel features, and extract temporal and spatial features through the spatiotemporal feature separation module. Finally, the discrimination result is output by the fully connected layer classifier.

[0023] S5: Construct a spatiotemporal attention module based on global pixel features to perform temporal modeling on the input feature map and fully explore the temporal correlation features between the distance-Doppler feature maps of multiple frames;

[0024] S6: Construct a spatiotemporal feature separation module to extract temporal and spatial features from the input feature map, which can reduce the impact of feature confounding;

[0025] S7: Output action recognition results. After multi-dimensional spatiotemporal feature extraction and fusion, the obtained feature data is input into the fully connected layer. The number of hidden units in the fully connected layer is the number of behavior categories contained in the data. The data after the fully connected layer is passed through the Softmax classifier to calculate the probability of the corresponding behavior. The behavior type with the highest probability is the final judgment result.

[0026] S8: Human behavior information display and application. Through the human behavior information application module, the behavior recognition results are visualized and displayed for use in smart healthcare, smart transportation and smart home fields.

[0027] As a further limitation of this technical solution, S3 includes:

[0028] S31: Data framing and chirp matrix reshaping; assuming the original signal sampling rate is... Duration of each frame With an inter-frame overlap rate of 25%, the framing formula is as shown in equation (1):

[0029] (1);

[0030] in: , This refers to the number of sampling points per frame. For the number of frames;

[0031] Will M The frame data is reconstructed into a two-dimensional matrix with dimensions of . Each column is a sampled sequence of a single chirp;

[0032] S32: Distance dimension Fast Fourier Transform; Perform a 128-point Fast Fourier Transform on each column of chirped data, calculated as shown in equation (2):

[0033] (2);

[0034] in: This is the result matrix of the Fast Fourier Transform. The input signal matrix, For rotation factor, , This indicates the sampling point index of the input signal matrix in the distance dimension. Indicates the column index of the input signal matrix. Indicates the distance unit number;

[0035] S33: Matched filtering; Increasing the pulse width will reduce the resolution, while pulse compression achieves both through signal modulation. The matched filter compresses the received wide pulse into a narrow pulse, and its output is shown in equation (3):

[0036] (3);

[0037] in: For the output of the matched filter, The input signal for the matched filter, For the complex conjugate of the transmitted signal, The variable to be integrated represents the time dimension of the signal, and the integration range covers the entire duration of the signal. This is a delay variable, representing the time offset between the received signal and the transmitted signal;

[0038] S34: Moving target display; The input matrix is ​​filtered using a fourth-order Butterworth filter, and the transfer function of the fourth-order Butterworth filter is shown in equation (4):

[0039] (4);

[0040] in: The transfer function of a fourth-order Butterworth filter. and For coefficients, For delay operators;

[0041] S35: Velocity dimension Fast Fourier Transform; Perform a 256-point Fast Fourier Transform on each row distance cell, calculated as shown in equation (5):

[0042] (5);

[0043] in: This is the result matrix of the Fast Fourier Transform. The input signal matrix, , Indicates the distance cell number. Indicates the pulse number. For the summation variable.

[0044] As a further limitation of this technical solution, S5 includes:

[0045] S51: Spatiotemporal attention module based on global pixel features; performs temporal modeling on the input feature map to fully explore the temporal correlation features between distance-Doppler feature maps of multiple frames;

[0046] The dimension of the input feature map is ,in It is the number of channels. It is the time dimension. and These are the space's height and width, respectively.

[0047] S52: Spatial downsampling; the input feature map undergoes spatial downsampling to compress the spatial dimension of the feature map, reducing computational complexity. The kernel size of the 3D convolutional layer is... The stride size is 2, and the filling method is a 3D convolution module. The calculation method of the 3D convolution module is shown in equation (6):

[0048] (6);

[0049] in: This represents the output feature map of a 3D convolution. This represents the input feature map for a 3D convolution. Represents a three-dimensional convolution function;

[0050] S53: Self-attention mechanism; The self-attention mechanism allows the model to dynamically focus on the importance of different parts when processing the input sequence;

[0051] Dimension is The input sequence is passed through three linear transformation layers to generate the query matrix. Key matrix Sum matrix The calculation method is shown in equation (7):

[0052] (7);

[0053] in: This represents the output feature map of a 3D convolution;

[0054] The attention score matrix can be obtained through dot product operations. The calculation method is shown in equation (8):

[0055] (8);

[0056] in: Represents the feature dimension of the key matrix. Scaling factor softmax () represents the Softmax function;

[0057] Finally, the context vector is obtained through a weighted summation operation. The calculation method is shown in equation (9):

[0058] (9);

[0059] S54: Global Mean Filtering; First, calculate the interaction between any two locations to directly capture long-range dependencies, without being limited to adjacent points, using the correlation function. The calculation method is shown in equation (10):

[0060] (10);

[0061] in: Indicates the inner product. and These are the position matrices;

[0062] S55: Feature weighting and residual connection; via attention score matrix A Log-value matrix V Weighting, aggregating global context information, and residual connections combine the output feature maps of 3D convolution. It is directly superimposed on the global features to avoid information loss;

[0063] The calculation method for the residual connection result is shown in Equation (11):

[0064] (11).

[0065] As a further limitation of this technical solution, S6 includes:

[0066] S61: Spatiotemporal feature separation module; decomposes the traditional three-dimensional convolution kernel into a combination of convolution kernels with spatial domain two-dimensional convolution and temporal domain one-dimensional convolution;

[0067] S62: Channel compression; First, channel compression is performed on the input feature map to reduce computational complexity by pruning redundant channels;

[0068] S63: Spatial feature extraction; Spatial features of different dimensions are extracted through two parallel convolutional branches. Branch 1 uses a 1×3×3 convolutional kernel to focus on extracting spatial relationships in the horizontal-vertical direction. Branch 2 uses a 1×1×1 convolutional kernel to achieve cross-channel interaction. The calculation method of the two branches is shown in Equation (12), and the calculation method of the feature fusion result is shown in Equation (13).

[0069] (12);

[0070] (13);

[0071] in: Extracting local spatial features, Pay attention to channel interaction. For spatial convolution, It is a 1×1×1 convolution;

[0072] S64: Temporal feature extraction; 3×1×1 convolutional kernels cover 3 consecutive frames, calculate weighted sums to capture short-term motion patterns, slide only on the time axis, and keep the spatial dimension processed independently.

[0073] Compared with related technologies, the millimeter-wave radar human behavior recognition system and method based on global pixel features provided by this invention has the following beneficial effects:

[0074] The technical solution provided by this invention includes a millimeter-wave radar raw data acquisition module, a millimeter-wave radar raw data transmission module, a millimeter-wave radar raw data preprocessing module, a TEN-GPF range-Doppler feature extraction and classification module, and a human behavior information application module. The TEN-GPF range-Doppler feature extraction and classification module includes a spatial downsampling unit, a spatiotemporal attention module based on global pixel features, and an STFSM spatiotemporal feature separation module. This system better utilizes the rich temporal and spatial information in multi-frame range-Doppler feature maps, improving the recognition accuracy of easily confused behaviors and enhancing the recognition performance of existing models. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the structure of the present invention.

[0076] Figure 2 This is a schematic diagram of the structure of the TEN-GPF of the present invention.

[0077] Figure 3 This is a schematic diagram of the spatiotemporal attention module based on global pixel features of the present invention.

[0078] Figure 4 This is a schematic diagram of the STFSM spatiotemporal feature separation module of the present invention. Detailed Implementation

[0079] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0080] A millimeter-wave radar human behavior recognition system based on global pixel features, comprising:

[0081] The system consists of a millimeter-wave radar raw data acquisition module, a millimeter-wave radar raw data transmission module, a millimeter-wave radar raw data preprocessing module, a time extraction network based on global pixel feature (TEN-GPF) distance-Doppler feature extraction and classification module, and a human behavior information application module, which are connected in sequence.

[0082] The millimeter-wave radar raw data acquisition module includes a millimeter-wave radar transmitting unit, a receiving unit, a signal mixing unit, and a power amplification unit;

[0083] In this embodiment of the invention, the millimeter-wave radar raw data acquisition module employs frequency-modulated continuous wave (FMCV) technology to achieve high-precision detection. The transmitting unit is based on the Texas Instruments IWR1843 Boost radar system, operating in the 77-80 GHz frequency band with a bandwidth of 2998.2 MHz. Signal transmission and reception are performed through an antenna array consisting of one transmitting antenna and one receiving antenna (40 dB gain). The system sampling rate is 10000 kps, each frame contains 128 chirped signals, and the frame period is set to 500 ms.

[0084] In this embodiment of the invention, after the receiving unit acquires the target reflection signal, the signal mixing unit performs key processing: mixing the received signal with the original transmitted signal and its orthogonal signal respectively to generate an orthogonal intermediate frequency signal containing target distance and velocity information.

[0085] In this embodiment of the invention, the power amplification unit adopts a three-stage amplification structure: a pre-stage low-noise amplifier provides 20dB gain to compensate for transmission loss, an intermediate frequency amplifier provides 30dB gain to optimize the signal-to-noise ratio, and a final-stage programmable gain amplifier achieves 40dB dynamic range adjustment. The system detection range is set to a 5m × 4m rectangular area in front of the radar, with an installation height of 1.2m, ensuring complete coverage of human activities.

[0086] In this embodiment of the invention, experimental data collection shows that the system can clearly identify six types of daily activities, including bending over, falling, and jogging, within a range of 1-5m. Through distance-dimensional FFT processing of 128 sampling points, combined with a 2998.2MHz bandwidth, the theoretical distance resolution reaches 5cm.

[0087] The millimeter-wave radar raw data preprocessing module includes a raw data storage unit, a raw data partitioning unit, a chirp matrix reshaping unit, a range-dimensional fast Fourier transform unit, a moving target display unit, a velocity-dimensional fast Fourier transform unit, and a range-Doppler feature map visualization unit.

[0088] The raw data partitioning unit partitions the raw data according to a specific period; the chirp matrix reshaping unit reassembles the raw radar signal into a two-dimensional matrix according to the chirp sequence; the range-dimensional fast Fourier transform unit extracts the range information of the signal through fast Fourier transform operation; the velocity-dimensional fast Fourier transform unit extracts the velocity information of the signal through fast Fourier transform operation; the range-Doppler feature map visualization unit generates a range-velocity feature map containing target motion information; and the matched filtering unit uses pulse compression technology to improve the resolution of the radar signal, thereby achieving more accurate ranging and velocity measurement.

[0089] In this embodiment of the invention, the data preprocessing module converts the continuous signal into discrete frames suitable for feature extraction by segmenting the original radar signal. The original ADC sampling data first enters the segmentation processing unit, which divides it into segments of 128 chirped periods, each containing 128 sampling points. The segmented data is then reassembled by the chirped matrix reshaping unit to form a two-dimensional matrix. This reassembly method effectively preserves the time-frequency characteristics of the radar signal, establishing a data structure foundation for subsequent processing. Experiments show that using a Hamming window for windowing can effectively alleviate the spectral leakage problem. The Fast Fourier Transform (FFT) unit performs spectral analysis in both the range and velocity dimensions to extract the target's range and velocity information. The range-dimensional Fourier Transform converts the time-domain sampling sequence into the frequency domain, with the peak frequency corresponding to the target's range. Different target ranges result in different echo delays, leading to differences in the intermediate frequency. The FFT decomposes the mixed time-domain signal into discrete frequency points, each corresponding to a specific range gate, achieving multi-target range resolution. The phase difference of a static target is zero (zero-frequency component), while a moving target produces a non-zero frequency shift. The range-Doppler feature map formed by the velocity-dimensional fast Fourier transform can visually distinguish between stationary clutter and moving targets. The moving target display unit effectively suppresses static background interference through digital filtering technology.

[0090] The distance-Doppler feature extraction and classification module based on global pixel features includes a spatial downsampling unit, a spatio-Temporal Attention Module Based on Global Pixel Features (STAM-GPF), a spatio-Temporal Feature Separation Module (STFSM), and a fully connected layer classifier.

[0091] In this embodiment of the invention, the spatiotemporal attention module based on global pixel features performs channel dimensionality reduction on the input features through 1×1 convolution, projecting high-dimensional features onto a low-dimensional latent space. This compression preserves key spatial information while reducing subsequent computational burden. The feature matrix is ​​reshaped into a two-dimensional form, laying the foundation for pixel-level correlation analysis. Simultaneously, a self-attention mechanism is employed to process features in parallel: the query matrix captures the importance of target features, the key matrix establishes cross-spatiotemporal correlations, the value matrix retains the original feature content, attention weights are calculated through scaling dot products, and temporal position encoding is introduced to enable the model to perceive temporal order. Temporal modeling is performed by calculating the similarity between each pixel, and finally, the original features and attention features are fused through residual connections. Layer normalization stabilizes the training process, and DropPath technology prevents overfitting. This design preserves low-level detail information while enhancing high-level semantic features. This mechanism, through spatiotemporal joint modeling and dynamic feature selection, significantly improves feature discrimination capabilities in complex scenarios.

[0092] In this embodiment of the invention, the STFSM spatiotemporal feature separation module decouples the traditional 3D convolutional kernel (3×3×3) into two independent processing paths: the spatial domain and the temporal domain.

[0093] Spatial processing path: A 1×3×3 two-dimensional spatial convolution kernel is used to focus on extracting spatial features in the image plane, capturing spatial relationships in the horizontal and vertical directions, and effectively extracting geometric features such as edges and textures; at the same time, a 1×1×1 convolution kernel is added to realize cross-channel feature recombination and enhance important feature channels through linear combination.

[0094] Temporal processing path: A 3×1×1 one-dimensional temporal convolution kernel is used. The 3×1×1 convolution kernel slides along the time axis to model the dynamic changes between three consecutive frames, keeping the spatial dimension processed independently and avoiding mutual interference between spatiotemporal features to focus on analyzing the motion features of the temporal dimension.

[0095] By decomposing the convolution kernel, the number of parameters is greatly reduced, and the transfer and use of pre-trained weights of two-dimensional convolution are supported. In multi-scale feature fusion, the spatial dual-branch captures local details and global context respectively, the temporal path establishes short-term motion dependencies, and the residual connection preserves the original feature information.

[0096] Spatial downsampling units compress feature map size through convolutional layers.

[0097] The millimeter-wave radar raw data transmission module supports multiple transmission protocols including Wi-Fi, serial communication, Ethernet, LoRa, and ZigBee.

[0098] In this embodiment of the invention, the data transmission module adopts a multi-protocol compatible design, allowing the selection of the optimal transmission method based on different application scenarios. In the local data transmission scenario, the system uses the 1000Base-T gigabit Ethernet protocol for high-speed transmission.

[0099] The spatiotemporal attention module based on global pixel features includes a spatial downsampling unit, a self-attention unit, a global mean filtering unit, and a residual connection unit connected in sequence; the self-attention unit calculates the global correlation weights of the spatiotemporal features; the global mean filtering unit smooths the attention weights; and the residual connection unit preserves the original features to avoid gradient vanishing.

[0100] The spatiotemporal feature separation module includes a channel downsampling unit, a spatial feature extraction unit, a temporal feature extraction unit, and a residual connection unit. The channel downsampling unit can reduce the channel dimension. The spatial feature extraction unit separates spatial dimension features through two-dimensional spatial convolution. The temporal feature extraction unit uses one-dimensional temporal convolution to capture dynamic changes in behavior. The residual connection unit realizes cross-layer feature reuse.

[0101] The fully connected layer classifier of the distance-Doppler feature extraction and classification module of the time extraction network based on global pixel features maps features to specific behavior category probabilities through the normalized exponential function Softmax to identify different behaviors.

[0102] The human behavior information application module includes smart healthcare, smart transportation, and smart home scenarios.

[0103] At the application level, this system can be widely used in multiple smart scenarios. In the field of medical monitoring, it can detect falls by analyzing the micro-motion characteristics of the human body; in intelligent transportation scenarios, it can predict pedestrian behavior using motion trajectories of continuous frames; and in smart homes, it can achieve contactless control by recognizing specific gestures. The system adopts an end-to-end deep learning architecture, directly mapping raw radar signals to behavior categories, greatly simplifying the traditional processing flow.

[0104] The innovation of this invention is mainly reflected in three aspects: First, the proposed spatiotemporal attention mechanism breaks through the limitations of traditional local receptive fields, realizing true global feature modeling; second, the spatiotemporal separation feature extraction architecture effectively solves the problem of unclear mixed feature representation; finally, the entire system uses millimeter-wave radar as the sensing means, achieving all-weather, high-precision behavior recognition while protecting privacy. These technological innovations significantly improve the system's recognition performance in complex scenarios.

[0105] A method for human behavior recognition in a millimeter-wave radar system based on global pixel features includes the following steps:

[0106] S1: Millimeter-wave radar raw data acquisition. The millimeter-wave radar in the millimeter-wave radar raw data acquisition module acquires raw data of user behavior. The raw data is in IQ data format and is stored in binary file format.

[0107] S2. Millimeter-wave radar raw data transmission: The raw data acquired by the millimeter-wave radar raw data acquisition module is transmitted to the raw data storage unit of the millimeter-wave radar raw data preprocessing module via an Ethernet cable.

[0108] S3. Millimeter-wave radar raw data preprocessing: The raw data storage unit of the millimeter-wave radar raw data preprocessing module cuts the radar raw data frame by frame with a 25% overlap rate. Each frame contains 128 chirps.

[0109] The chirp matrix reshaping unit reconstructs the chirp matrix into a two-dimensional matrix, with the two dimensions being chirp and frame, respectively.

[0110] The distance dimension Fast Fourier Transform unit performs Fast Fourier Transform on the chirp matrix along the fast time dimension to extract the target's distance information. The Fast Fourier Transform result will show the intensity of different frequency components. Since the frequency is proportional to the target distance, this can be directly converted into distance information.

[0111] The matched filtering unit uses pulse compression technology to enhance the detection capability of useful signals and improve the resolution of radar signals, thereby achieving more accurate ranging and speed measurement.

[0112] The moving target display unit separates the characteristics of high-frequency dynamic targets from the raw data acquired by the millimeter-wave radar through a fourth-order Butterworth filter, while suppressing static or slowly changing background clutter.

[0113] The velocity dimension Fast Fourier Transform unit performs Fast Fourier Transform along the slow time dimension to extract the velocity information of the target. The velocity of the target can be calculated by analyzing the frequency shift in the Fast Fourier Transform result.

[0114] S3 includes:

[0115] S31: Data framing and chirp matrix reshaping; assuming the original signal sampling rate is... Duration of each frame With an inter-frame overlap rate of 25%, the framing formula is as shown in equation (1):

[0116] (1);

[0117] in: , This refers to the number of sampling points per frame. For the number of frames;

[0118] Will M The frame data is reconstructed into a two-dimensional matrix with dimensions of . Each column is a sampled sequence of a single chirp;

[0119] S32: Distance dimension Fast Fourier Transform; Perform a 128-point Fast Fourier Transform on each column of chirped data, calculated as shown in equation (2):

[0120] (2);

[0121] in: This is the result matrix of the Fast Fourier Transform. The input signal matrix, For rotation factor, , This indicates the sampling point index of the input signal matrix in the distance dimension. Indicates the column index of the input signal matrix. Indicates the distance unit number;

[0122] S33: Matched filtering; Increasing the pulse width will reduce the resolution, while pulse compression achieves both through signal modulation. The matched filter compresses the received wide pulse into a narrow pulse, and its output is shown in equation (3):

[0123] (3);

[0124] in: For the output of the matched filter, The input signal for the matched filter, For the complex conjugate of the transmitted signal, The variable to be integrated represents the time dimension of the signal, and the integration range covers the entire duration of the signal. This is a delay variable, representing the time offset between the received signal and the transmitted signal;

[0125] S34: Moving target display; The input matrix is ​​filtered using a fourth-order Butterworth filter, and the transfer function of the fourth-order Butterworth filter is shown in equation (4):

[0126] (4);

[0127] in: The transfer function of a fourth-order Butterworth filter. and For coefficients, For delay operators;

[0128] S35: Velocity dimension Fast Fourier Transform; Perform a 256-point Fast Fourier Transform on each row distance cell, calculated as shown in equation (5):

[0129] (5);

[0130] in: This is the result matrix of the Fast Fourier Transform. The input signal matrix, , Indicates the distance cell number. Indicates the pulse number. For the summation variable.

[0131] S4: Construct a distance-Doppler feature extraction and classification module based on global pixel features for time extraction network to perform human behavior recognition and classification; input the 6 frames of distance-Doppler feature maps with completed feature preprocessing into the distance-Doppler feature extraction and classification module based on global pixel features in batches, and then perform time modeling through the spatiotemporal attention module based on global pixel features, and display the extraction of time features and spatial features through the spatiotemporal feature separation module. Finally, the discrimination result is output by the fully connected layer classifier.

[0132] S5: Construct a spatiotemporal attention module based on global pixel features to perform temporal modeling on the input feature map and fully explore the temporal correlation features between the distance-Doppler feature maps of multiple frames;

[0133] The input feature map is used to perform dimensionality reduction of spatial features in the spatial downsampling unit, which reduces the amount of subsequent computation while performing feature extraction;

[0134] The self-attention unit calculates the spatiotemporal dependencies between global pixels and establishes cross-frame feature associations through the Query-Key-Value mechanism.

[0135] The global mean filtering unit calculates the correlation of global pixels through matrix multiplication, which can model both temporal and spatial information.

[0136] The residual connection unit adds the original input to the global features output by the global mean filter unit, enabling collaborative modeling of local details and global context.

[0137] S5 includes:

[0138] S51: Spatiotemporal attention module based on global pixel features; performs temporal modeling on the input feature map to fully explore the temporal correlation features between distance-Doppler feature maps of multiple frames; for a certain pixel in the image, its return value is obtained by weighted summation of all other pixels, which means that each pixel is associated with all other pixels in the image, thus realizing global attention;

[0139] The dimension of the input feature map is ,in It is the number of channels. It is the time dimension. and These are the space's height and width, respectively.

[0140] S52: Spatial downsampling; the input feature map undergoes spatial downsampling to compress the spatial dimension of the feature map, reducing computational complexity. The kernel size of the 3D convolutional layer is... The stride size is 2, and the filling method is a three-dimensional convolutional module (SAME). The calculation method of the three-dimensional convolutional module is shown in equation (6):

[0141] (6);

[0142] in: This represents the output feature map of a 3D convolution. This represents the input feature map for a 3D convolution. Represents a three-dimensional convolution function;

[0143] Original feature map H × W The computational complexity of attention is After downsampling, it drops to This can greatly reduce the amount of subsequent calculations;

[0144] S53: Self-attention mechanism; The self-attention mechanism allows the model to dynamically focus on the importance of different parts when processing the input sequence; Compared with recurrent neural networks, the self-attention mechanism can compute the output of each time step in parallel, thereby significantly improving the training speed and efficiency of the model;

[0145] Dimension is The input sequence is passed through three linear transformation layers to generate the query matrix. Key matrix Sum matrix The calculation method is shown in equation (7):

[0146] (7);

[0147] in: This represents the output feature map of a 3D convolution;

[0148] The attention score matrix can be obtained through dot product operations. The calculation method is shown in equation (8):

[0149] (8);

[0150] in: Represents the feature dimension of the key matrix. This scaling factor helps prevent the dot product from becoming too large, which could lead to gradient instability. softmax () represents the Softmax function;

[0151] Finally, the context vector is obtained through a weighted summation operation. The calculation method is shown in equation (9):

[0152] (9);

[0153] In radar-based human behavior recognition, the self-attention mechanism captures long-distance dependencies by calculating the correlation between different locations in the feature map, enhances attention to key areas, and improves classification accuracy.

[0154] S54: Global Mean Filtering; First, it calculates the interaction between any two locations to directly capture long-range dependencies, rather than being limited to neighboring points. This is equivalent to constructing a convolutional kernel the same size as the feature map, thus preserving more information and the correlation function. The calculation method is shown in equation (10):

[0155] (10);

[0156] in: Indicates the inner product. and These are the position matrices; they measure the similarity between two positions; the exponential function will... The inner product result is mapped to the positive real number space to ensure that the correlation is non-negative. Similar to the Gaussian kernel function, it gives the similarity calculation a smooth decay characteristic and can better suppress noise.

[0157] Traditional convolution can only capture local receptive fields (such as 3×3 neighborhoods), while Equation (10) calculates the correlation between any two points globally;

[0158] S55: Feature weighting and residual connection; via attention score matrix A Log-value matrix V Weighting, aggregating global context information, and residual connections combine the output feature maps of 3D convolution. It is directly superimposed on the global features to avoid information loss;

[0159] The calculation method for the residual connection result is shown in Equation (11):

[0160] (11).

[0161] S6: Construct a spatiotemporal feature separation module to extract temporal and spatial features from the input feature map, which can reduce the impact of feature confounding;

[0162] The input feature map undergoes dimensionality reduction of the number of channels in the channel downsampling unit, which significantly reduces the number of parameters and computation in subsequent layers. By reducing redundant feature channels, key information is preserved, and the generalization ability of the model is improved.

[0163] The spatial feature extraction unit uses two-dimensional spatial convolution with kernel sizes of 1×3×3 and 1×1×1 to extract spatial features, respectively.

[0164] The temporal feature extraction unit uses a one-dimensional temporal convolution with a kernel size of 3×1×1 to extract temporal features.

[0165] Residual connection units extract high-level semantic features, which is more conducive to capturing global context;

[0166] S6 includes:

[0167] S61: Spatiotemporal feature separation module; it separates traditional 3D convolution kernels... The combination of convolution kernels, which are decomposed into two-dimensional convolutions in the spatial domain and one-dimensional convolutions in the temporal domain, reduces the computational cost and training parameters of the model. This not only reduces the computational cost and the number of parameters, but also makes full use of the pre-training advantages of two-dimensional convolutional networks.

[0168] S62: Channel compression; First, channel compression is performed on the input feature map to reduce computational complexity by pruning redundant channels;

[0169] S63: Spatial feature extraction; Spatial features of different dimensions are extracted through two parallel convolutional branches. Branch 1 uses a 1×3×3 convolutional kernel, which focuses on extracting the spatial relationship in the horizontal-vertical direction and can focus on capturing the geometric features such as edges and textures of local areas of the image. Branch 2 uses a 1×1×1 convolutional kernel to achieve cross-channel interaction, linearly combine the channel dimensions, and enhance or suppress the feature response of specific channels. The calculation method of the two branches is shown in Equation (12), and the calculation method of the feature fusion result is shown in Equation (13).

[0170] (12);

[0171] (13);

[0172] in: Extracting local spatial features, Pay attention to channel interaction. For spatial convolution, It is a 1×1×1 convolution;

[0173] S64: Temporal feature extraction; 3×1×1 convolutional kernels cover 3 consecutive frames, calculate weighted sums to capture short-term motion patterns, slide only on the time axis, and keep the spatial dimension processed independently, thus reducing the impact of feature confounding to a certain extent.

[0174] S7: Output action recognition results. After multi-dimensional spatiotemporal feature extraction and fusion, the obtained feature data is input into the fully connected layer. The number of hidden units in the fully connected layer is the number of behavior categories contained in the data. The data after the fully connected layer is passed through the Softmax classifier to calculate the probability of the corresponding behavior. The behavior type with the highest probability is the final judgment result.

[0175] S8: Human behavior information display and application. Through the human behavior information application module, the behavior recognition results are visualized and displayed for use in fields such as smart healthcare, smart transportation, and smart homes.

[0176] This invention achieves human behavior classification and recognition based on millimeter-wave radar by collecting and analyzing radar data, making up for the shortcomings of video-based behavior recognition, such as high cost, susceptibility to interference, and poor privacy. Through attention mechanism and residual structure, it effectively improves the effectiveness, comprehensiveness and ability to capture complex dynamics of feature extraction. Compared with mainstream and latest models, it has obvious advantages in adaptability, reliability and practicality.

[0177] The technical solution provided by this invention includes a millimeter-wave radar raw data acquisition module, a millimeter-wave radar raw data transmission module, a millimeter-wave radar raw data preprocessing module, a TEN-GPF range-Doppler feature extraction and classification module, and a human behavior information application module. The spatiotemporal attention module based on global pixel features includes a spatial downsampling unit, a self-attention unit, a global mean filtering unit, and a residual linking unit connected in sequence. The spatiotemporal feature separation module includes channel downsampling, a spatial feature extraction unit, a temporal feature extraction unit, and a residual linking unit. This system better utilizes the temporal features contained in human behavior data, improves the recognition accuracy of easily confused behaviors, and enhances the recognition performance of existing models.

[0178] The various steps in the embodiments of the present invention can be performed by electronic devices. These electronic devices include, but are not limited to, mobile phones, tablet computers, portable PCs, and desktop computers.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0180] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A recognition method for a millimeter-wave radar human behavior recognition system based on global pixel features, characterized in that, Includes the following steps: S1: Millimeter-wave radar raw data acquisition. Raw data of user behavior is collected through millimeter-wave radar. The raw data is in IQ data format and stored in binary file format. S2. Millimeter-wave radar raw data transmission: The raw data acquired by the millimeter-wave radar raw data acquisition module is transmitted to the raw data storage unit of the millimeter-wave radar raw data preprocessing module via an Ethernet cable. S3. Preprocessing of raw millimeter-wave radar data, including data framing and chirp matrix reshaping, range-dimensional fast Fourier transform, matched filtering, moving target indication, and velocity-dimensional fast Fourier transform. The raw data storage unit segments the raw radar data into frames with a 25% overlap rate, with each frame containing 128 chirps; the chirp matrix is ​​reconstructed into a two-dimensional matrix, with the two dimensions being chirp and frame, respectively. Range information of the target is extracted by performing a Fast Fourier Transform (FFT) along the fast time dimension on the chirp matrix. The FFT result will show the intensity of different frequency components. Since the frequency is proportional to the target distance, it can be directly converted into range information. Pulse compression technology is used to enhance the detection capability of useful signals and improve the resolution of radar signals, thereby achieving more accurate range and velocity measurement. The characteristics of high-frequency dynamic targets are separated from the raw data acquired by millimeter-wave radar through a fourth-order Butterworth filter, while suppressing stationary or slowly changing background clutter. The velocity information of the target is extracted by performing a FFT along the slow time dimension, and the target velocity is calculated by analyzing the frequency shift in the FFT result. S4: Construct a distance-Doppler feature extraction and classification module based on global pixel features for temporal extraction network to perform human behavior recognition and classification; input the 6 frames of distance-Doppler feature maps with completed feature preprocessing into the distance-Doppler feature extraction and classification module based on global pixel features in batches, then perform temporal modeling through the spatiotemporal attention module based on global pixel features, and extract temporal and spatial features through the spatiotemporal feature separation module. Finally, the discrimination result is output by the fully connected layer classifier. S5: Output action recognition results. After multi-dimensional spatiotemporal feature extraction and fusion, the obtained feature data is input into the fully connected layer. The number of hidden units in the fully connected layer is the number of behavior categories contained in the data. The data after the fully connected layer is passed through the Softmax classifier to calculate the probability of the corresponding behavior. The behavior type with the highest probability is the final judgment result. S6: Human behavior information display and application. Through the human behavior information application module, the behavior recognition results are visualized and displayed for use in smart healthcare, smart transportation and smart home fields. S4 includes: S41: Perform temporal modeling on the input feature map to fully explore the temporal correlation features between the distance-Doppler feature maps of multiple frames; The dimension of the input feature map is ,in It is the number of channels. It is the time dimension. and These are the space's height and width, respectively. S42: Spatial downsampling; the input feature map undergoes spatial downsampling to compress the spatial dimension of the feature map, reducing computational complexity. The kernel size of the 3D convolutional layer is... The stride size is 2, and the padding method is a 3D convolution module. The calculation method of the 3D convolution module is shown in the following formula: ； in: This represents the output feature map of a 3D convolution. This represents the input feature map for a 3D convolution. Represents a three-dimensional convolution function; S43: Self-attention mechanism; The self-attention mechanism allows the model to dynamically focus on the importance of different parts when processing the input sequence; Dimension is The input sequence is passed through three linear transformation layers to generate the query matrix. Key matrix Sum matrix The calculation method is shown in the following formula: ； in: This represents the output feature map of a 3D convolution; The attention score matrix can be obtained through dot product operations. The calculation method is shown in the following formula: ； in: Represents the feature dimension of the key matrix. Scaling factor softmax () represents the Softmax function; Finally, the context vector is obtained through a weighted summation operation. The calculation method is shown in the following formula: ； S44: Global Mean Filtering; First, calculate the interaction between any two locations to directly capture long-range dependencies, without being limited to adjacent points, using the correlation function. The calculation method is shown in the following formula: ； in: Indicates the inner product. and These are the position matrices; S45: Feature weighting and residual connection; via attention score matrix A Log-value matrix V Weighting, aggregating global context information, and residual connections combine the output feature maps of 3D convolution. It is directly superimposed on the global features to avoid information loss; The residual connection result is calculated as follows: ； S4 further includes: The traditional three-dimensional convolution kernel is decomposed into a combination of two-dimensional convolution kernels in the spatial domain and one-dimensional convolution kernels in the temporal domain; Channel compression; First, channel compression is performed on the input feature map by pruning redundant channels to reduce computational complexity; Spatial feature extraction: Spatial features of different dimensions are extracted through two parallel convolutional branches. Branch 1 uses a 1×3×3 convolutional kernel to focus on extracting spatial relationships in the horizontal-vertical direction, while branch 2 uses a 1×1×1 convolutional kernel to achieve cross-channel interaction. The calculation methods of the two branches are shown in the following formula: ； The feature fusion result is calculated as follows: ； in: Extracting local spatial features, Pay attention to channel interaction. For spatial convolution, It is a 1×1×1 convolution; Temporal feature extraction: 3×1×1 convolutional kernels cover 3 consecutive frames, and weighted sums are calculated to capture short-term motion patterns. The kernels slide only on the temporal axis, while the spatial dimension is processed independently.

2. The identification method according to claim 1, characterized in that: S3 includes: S31: Assume the original signal sampling rate is... Duration of each frame With an inter-frame overlap rate of 25%, the framing formula is as follows: ； in: , This refers to the number of sampling points per frame. For the number of frames; Will M The frame data is reconstructed into a two-dimensional matrix with dimensions of . Each column is a sampled sequence of a single chirp; S32: Distance-dimensional Fast Fourier Transform; Perform a 128-point Fast Fourier Transform on each column of chirped data, calculated as shown in the following formula: ； in: This is the result matrix of the Fast Fourier Transform. The input signal matrix, For rotation factor, , This indicates the sampling point index of the input signal matrix in the distance dimension. Indicates the column index of the input signal matrix. Indicates the distance unit number; S33: Matched filtering; Increasing the pulse width reduces resolution, while pulse compression achieves both through signal modulation. The matched filter compresses the received wide pulse into a narrow pulse, and its output is shown in the following formula: ； in: For the output of the matched filter, The input signal for the matched filter, For the complex conjugate of the transmitted signal, The variable to be integrated represents the time dimension of the signal, and the integration range covers the entire duration of the signal. This is a delay variable, representing the time offset between the received signal and the transmitted signal; S34: Moving target display; A fourth-order Butterworth filter is used to filter the input matrix. The transfer function of the fourth-order Butterworth filter is shown in the following equation: ； in: The transfer function of a fourth-order Butterworth filter. and For coefficients, For delay operators; S35: Velocity-dimensional Fast Fourier Transform; Perform a 256-point Fast Fourier Transform on each row of distance cells, calculated as shown in the following formula: ； in: This is the result matrix of the Fast Fourier Transform. The input signal matrix, , Indicates the distance cell number. Indicates the pulse number. For the summation variable.

3. A millimeter-wave radar human behavior recognition system based on global pixel features, utilizing the millimeter-wave radar human behavior recognition method based on global pixel features according to any one of claims 1-2, characterized in that: include: The system consists of a millimeter-wave radar raw data acquisition module, a millimeter-wave radar raw data transmission module, a millimeter-wave radar raw data preprocessing module, a distance-Doppler feature extraction and classification module based on global pixel features, and a human behavior information application module, which are connected in sequence. The millimeter-wave radar raw data acquisition module includes a millimeter-wave radar transmitting unit, a receiving unit, a signal mixing unit, and a power amplification unit; The millimeter-wave radar raw data preprocessing module includes a raw data storage unit, a raw data partitioning unit, a chirp matrix reshaping unit, a range-dimensional fast Fourier transform unit, a moving target display unit, a velocity-dimensional fast Fourier transform unit, and a range-Doppler feature map visualization unit. The time extraction network distance-Doppler feature extraction and classification module based on global pixel features includes a spatial downsampling unit, a spatiotemporal attention module based on global pixel features, a spatiotemporal feature separation module, and a fully connected layer classifier.

4. The identification system according to claim 3, characterized in that: The millimeter-wave radar raw data transmission module supports multiple transmission protocols including Wi-Fi, serial communication, Ethernet, LoRa, and ZigBee.

5. The identification system according to claim 3, characterized in that: The spatiotemporal attention module based on global pixel features includes a spatial downsampling unit, a self-attention unit, a global mean filtering unit, and a residual connection unit connected in sequence. The spatiotemporal feature separation module includes a channel downsampling unit, a spatial feature extraction unit, a temporal feature extraction unit, and a residual connection unit.

6. The identification system according to claim 3, characterized in that: The fully connected layer classifier of the distance-Doppler feature extraction and classification module of the time extraction network based on global pixel features maps features to specific behavior category probabilities through the normalized exponential function Softmax to identify different behaviors.

7. The identification system according to claim 3, characterized in that: The human behavior information application module includes smart healthcare, smart transportation, and smart home scenarios.

Images