Continuous human action recognition method based on point cloud data and range-doppler spectrogram

Abstract

The application provides a continuous human action recognition method based on point cloud data and range Doppler spectrograms, comprising the following steps: collecting echo containing multiple continuous human actions to obtain sampled intermediate frequency signals, and obtaining corresponding range Doppler spectrograms and point cloud data after processing; building a PVT-BiLSTM network, wherein the PVT-BiLSTM network comprises a Pointnet-BiLSTM module, a ViT module and a fusion module; dividing the range Doppler spectrograms and the point cloud data into a training set and a verification set, and inputting them into the PVT-BiLSTM network for training and verification to obtain a trained PVT-BiLSTM network model, and then using the trained PVT-BiLSTM network model to recognize continuous human actions. The method can significantly improve the recognition performance of continuous human actions.

Description

Technical Field

[0001] This invention belongs to the field of radar target recognition, specifically relating to a method for continuous human motion recognition based on point cloud data and range Doppler spectrum. Background Technology

[0002] Human motion recognition technology holds significant research value and application prospects in fields such as smart homes, security monitoring, and human-computer interaction. Currently, sensors suitable for human motion recognition include cameras, wearable devices, and radar. Cameras are sensitive to light and environmental conditions and pose a risk of privacy breaches. Wearable devices may cause discomfort and burden for individuals. Compared to other sensors, radar detects and senses targets by emitting electromagnetic waves, offering advantages such as all-weather operation, non-contact operation, immunity to light conditions, and no risk of user privacy leaks. Therefore, radar-based human motion recognition methods have received increasing attention, and many radar-based human motion recognition methods have been proposed. Furthermore, human movements are typically continuous and dynamic; compared to discrete human motion recognition, continuous human motion recognition more closely reflects real-world scenarios.

[0003] In recent years, deep learning technology has gradually emerged in the field of radar-based continuous human action recognition. However, most methods are still based on single-modal radar human action representations, such as range-Doppler features and radar point clouds. Although these methods are practical, range-Doppler features based on two-dimensional spectral maps have limitations in resolution, and the point cloud data of conventional multi-transmitter multi-receiver radars are sparse, limiting their ability to provide human action features. Therefore, they have certain shortcomings in achieving high-precision radar continuous human action recognition.

[0004] Therefore, proposing a new method for continuous human motion recognition to improve the accuracy of continuous human motion recognition has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a continuous human motion recognition method based on point cloud data and range Doppler spectrum, so as to at least solve the problem of low recognition accuracy of existing radar continuous human motion recognition methods.

[0006] To achieve the above objectives, this invention provides a continuous human action recognition method based on point cloud data and distance Doppler spectral mapping, comprising:

[0007] Step 1: Build an FMCW radar data acquisition platform and use the platform to acquire a series of echoes containing multiple continuous human movements to obtain sampled intermediate frequency (IF) signals. Then, preprocess the IF signals to obtain an overall IF signal matrix. Next, use a sliding window to segment the overall IF signal matrix to obtain the corresponding discrete IF signal matrix. Finally, perform signal processing on the IF signal matrix to obtain the corresponding range Doppler spectrum and point cloud data.

[0008] Step 2: Construct a PVT-BiLSTM network, which includes a Pointnet-BiLSTM module, a ViT module, and a fusion module. The Pointnet-BiLSTM module uses two T-net networks, five one-dimensional convolutional layers, two BiLSTM layers with 64 and 128 neurons respectively, one one-dimensional global max pooling layer, and two fully connected layers with 256 and 128 neurons respectively. Each convolutional layer and fully connected layer contains a BN layer and a ReLU activation function. The first three one-dimensional convolutional layers have 32 kernels and a size of 1, the fourth convolutional layer has 64 kernels and a size of 1, and the fifth convolutional layer has 512 kernels and a size of 1. The T-net network includes three one-dimensional convolutional layers with 32, 64, and 512 kernels and a size of 1, one one-dimensional global max pooling layer, two fully connected layers with 256 and 128 neurons respectively, and a fusion module. 2 The fully connected layer of each neuron and the reshaped transformation matrix, the transformation matrix of the T-net network is r×r, where r is the dimension of the transformation matrix. For the two T-net networks, the dimensions of the transformation matrix are 4 and 32, respectively. The ViT module consists of five processing steps: Patch segmentation, linear projection, class labeling, location embedding, Transformer encoder and multilayer perceptron.

[0009] Step 3: Divide the distance Doppler spectrum and point cloud data into training and validation sets, and input them into the PVT-BiLSTM network for training and validation to obtain a trained PVT-BiLSTM network model. Then, use the trained PVT-BiLSTM network model to recognize continuous human movements.

[0010] Preferably, step 1 includes the following steps:

[0011] (1) The FMCW radar data acquisition platform is used to measure the echo of a complete human body in continuous motion, and the echo of the continuous human body motion is sampled to obtain the sampled intermediate frequency signal. Then, the sampled intermediate frequency signals are stacked column by column to form an overall intermediate frequency signal matrix S. IF The overall matrix S of the intermediate frequency signalIF Each element is represented as s IF (m, n), where m = 1, 2, ..., M, n = 1, 2, ..., N total m is the fast time index, n is the slow time index, M is the total number of fast time samples corresponding to a single chirp, and N is the slow time index. total This represents the total number of Chirps launched.

[0012] (2) Using a fixed window length and sliding step size, the overall matrix S of the intermediate frequency signal is... IF By performing sliding window segmentation along the slow time dimension, the intermediate frequency signal matrix X corresponding to each individual discrete action is obtained. i Let i = 1, 2, ..., I, where I is the number of discrete action samples generated by sliding window segmentation. Each element in the i-th intermediate frequency signal matrix is ​​represented as X. i (m, n) sub ), n sub = 1, 2, ..., N, where N is the total number of transmit chirps corresponding to each intermediate frequency signal matrix;

[0013] (3) For the intermediate frequency signal matrix X i Perform a fast Fourier transform in the fast time dimension to obtain the distance-time matrix Y. i Then, for the distance-time matrix Y i Performing a Fast Fourier Transform in a slow time dimension yields the range Doppler matrix Z. i Next, the distance Doppler matrix Z is... i The image is mapped to a three-channel RGB color image to obtain a distance Doppler spectrum. Finally, the size of the distance Doppler spectrum is scaled to 30×30×3.

[0014] (4) For the distance-time matrix Y i Further segmented along the slow time dimension into L Z Distance time submatrix at the same time in, This is the index of the distance-time submatrix;

[0015] (5) For each distance-time submatrix Performing Fast Fourier Transform on the slow time dimension separately yields L. Z Distance Doppler submatrices

[0016] (6) For each distance Doppler submatrix A static threshold filter is applied to select useful human motion features. The filtering formula is as follows:

[0017]

[0018] In the formula, μ is the threshold ratio, with a value range of [0, 1], and n a =1,2,...,N a n a For each distance-Doppler submatrix, the Doppler frequency index, N a The number of samples for the Doppler frequency. for The dB value of the element in the middle. for The maximum dB value of the element in the middle. This is the range Doppler submatrix after processing with a static threshold filter;

[0019] (7) The range Doppler submatrix after static thresholding is processed. Stacking points along the slow time dimension forms a distance-Doppler-time three-dimensional tensor. The coordinates of points with intensity greater than 0 in the tensor are stored in a list and assigned four basic variables: distance, Doppler, time, and intensity, thus obtaining a four-dimensional point cloud that represents the evolution of human motion features over a certain time window. Then, the number of points in the stored list is filtered according to the order of maximum intensity, and the top 1024 points with the highest intensity are selected.

[0020] Further preferred, in step 2, the loss function of the PVT-BiLSTM network is as follows:

[0021] L(p t )=α·CE(p t )+β·FL(p t (3)

[0022] In the formula, p t C(p) represents the probability of the predicted true instance, t = 1, 2, ..., Q, where Q is the sample size in a batch, and C(p) represents the probability of the predicted true instance. t ) represents the cross-entropy loss value for sample t, FL(p) t L(p) represents the focal loss value for sample t. t ) represents the combined loss value for sample t, where α and β are weighting coefficients used to adjust the weights of the two loss functions;

[0023] The formula for focus loss is as follows:

[0024] FL(p t )=-α t (1-p t ) γ log(p t (2)

[0025] In the formula, p tLet α represent the predicted true value, t = 1, 2, ..., Q, where Q is the sample size in a batch. t γ is the balancing factor used to adjust the ratio between positive and negative sample losses. γ is the focusing parameter used to adjust the weight of easily classified samples. When γ is 0, the focus loss degenerates into the standard cross-entropy loss.

[0026] Further optimization, α = β = 0.5, α t =0.25, γ=2.

[0027] Further optimization, in step 3, the complete network model training process is as follows:

[0028] For each 1024x4 point cloud data sample input, the Pointnet-BiLSTM module multiplies it with a 4×4 transformation matrix learned by the T-net network through feature extraction, resulting in a spatially corrected 1024×4 point cloud. This corrected point cloud is then fed into the first two convolutional layers for feature extraction. Next, it is multiplied with a 32×32 transformation matrix from the second T-net network, and then fed into the third and fourth convolutional layers to map the features to 1024×64. The two BiLSTM layers further enhance the feature dimensionality. The feature vector is mapped to 1024×256. The point cloud features extracted by the one-dimensional convolutional layer meet the input requirements of BiLSTM. BiLSTM treats the point cloud features as a sequence and obtains more comprehensive contextual information by running two LSTMs simultaneously in the forward and backward directions of the sequence. This complements the convolutional layer in performing dimensionality upscaling of the point cloud features. The feature vector is then input into the last convolutional layer and mapped to 1024×512. Then, a one-dimensional global max pooling operation is used to preserve the features to a 1×512 vector. After dimensionality reduction by two fully connected layers, the optimal feature vector is obtained.

[0029] The ViT module first processes each input 30×30×3 RGB distance Doppler spectrum. Divided into a series of flattened squares Where H and W are the height and width of the spectrum, H = 30, W = 30, Patch size P = 6, and a total of N are generated. z =HW / P 2The number of patches is divided into several parts. The entire distance Doppler map is reshaped into a one-dimensional sequence by the patch segmentation. The sliced ​​and flattened patches are then linearly projected through a fully connected layer without an activation function to produce patch embeddings. The feature vector of each patch is mapped to a new feature space. After the patch embedding, class labels are added to the sequence and passed through all layers of the model to capture global information of the entire input sequence. Finally, position embedding is introduced by adding an additional vector associated with each patch and its position in the original distance Doppler map to the patch embedding.

[0030] The Transformer encoder consists of two key layers: the MSA layer and the MLP layer. Both layers are designed with residual connections, and an LN layer is applied before each layer.

[0031] The MSA layer contains multiple independent and parallel self-attention modules. The MSA layer assigns different weights to each feature vector, as shown in the following formula:

[0032]

[0033] MSA(Q,K,V)=Concatenate(head1,head2,...,head θ W O (5)

[0034]

[0035] In the formula, Q, K, and V represent the query, key, and value matrices, respectively, and QK T This represents the similarity score between the query and the key pair, where d represents the dimension of the feature vector processed by each self-attention head. Indicates the scaling factor. and W O This represents the learnable weight matrix. It is the first There are 1 self-attention heads, where θ is the number of self-attention heads;

[0036] After the Transformer encoder layer, an LN layer is applied. The final MLP layer of the network consists of two fully connected layers with 512 and 256 neurons respectively, accompanied by a dropout layer with a dropout rate of 0.5. The GELU function is used as the activation function, and its formula is expressed as:

[0037]

[0038] In the formula, Φ(x) represents the cumulative distribution function of the standard Gaussian distribution;

[0039] The entire calculation process of the ViT module can be represented as follows:

[0040]

[0041] z′ l =MSA(LN(z) l-1 ))+z l-1 (9)

[0042] z l =MLP(LN(z′) l ))+z′ l (10)

[0043]

[0044] In the formula, W represents the weight of the linear projection layer. D represents the projection dimension, D = 64, x class Represents learnable category labels. W pos Indicates positional embedding, L represents the total number of stacked Transformer encoders, L = 8, and l represents the sequence number of the Transformer encoder, l = 1, 2, ..., L, z l The output feature of the l-th Transformer encoder is represented by the category label from the output of the last Transformer encoder. Apply the final MLP and LN respectively. The optimal eigenvector y that can represent the entire range Doppler spectrum is obtained.

[0045] The fusion module will combine vector x best The formula for concatenating with y is as follows:

[0046] g = Concatenate(x) best ,y) (12)

[0047] In the formula, g represents the fused feature vector;

[0048] Residual connections are made after two fully connected layers with 512 and 384 neurons each, using GELU as the activation function, and two dropout layers with a dropout rate of 0.5. The final layer is a fully connected layer with 5 neurons using Softmax as the activation function, which outputs the predicted probability for each class.

[0049] Further optimization involves using the AdamW optimizer in step 3 during network model training. The weight decay coefficient is 0.0001, the batch size is 32, the learning rate is 0.001, and the number of iterations is 60.

[0050] The continuous human action recognition method based on point cloud data and range-Doppler spectra provided by this invention can accurately recognize continuous human actions. Utilizing multimodal data of radar four-dimensional point cloud (range-Doppler-time-intensity) and range-Doppler spectra, it provides sufficient representation of human actions, outperforming traditional single-modal data. The employed PVT-BiLSTM dual-stream network combines PointNet and BiLSTM, offering advantages in both local feature extraction and time-series feature extraction. Combining PointNet-BiLSTM with the ViT network enables complementary features from point cloud and spectra. The loss function combining cross-entropy loss and focus loss addresses the sample imbalance problem in continuous human action recognition, significantly improving the performance of continuous human action recognition. Attached Figure Description

[0051] Figure 1 A flowchart of the continuous human motion recognition method based on point cloud data and distance Doppler spectrum provided by the present invention;

[0052] Figure 2 This is a diagram showing the overall structure of the PVT-BiLSTM network.

[0053] Figure 3 This is a network structure diagram of the T-net module;

[0054] Figure 4 Here is the network structure diagram of the ViT module;

[0055] Figure 5 The graph shows how the action recognition accuracy of the training and validation sets changes with the number of iterations.

[0056] Figure 6 Confusion matrix diagram for the test set;

[0057] Figure 7 A bar chart showing the accuracy of the test set and related evaluation metrics. Detailed Implementation

[0058] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention. Figure 1 As shown, this invention provides a method for continuous human motion recognition based on point cloud data and distance Doppler spectrograms, comprising the following steps:

[0059] Step 1: Build an FMCW radar data acquisition platform and use the platform to acquire a series of echoes containing various continuous human movements to obtain sampled intermediate frequency (IF) signals. Then, preprocess the IF signals to obtain an overall IF signal matrix. Next, segment the overall IF signal matrix using a sliding window to obtain the corresponding discrete IF signal matrices. Finally, perform signal processing on the IF signal matrices to obtain the corresponding range Doppler spectrum and point cloud data. Specifically, this includes the following steps:

[0060] (1) The FMCW radar data acquisition platform is used to measure the echo of a complete human body in continuous motion, and the echo of the continuous human body motion is sampled to obtain the sampled intermediate frequency signal. Then, the sampled intermediate frequency signals are stacked column by column to form an overall intermediate frequency signal matrix S. IF The overall matrix s of the intermediate frequency signal IF Each element is represented as S IF (m, n), m=1, 2,..., M, n=1, 2,..., N total m is the fast time index, n is the slow time index, M is the total number of fast time samples corresponding to a single chirp, and N is the slow time index. total This represents the total number of Chirps launched.

[0061] (2) Using a fixed window length and sliding step size, the overall matrix s of the intermediate frequency signal is subjected to a fixed window length and a sliding step size. IF By performing sliding window segmentation along the slow time dimension, the intermediate frequency signal matrix X corresponding to each individual discrete action is obtained. i Let i = 1, 2, ..., I, where I is the number of discrete action samples generated by sliding window segmentation. Each element in the i-th intermediate frequency signal matrix is ​​represented as X. i (m, n) sub ), n sub = 1, 2, ..., N, where N is the total number of transmit chirps corresponding to each intermediate frequency signal matrix;

[0062] (3) For the intermediate frequency signal matrix X i Perform a Fast Fourier Transform (FFT) on the fast time dimension to obtain the distance-time matrix Y. i Then, for the distance-time matrix Y i Perform a slow-time FFT to obtain the distance Doppler matrix Z. i Next, the distance Doppler matrix Z is... i The image is mapped to a three-channel RGB color image to obtain a distance Doppler spectrum. Finally, the size of the distance Doppler spectrum is scaled to 30×30×3.

[0063] (4) For the distance-time matrix Y i Further segmented along the slow time dimension into LZ Distance time submatrix at the same time in, This is the index of the distance-time submatrix;

[0064] (5) For each distance-time submatrix Perform FFT on the slow time dimension separately to obtain L. Z Distance Doppler submatrices

[0065] (6) For each distance Doppler submatrix A static threshold filter is applied to select useful human motion features. The filtering formula is as follows:

[0066]

[0067] In the formula, μ is the threshold ratio, with a value range of [0, 1]. Preferably, μ = 0.5, n a =1,2,...,N a n a For each distance-Doppler submatrix, the Doppler frequency index, N a The number of samples for the Doppler frequency. for The dB value of the element in the middle. for The maximum dB value of the element in the middle. This is the range Doppler submatrix after processing with a static threshold filter;

[0068] (7) The range Doppler submatrix after static thresholding is processed. Stacking points along the slow time dimension forms a distance-Doppler-time three-dimensional tensor. The coordinates of points with intensity greater than 0 in the tensor are stored in a list and assigned four basic variables: distance, Doppler, time, and intensity, thus obtaining a four-dimensional point cloud, representing the evolution of human motion features over a certain time window. At this point, the stored list contains point clouds with all points. The number of point clouds is fixed during the network input stage because the method for generating four-dimensional point clouds does not need to worry about insufficient quantity. The point cloud points are then filtered according to the order of maximum intensity, selecting the top 1024 points with the highest intensity.

[0069] Step 2: Build a PVT-BiLSTM network, such as... Figure 2 As shown, the PVT-BiLSTM network includes a Pointnet-BiLSTM module, a ViT module, and a fusion module. The Pointnet-BiLSTM module is designed as follows: Figure 2As shown, this network employs two T-net networks, five one-dimensional convolutional (Conv1D) layers, two BiLSTM layers with 64 and 128 neurons respectively, one one-dimensional global max pooling (GlobalMaxPooling1D) layer, and two fully connected (Dense) layers with 256 and 128 neurons respectively. Each convolutional and fully connected layer contains a BatchNorm (BN) layer and a ReLU activation function. The first three one-dimensional convolutional layers have 32 kernels of size 1, the fourth convolutional layer has 64 kernels of size 1, and the fifth convolutional layer has 512 kernels of size 1. The T-net network design is as follows: Figure 3 As shown, the network contains three one-dimensional convolutional layers with 32, 64, and 512 kernels of size 1, a one-dimensional global max-pooling layer, and two fully connected layers with 256 and 128 neurons respectively. 2 The T-net network consists of a fully connected layer with 10 neurons and a reshaped transformation matrix. The transformation matrix of the T-net network is r×r, where r is the dimension of the transformation matrix. For the two T-net networks, the dimensions of the transformation matrix are 4 and 32, respectively.

[0070] The ViT module design is as follows: Figure 4 As shown, this module consists of five processing steps: Patch segmentation, linear projection, class labeling, location embedding, Transformer encoder, and Multilayer Perceptron (MLP).

[0071] The loss function for the PVT-BiLSTM network is set up. Traditional classification tasks typically use the cross-entropy function as the loss function, but continuous human action recognition suffers from an imbalance problem between different action categories. Focus loss is proposed to address this imbalance problem. It reduces the weight of easily classified samples and increases the weight of difficult-to-classify samples, thus making the model focus more on the difficult-to-classify samples. Therefore, this invention designs a loss function that combines focus loss and cross-entropy loss. The formula for focus loss is as follows:

[0072] FL(p t )=-α t (1-p t ) γ log(p t (2)

[0073] In the formula, p t Let α represent the predicted true value, t = 1, 2, ..., Q, where Q is the sample size in a batch. t It is a balancing factor used to adjust the ratio between positive and negative sample losses. Preferably, αt =0.25, γ is the focusing parameter used to adjust the weights of easily classified samples. Preferably, γ = 2. When γ is 0, the focus loss degenerates into the standard cross-entropy loss, FL(p t The focus loss value for sample t is represented by , and the loss function combining focus loss and cross-entropy loss is defined as follows:

[0074] L(p t )=α·CE(p t )+β·FL(p t (3)

[0075] In the formula, CE(p) t L(p) represents the cross-entropy loss value for sample t. t ) represents the combined loss value for sample t, where α and β are both weighting coefficients used to adjust the weights of the two loss functions. Preferably, α = β = 0.5.

[0076] Step 3: Divide the distance Doppler spectrum and point cloud data into training and validation sets, and input them into the PVT-BiLSTM network for training and validation to obtain a trained PVT-BiLSTM network model. Then, use the trained PVT-BiLSTM network model to recognize continuous human actions. It is particularly important to note that since this network is a dual-stream input framework, the sizes and label correspondences of the two datasets must be consistent. The complete network model training process includes the following steps:

[0077] The Pointnet-BiLSTM module takes a 1024×4 point cloud data sample as input and multiplies it with a 4×4 transformation matrix learned by the T-net network through feature extraction. This multiplication yields a spatially corrected 1024×4 point cloud, which is then fed into the first two convolutional layers for feature extraction. Next, it is multiplied with a 32×32 transformation matrix from the second T-net network. Finally, it is fed into the third and fourth convolutional layers to map the features to 1024×64. Two BiLSTM layers further enhance the feature dimensionality, thus... The feature mapping is 1024×256. The point cloud features extracted by the one-dimensional convolutional layer meet the input requirements of BiLSTM. BiLSTM can treat point cloud features as a sequence. By running two LSTMs simultaneously in the forward and backward directions of the sequence, more comprehensive contextual information is obtained. Complementing the convolutional layer, it can effectively perform dimensionality upscaling of point cloud features. Then, the input to the last convolutional layer maps the features to 1024×512. Then, a one-dimensional global max pooling operation is used to preserve the features to a 1×512 vector. After dimensionality reduction by two fully connected layers, the optimal feature vector is obtained. To prepare for future integration.

[0078] The Vision Transformer (ViT) module first processes each input 30×30×3 RGB distance-Doppler spectrum. Divided into a series of flattened squares Where H and W are the height and width of the spectrum, H = 30, W = 30, Patch size P = 6, and a total of N are generated. z =HW / P 2 The system uses a number of patches to segment the entire distance-Doppler spectrum, reshaping it into a one-dimensional sequence. The flattened patches are then linearly projected through a fully connected layer without activation functions to produce patch embeddings. This linear projection is a direct mathematical transformation that maps the feature vector of each patch to a new feature space without introducing any non-linear changes. After the patch embeddings, class labels are added to the sequence. These class labels are special vectors, typically located at the beginning of the sequence and propagated throughout all layers of the model, capturing global information about the entire input sequence. This label is crucial for classification tasks as it allows the model to distinguish between different classes and plays a role in the final classification decision. Finally, to preserve the positional information of the patches in the original distance-Doppler spectrum, positional embeddings are introduced. These are achieved by adding an additional vector associated with each patch and its position in the original distance-Doppler spectrum to the patch embedding. This allows the model to identify the position of each individual patch, a crucial step because the multi-head self-attention (MSA) layers in the Transformer encoder do not consider the order or position of the patches, focusing only on their relationships. The positional embedding of the category label is usually a learnable parameter that allows the model to adjust its value during training to better perform classification tasks. In this way, the ViT model can not only understand the relationships between patches, but also use positional information to improve classification accuracy.

[0079] The Transformer encoder consists of two key layers: the MSA layer and the MLP layer. Both layers are designed with residual connections, and a LayerNorm (LN) layer is applied before each layer. In finding the most efficient representation of the learnable class label, ViT operates in a dictionary-like manner.

[0080] The MSA layer contains multiple independent and parallel self-attention modules. The advantage of MSA is that it assigns different weights to each feature vector, which helps extract useful features. Its formula is as follows:

[0081]

[0082] MSA(Q,K,V)=Concatenate(head1,head2,...,head θ W O (5)

[0083]

[0084] In the formula, Q, K, and v represent the query, key, and value matrices, respectively. T This represents the similarity score between the query and the key pair, where d represents the dimension of the feature vector processed by each self-attention head. Indicates the scaling factor. and W O This represents the learnable weight matrix. It is the first There are 4 self-attention heads, where θ = 4 is the number of self-attention heads.

[0085] After the Transformer encoder layer, an LN layer is applied. The final MLP layer of the network consists of two fully connected layers with 512 and 256 neurons respectively, accompanied by a dropout layer with a dropout rate of 0.5 to avoid overfitting. The GELU function is used as the activation function, and its formula can be expressed as:

[0086]

[0087] In the formula, Φ(x) represents the cumulative distribution function of the standard Gaussian distribution.

[0088] Finally, the entire calculation process of ViT can be represented as:

[0089]

[0090] z′ l =MSA(LN(z) l-1 ))+z l-1 (9)

[0091] z l =MLP(LN(z′) l ))+z′ l (10)

[0092]

[0093] In the formula, W represents the weight of the linear projection layer. D represents the projection dimension, D = 64, x class Represents learnable category labels. W pos Indicates positional embedding, L represents the total number of stacked Transformer encoders, L = 8, and l represents the sequence number of the Transformer encoder, l = 1, 2, ..., L, z l The output feature of the l-th Transformer encoder is represented by the category label from the output of the last Transformer encoder. Apply the final MLP and LN respectively. The optimal eigenvector y that can represent the entire range Doppler spectrum is obtained.

[0094] The fusion module will combine vector x best Concatenate with y, as shown in the following formula:

[0095] g = Concatenate(x) best ,y) (12)

[0096] In the formula, Represented as the fused feature vector;

[0097] Residual connections are made after two fully connected layers with 512 and 384 neurons each, using GELU as the activation function, and two dropout layers with a dropout rate of 0.5. The final layer is a fully connected layer with 5 neurons that uses Softmax as the activation function, which outputs the predicted probability for each class.

[0098] The network model was trained from scratch using the training set data and the AdamW optimizer. A weight decay factor of 0.0001 was used to help avoid the impact of weight decay on the bias parameters. The batch size was 32, the learning rate was 0.001, and the number of iterations was 60. The network model performance was monitored using the validation set data until the network converged, resulting in the trained network model.

[0099] The following detailed explanation uses specific examples to illustrate the aforementioned FMCW radar continuous human motion recognition method based on the fusion of point cloud data and range Doppler spectrograms:

[0100] In this embodiment, the FMCW radar system was placed on a 1.2-meter-high table, and continuous motion sequences containing five actions were measured on five volunteers in an indoor environment. These actions included (a) stepping in place, (b) remaining still, (c) walking, (d) jumping, and (e) bending over. The starting frequency was 77 GHz, the bandwidth was 2 GHz, the ADC sampling frequency was 32 MHz, the fast-time sampling points were 128, the slow-time sampling points were 64,000, the frame period was 100 ms, the number of frames was 500, and the number of chirp signals per frame was 128. Each participant had four sets of motion sequences collected, for a total of 20 sets of motion sequences. Each continuous motion sequence was measured for 50 seconds and consisted of five different combinations of actions.

[0101] Point cloud data and range-Doppler spectra were obtained by sliding window segmentation and signal processing of the overall intermediate frequency signal matrix. The sliding window segmentation used a window length of 2 seconds and a sliding step of 0.1 seconds. The point cloud data was generated when the distance-time submatrix corresponded to 0.25 seconds of data, with a sliding step of 0.125 seconds. Each point cloud data set contained 15 range-Doppler submatrices, and the static filter threshold ratio was 0.5. After preprocessing, 7680 four-dimensional point cloud data points and 7680 three-channel RGB range-Doppler spectra were obtained.

[0102] The point cloud data and range-Doppler spectra were randomly divided into training and test sets in a 4:1 ratio. Then, 20% of the training set was allocated as a validation set. Taking the range-Doppler spectra as an example, a total of 4916 training images, 1228 validation images, and 1536 test images were generated.

[0103] The point cloud dataset used was 1024 points, and the distance-Doppler spectrum size was 30×30×3. The AdamW optimizer was used for training from scratch, with a weight decay factor of 0.0001 to help avoid the impact of weight decay on the bias parameters. The batch size was 32, the learning rate was 0.001, and the number of iterations was 60. The network model of this invention uses the TensorFlow 2.8 deep learning framework, and all model training, validation, and testing experiments were performed on an NVIDIA GTX 1050Ti GPU.

[0104] The experimental results are shown in the graphs illustrating how the action recognition accuracy of the training and validation sets changes with the number of iterations. Figure 5 As shown, the validation set accuracy reached 83% in the first round, and stabilized at around 98% as the number of iterations increased. Figure 6 The confusion matrix for the test set of five actions is shown, with the horizontal axis representing the predicted label and the vertical axis representing the true label. Due to transitions between consecutive actions and human labeling errors, some prediction errors are unavoidable. However, the proposed network model still maintains high recognition performance. Figure 7 The test accuracy, precision, recall, and F1 score of the network model are presented. Its test set accuracy reaches 98.24%.

[0105] This invention provides a continuous human action recognition method based on point cloud data and range Doppler spectra. Within a PVT-BiLSTM dual-stream network framework, a Pointnet-BiLSTM network is used to learn from four-dimensional radar point cloud data. Simultaneously, a ViT network with a self-attention mechanism is fully utilized to learn features from the range Doppler spectra. Finally, classification is performed by fusing features from both single-stream modes. The method combines cross-entropy loss and focus loss functions to address the imbalanced classification problem in continuous human action recognition, effectively improving the accuracy of continuous human action recognition.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for continuous human motion recognition based on point cloud data and range Doppler spectrograms, characterized in that, include: Step 1: Build an FMCW radar data acquisition platform and use the platform to acquire a series of echoes containing multiple continuous human movements to obtain sampled intermediate frequency (IF) signals. Then, preprocess the IF signals to obtain an overall IF signal matrix. Next, use a sliding window to segment the overall IF signal matrix to obtain the corresponding discrete IF signal matrix. Finally, perform signal processing on the IF signal matrix to obtain the corresponding range Doppler spectrum and point cloud data. Step 2: Construct a PVT-BiLSTM network, which includes a Pointnet-BiLSTM module, a ViT module, and a fusion module. The Pointnet-BiLSTM module uses two T-net networks, five one-dimensional convolutional layers, two BiLSTM layers with 64 and 128 neurons respectively, one one-dimensional global max pooling layer, and two fully connected layers with 256 and 128 neurons respectively. Each convolutional layer and fully connected layer contains a BN layer and a ReLU activation function. The first three one-dimensional convolutional layers have 32 kernels and a size of 1, the fourth convolutional layer has 64 kernels and a size of 1, and the fifth convolutional layer has 512 kernels and a size of 1. The T-net network includes three one-dimensional convolutional layers with 32, 64, and 512 kernels and a size of 1, one one-dimensional global max pooling layer, two fully connected layers with 256 and 128 neurons respectively, and a fusion module. 2 The fully connected layer of each neuron and the reshaped transformation matrix, the transformation matrix of the T-net network is r×r, where r is the dimension of the transformation matrix. For the two T-net networks, the dimensions of the transformation matrix are 4 and 32, respectively. The ViT module consists of five processing steps: Patch segmentation, linear projection, class labeling, location embedding, Transformer encoder and multilayer perceptron. Step 3: Divide the distance Doppler spectrum and point cloud data into training and validation sets, and input them into the PVT-BiLSTM network for training and validation to obtain a trained PVT-BiLSTM network model. Then, use the trained PVT-BiLSTM network model to recognize continuous human movements.

2. The continuous human motion recognition method based on point cloud data and range Doppler spectrum as described in claim 1, characterized in that: Step 1 includes the following steps: (1) The FMCW radar data acquisition platform is used to measure the echo of a complete human body in continuous motion, and the echo of the continuous human body motion is sampled to obtain the sampled intermediate frequency signal. Then, the sampled intermediate frequency signals are stacked column by column to form an overall intermediate frequency signal matrix s. IF The overall matrix S of the intermediate frequency signal IF Each element is represented as S IF (m, n), where m = 1, 2, ..., M, n = 1, 2, ..., N total m is the fast time index, n is the slow time index, M is the total number of fast time samples corresponding to a single chirp, and N is the slow time index. total This represents the total number of Chirps launched. (2) Using a fixed window length and sliding step size, the overall matrix S of the intermediate frequency signal is... IF By performing sliding window segmentation along the slow time dimension, the intermediate frequency signal matrix X corresponding to each individual discrete action is obtained. i Let i = 1, 2, ..., I, where I is the number of discrete action samples generated by sliding window segmentation. Each element in the i-th intermediate frequency signal matrix is ​​represented as X. i (m, n) sub ), n sub = 1, 2, ..., N, where N is the total number of transmit chirps corresponding to each intermediate frequency signal matrix; (3) For the intermediate frequency signal matrix X i Perform a fast Fourier transform in the fast time dimension to obtain the distance-time matrix Y. i Then, for the distance-time matrix Y i Performing a Fast Fourier Transform in a slow time dimension yields the range Doppler matrix Z. i Next, the distance Doppler matrix Z is... i The image is mapped to a three-channel RGB color image to obtain a distance Doppler spectrum. Finally, the size of the distance Doppler spectrum is scaled to 30×30×3. (4) For the distance-time matrix Y i Further segmented along the slow time dimension into L Z Distance time submatrix at the same time in, This is the index of the distance-time submatrix; (5) For each distance-time submatrix Performing Fast Fourier Transform on the slow time dimension separately yields L. Z Distance Doppler submatrices (6) For each distance Doppler submatrix A static threshold filter is applied to select useful human motion features. The filtering formula is as follows: In the formula, μ is the threshold ratio, with a value range of [0, 1], and n a =1,2,...,N a n a For each distance-Doppler submatrix, the Doppler frequency index, N a The number of samples for the Doppler frequency. for The dB value of the element in the middle. for The maximum dB value of the element in the middle. This is the range Doppler submatrix after processing with a static threshold filter; (7) The range Doppler submatrix after static thresholding is processed. Stacking points along the slow time dimension forms a distance-Doppler-time three-dimensional tensor. The coordinates of points with intensity greater than 0 in the tensor are stored in a list and assigned four basic variables: distance, Doppler, time, and intensity, thus obtaining a four-dimensional point cloud that represents the evolution of human motion features over a certain time window. Then, the number of points in the stored list is filtered according to the order of maximum intensity, and the top 1024 points with the highest intensity are selected.

3. The continuous human motion recognition method based on point cloud data and range Doppler spectrum as described in claim 1, characterized in that: In step 2, the loss function of the PVT-BiLSTM network is as follows: L(p t )=α·CE(p t )+β·FL(p t ) (3) In the formula, p t Let t represent the predicted true value, t = 1, 2, ..., Q, where Q is the sample size in a batch, and CE(p) = ... t ) represents the cross-entropy loss value for sample t, FL(p) t L(p) represents the focal loss value for sample t. t ) represents the combined loss value for sample t, where α and β are weighting coefficients used to adjust the weights of the two loss functions; The formula for focus loss is as follows: FL(p t )D-α t (1-px) γ log ( p ) t ) (2) In the formula, p t Let α represent the predicted true value, t = 1, 2, ..., Q, where Q is the sample size in a batch. t γ is the balancing factor used to adjust the ratio between positive and negative sample losses. γ is the focusing parameter used to adjust the weight of easily classified samples. When γ is 0, the focus loss degenerates into the standard cross-entropy loss.

4. The continuous human motion recognition method based on point cloud data and range Doppler spectrum as described in claim 3, characterized in that: α=β=0.5,α t =0.25, γ=2.

5. The continuous human motion recognition method based on point cloud data and range Doppler spectrum as described in claim 1, characterized in that: In step 3, the complete network model training process is as follows: For each 1024×4 point cloud data sample input, the Pointnet-BiLSTM module multiplies it with a 4×4 transformation matrix learned by the T-net network through feature extraction, resulting in a spatially corrected 1024×4 point cloud. This corrected point cloud is then fed into the first two convolutional layers for feature extraction. Next, it is multiplied with a 32×32 transformation matrix from the second T-net network, and then fed into the third and fourth convolutional layers to map the features to 1024×64. The two BiLSTM layers further enhance the feature dimensionality. The feature vector is mapped to 1024x256. The point cloud features extracted by the one-dimensional convolutional layer meet the input requirements of BiLSTM. BiLSTM treats the point cloud features as a sequence and obtains more comprehensive contextual information by running two LSTMs simultaneously in the forward and backward directions of the sequence. This complements the convolutional layer in performing dimensionality upscaling of the point cloud features. The feature vector is then input into the last convolutional layer to map to 1024×512. Then, a one-dimensional global max pooling operation is used to preserve the features to a 1×512 vector. After dimensionality reduction by two fully connected layers, the optimal feature vector is obtained. The ViT module first processes each input 30×30×3 RGB distance Doppler spectrum. Divided into a series of flattened squares Where H and W are the height and width of the spectrum, H = 30, W = 30, Patch size P = 6, and a total of N are generated. z =HW / P 2 The number of patches is divided into several parts. The entire distance Doppler map is reshaped into a one-dimensional sequence by the patch segmentation. The sliced ​​and flattened patches are then linearly projected through a fully connected layer without an activation function to produce patch embeddings. The feature vector of each patch is mapped to a new feature space. After the patch embedding, class labels are added to the sequence and passed through all layers of the model to capture global information of the entire input sequence. Finally, position embedding is introduced by adding an additional vector associated with each patch and its position in the original distance Doppler map to the patch embedding. The Transformer encoder consists of two key layers: the MSA layer and the MLP layer. Both layers are designed with residual connections, and an LN layer is applied before each layer. The MSA layer contains multiple independent and parallel self-attention modules. The MSA layer assigns different weights to each feature vector, as shown in the following formula: MSA(Q,K,V)=Concatenate(head1,head2,...,head θ )W O (5) In the formula, Q, K, and V represent the query, key, and value matrices, respectively, and QK T This represents the similarity score between the query and the key pair, where d represents the dimension of the feature vector processed by each self-attention head. Indicates the scaling factor. and W O This represents the learnable weight matrix. It is the first There are 1 self-attention heads, where θ is the number of self-attention heads; After the Transformer encoder layer, an LN layer is applied. The final MLP layer of the network consists of two fully connected layers with 512 and 256 neurons respectively, accompanied by a dropout layer with a dropout rate of 0.

5. The GELU function is used as the activation function, and its formula is expressed as: In the formula, Φ(x) represents the cumulative distribution function of the standard Gaussian distribution; The entire calculation process of the ViT module can be represented as follows: z′ l =MSA(LN(z l-1 ))+z l-1 (9) With l =MLP(LN(z′ l ))+z′ l (10) In the formula, W represents the weight of the linear projection layer. D represents the projection dimension, D = 64, x class Represents learnable category labels. W pos Indicates positional embedding, L represents the total number of stacked Transformer encoders, L = 8, and l represents the sequence number of the Transformer encoder, l = 1, 2, ..., L, z l The output feature of the l-th Transformer encoder is represented by the category label from the output of the last Transformer encoder. Apply the final MLP and LN respectively. The optimal eigenvector y that can represent the entire range Doppler spectrum is obtained. The fusion module will combine vector x best The formula for concatenating with y is as follows: g=Concatenate(x best ,y) (12) In the formula, g represents the fused feature vector; Residual connections are made after two fully connected layers with 512 and 384 neurons each, using GELU as the activation function, and two dropout layers with a dropout rate of 0.

5. The final layer is a fully connected layer with 5 neurons using Softmax as the activation function, which outputs the predicted probability for each class.

6. The continuous human motion recognition method based on point cloud data and range Doppler spectrum as described in claim 5, characterized in that: In step 3, during the training of the network model, the AdamW optimizer is used, with a weight decay coefficient of 0.0001, a batch size of 32, a learning rate of 0.001, and 60 iterations.

Images