A zero-sample behavior perception method of millimeter-wave radar based on vision assistance

Abstract

The present invention belongs to the technical field of zero-sample wireless signal behavior perception, and discloses a zero-sample behavior perception method of millimeter-wave radar based on vision assistance, which collects video and millimeter-wave radar data containing various action categories at the same time, and divides the millimeter-wave radar data set into a training set and a test set; extracts global and local attributes related to vision respectively, and obtains the final attributes of each category after fusion; constructs a three-channel convolutional neural network for extracting highly discernible feature expressions in millimeter-wave radar signals, and compares the features obtained based on the test set with the attributes of each different category to obtain the category of the millimeter-wave radar signal. The present invention uses rich visual samples to generate attributes of various different categories of actions, designs a three-channel convolutional neural network to extract millimeter-wave radar features with strong discrimination, and constructs the relationship between attributes and millimeter-wave radar, so as to realize zero-sample learning tasks, reduce data collection and training costs, and broaden the application field of millimeter-wave radar.

Description

Technical Field

[0001] The present invention belongs to the technical field of zero-sample wireless signal behavior perception, and specifically relates to a vision-assisted millimeter-wave radar zero-sample behavior perception method. Background Art

[0002] At present, most of the behavior recognition methods based on millimeter-wave radar signals use the word vector method of semantic attributes, or use the traditional convolutional neural network-long short-term memory model (CNN-LSTM) to extract millimeter-wave radar features from the distance-velocity map information (RV); however, since the sample categories of the test set of the existing methods must have appeared in the training set, when encountering unseen test categories, the existing methods will fail. For this problem, the solution in the existing technology is to collect more action categories and train the model; however, this method has high data collection and model training costs. Unlike millimeter-wave radar signals, visual signals can come from photos, videos, YouTube websites, and many large public data sets in daily life, so they contain rich action categories. How to use existing visual signals to assist millimeter-wave radar perception has important research value and provides the possibility for zero-sample behavior perception of millimeter-wave radar based on vision assistance. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a zero-sample millimeter-wave radar behavior perception method based on vision assistance, which solves the problem in the prior art that unseen category millimeter-wave radar signals cannot be perceived, and aims to realize the zero-sample millimeter-wave radar perception task.

[0004] The present invention provides a vision-assisted millimeter-wave radar zero-sample behavior perception method, comprising the following steps:

[0005] S1. Collect video data and millimeter-wave radar data containing various action categories at the same time, construct video datasets and millimeter-wave radar datasets respectively, and divide the millimeter-wave radar dataset into a training set and a test set, wherein the categories of the training set and the test set do not overlap;

[0006] S2, fine-tune the pre-trained C3D network using the video dataset, extract the last fully connected layer features from the C3D network, and average the features of the same category as the global attributes related to vision;

[0007] S3, convert the data of the video dataset into a time series of skeleton points, describe the coordinates of different key positions of the human body over time, obtain the time series changes of the angles of the connecting parts of the body according to the coordinates, and characterize the signal characteristics of different categories as local attributes related to vision;

[0008] S4. Integrate the vision-related attributes obtained in S2 and S3 to obtain the final attributes for each category.

[0009] S5. Construct a three-channel convolutional neural network to extract the feature expressions with strong discriminability from the millimeter-wave radar signals.

[0010] S6. Based on the training set, obtain the range-velocity, range-horizontal angle, and range-elevation angle based on the millimeter-wave radar signals, input them into the three-channel convolutional neural network to obtain the feature expressions of the millimeter-wave radar based on the training set; use the features of this millimeter-wave radar and the attributes of the corresponding categories to construct a loss function, obtain the projection relationship between the two, and train the parameters of the three-channel convolutional neural network.

[0011] S7. Based on the test set, obtain the range-velocity, range-horizontal angle, and range-elevation angle based on the millimeter-wave radar signals, input them into the trained three-channel convolutional neural network to obtain the feature expressions of the millimeter-wave radar based on the test set, and compare the features of this millimeter-wave radar with the attributes of each different category, and regard the category with the closest distance as the category of this millimeter-wave radar signal.

[0012] Furthermore, in S1, it is stipulated that there are u action categories, each action category contains several samples, and at the same time, collect the videos and millimeter-wave radar data containing these actions to construct a video dataset and a millimeter-wave radar dataset ; among them, the video dataset is used to generate u action attributes, and then assist the millimeter-wave radar for zero-shot learning; divide the millimeter-wave radar dataset, and the millimeter-wave radar data of v categories constitute the training set , where v < u; the data of the remaining u - v categories constitute the test set , and .

[0013] Furthermore, S2 is specifically as follows:

[0014] Assume that the C3D network pre-trained on a large-scale visual behavior recognition dataset is , fine-tune the parameters of using the actually collected video dataset to obtain ; then, use to extract the features of the last fully connected layer for each sample in the dataset; assume that the feature representation of the th sample in the nd class is , , then the average feature of the th class action is expressed as:

[0015] ,

[0016] in, Indicates The number of class samples, .

[0017] Furthermore, S3 is specifically:

[0018] The positions of the skeleton points generated from the video data are expressed as , respectively using The coordinates of the upper or lower limb joints are used to calculate the angles of the joints, thereby describing the changing process of different actions. The calculation formula is as follows:

[0019] ,

[0020] in, represents the angle at the tth moment; They represent the coordinates of the mth point, the nth point, and the oth point at time t, respectively. , , ,and ; There are 8 different combinations, resulting in 8 different angles. The 8 angle vectors at each moment are used as the features of that moment, and the T time series are constructed. The dimensional features are input into the convolutional neural network (general convolutional neural network, such as resnet, etc.), the features are extracted to describe the characteristics of the skeleton points of each sample changing over time, and the features of the same category are averaged to obtain the first Characteristics of class actions .

[0021] Furthermore, S4 is specifically:

[0022] The attention mechanism is used to fuse the two visual attributes of video and skeleton points to obtain the final attributes of each category. :

[0023] ,

[0024] Among them, w represents the weight coefficient.

[0025] Furthermore, in S5, the three-channel convolutional neural network includes three parallel multi-layer convolutional neural layers, and the distance-pitch angle and the distance-horizontal angle are down-sampled along the time axis and input into the first and second multi-layer convolutional neural networks respectively to generate corresponding feature maps, and the two feature maps are connected; at the same time, the third multi-layer convolutional neural network is used to generate a feature map for the distance-speed signal, and the feature map is fused with the connection results of the first and second multi-layer convolutional neural layers; finally, the fused feature map is input into the fourth multi-layer convolutional layer, and then through the fully connected layer to generate the feature expression of the millimeter wave radar.

[0026] Furthermore, in S6, the loss function constructed is:

[0027] ,

[0028] in, Refers to the yth i categories, ; Indicates The semantic attributes of the training samples of the class, N represents the The number of training samples for each class, represents the cross entropy loss; Represents the characteristic expression of the i-th millimeter-wave radar signal; Represents projection error, mapping radar features to visual semantic attributes; represents the cross entropy loss for predicting activity types from radar features; represents the trade-off parameter between projection loss and prediction loss.

[0029] Furthermore, S7 is specifically as follows: based on the test set data, the distance-speed, distance-horizontal angle, and distance-pitch angle based on the millimeter-wave radar signal are obtained, and the distance-speed, distance-horizontal angle, and distance-pitch angle are input into the trained three-channel convolutional neural network to obtain the feature expression of the millimeter-wave radar based on the test set; then the cosine similarity is used to calculate the similarity between the millimeter-wave radar features and attributes, and the closest attribute is selected by comparing one by one, and the category corresponding to the attribute is regarded as the action category of the millimeter-wave radar data.

[0030] The beneficial effects described in the present invention are as follows: the method described in the present invention uses rich visual samples to generate attributes of various categories of actions, designs a new three-channel convolutional neural network to extract millimeter-wave radar features with strong recognition, and constructs the relationship between attributes and millimeter-wave radars to achieve zero-sample learning tasks based on millimeter-wave radars. In the present invention, the three-channel convolutional neural network can integrate data of different dimensions to improve perception accuracy. Without collecting new categories of millimeter-wave radar signals for training, the perception of this category can be achieved, which can reduce data collection and training costs and broaden the application field of millimeter-wave radars. Description of the Drawings

[0031] Figure 1 is a schematic flowchart of the millimeter-wave radar zero-shot behavior perception method based on visual assistance according to an embodiment of the present invention;

[0032] Figure 2 is a schematic structural diagram of a novel three-channel convolutional neural network in the embodiment;

[0033] Figure 3 is a comparison chart of the effects of the method of the present invention and the prior art method. Detailed Embodiments

[0034] In order to make the content of the present invention more clearly understood, the present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0035] The embodiment provides a millimeter-wave radar zero-shot behavior perception method based on visual assistance, as Figure 1 including the following steps:

[0036] S1. Simultaneously collect video data and millimeter-wave radar data containing various different action categories, and respectively construct a video data set and a millimeter-wave radar data set; the video data set is used to generate attributes of various different actions; the millimeter-wave radar data set is divided into a training set and a test set, where the categories of the training set and the test set do not overlap.

[0037] In S1, it is stipulated that there are u action categories, and at the same time, videos and millimeter-wave radar data containing these actions are collected, which are respectively represented as and ; the video data set is used to generate u action attributes, thereby assisting the millimeter-wave radar to perform zero-shot learning. For the millimeter-wave radar data, the data of v (v < u) categories serve as the training set , and the data of the remaining u - v categories serve as the test set , and .

[0038] S2. Use the collected video data set to fine-tune the pre-trained C3D network, and extract the features of the last fully connected layer from the network, and take the average value of the features of the same category as the visually related global attribute.

[0039] Assume that the C3D network pre-trained on a large visual behavior recognition data set is , use the video data set actually collected by us to fine-tune parameters to obtain ; then, use to extract the features of the last fully connected layer for each sample of the data set; assume the first No. The characteristics of a sample are expressed as , , then The average value of the feature of the class action It is expressed as:

[0040] ,

[0041] in, Indicates The number of class samples, .

[0042] S3. Convert the collected video data set data into a time series of skeleton points, describe the coordinates of different key positions of the human body over time, calculate the time series changes of the angles between the connection vectors, and characterize the signal characteristics of each different category as vision-related local attributes.

[0043] The positions of the skeleton points (including the neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, bottom of the torso, left thigh, left knee, left ankle, right thigh, right knee, and right ankle) are generated from the video data and are represented as ; Further use The coordinates of the upper or lower limb joints are used to calculate the angles of the joints, thereby describing the changing process of different actions. The calculation formula is as follows:

[0044] ,

[0045] in, represents the angle at the tth moment; There are 8 different combinations, and 8 different angles are obtained. For the upper limbs, the combinations of (m, n, o) are (3, 2, 1), (4, 3, 2), (6, 5, 1), (7, 6, 5); for the lower limbs, the combinations of (m, n, o) are (10, 9, 8), (11, 10, 9), (13, 12, 8), (14, 13, 12). The 8 angle vectors at each moment are used as the features of that moment, and the T time series are constructed. The dimensional features are input into the convolutional neural network, and the features are extracted to describe the characteristics of the skeleton points of each sample changing over time, and the features of the same category are averaged to obtain the first Characteristics of class actions .

[0046] S4, merge the visual related attributes obtained in S2 and S3 to obtain the final attributes of each category;

[0047] In S4, the attention mechanism is used to fuse the two visual attributes of video and skeleton points to obtain the final attributes of each category. :

[0048] ,

[0049] Among them, w represents the weight coefficient.

[0050] S5, construct a three-channel convolutional neural network, such as Figure 2 As shown, it is used to extract the feature expression with strong recognition power in millimeter wave radar signals.

[0051] The distance-speed, distance-horizontal angle, and distance-elevation angle feature maps are obtained from the original millimeter-wave radar signal, and these feature maps are preprocessed such as denoising; then, they are input into a three-channel convolutional neural network to extract millimeter-wave radar features. Specifically, assuming that the preprocessed distance-speed, distance-horizontal angle, and distance-elevation angle feature maps are expressed as , where a, b, and T represent the height, width, and time series length of the data, respectively. , Along the time dimension, downsampling is used to reduce the dimension and generate ( ), Refers to the length of the time series after dimensionality reduction; then, design a multi-layer convolutional neural network Acts on , Extract angle feature F angle :

[0052] ,

[0053] in, Represents a connector; similarly, the speed characteristics can be obtained:

[0054] ,

[0055] Finally, fusion and The features of the millimeter-wave radar signal are generated by

[0056] S6. During the training process, the distance-speed, distance-horizontal angle, and distance-pitch angle based on the millimeter-wave radar signal are obtained based on the training set, and are input into the three-channel convolutional neural network to obtain the feature expression of the millimeter-wave radar based on the training set; a loss function is constructed using the feature expression of the millimeter-wave radar and the attributes of the corresponding category, a projection relationship between the two is constructed, and the parameters of the three-channel convolutional neural network are trained.

[0057] The model is trained using data from predefined v known categories, the features of the millimeter-wave radar are projected to the corresponding attributes, and the loss function L is constructed:

[0058] ,

[0059] in, Refers to the yth i categories, ; Indicates The semantic attributes of the training samples of the class, N represents the The number of training samples for each class, represents the cross entropy loss; represents the feature expression of the i-th millimeter-wave radar signal; the first term represents the projection error, which maps the radar features to visual semantic attributes; the second term represents the cross entropy loss for predicting the activity type from the radar features; represents the trade-off parameter between projection loss and prediction loss.

[0060] S7. During the test, the distance-speed, distance-horizontal angle, and distance-pitch angle based on the millimeter-wave radar signal are obtained based on the test set, and are input into the trained three-channel convolutional neural network to obtain the feature expression of the millimeter-wave radar based on the test set, and the feature expression of the millimeter-wave radar is compared with the attributes of each different category, and the category with the closest distance is regarded as the category of the millimeter-wave radar signal.

[0061] The cross-domain wireless behavior perception method based on the meta-learning training mechanism of the embodiment is experimentally verified as follows:

[0062] In the laboratory scene, 27 types of human activity samples were collected, 21 of which were used as known categories for model training, and the remaining 6 were used as unknown categories for performance testing. The visual-assisted millimeter-wave radar zero-sample behavior perception method of the embodiment was compared with two commonly used methods in the prior art. The experimental results are shown in Figure 2. Figure 3 As shown. Figure 3 The results show that when the semantic attributes are obtained by using a word vector method or a traditional convolutional neural network-long short-term memory model (CNN-LSTM) method for extracting millimeter-wave radar features from distance-velocity graph information (RV), the performance is lower than that of the embodiment method.

[0063] The method described in the present invention uses the existing rich visual signals to characterize the attribute characteristics of different categories, and uses them to construct the correlation between millimeter-wave radar signals and categories, innovatively proposes an attribute construction scheme and a three-channel millimeter-wave radar feature extraction scheme, and realizes a reliable zero-sample millimeter-wave radar perception algorithm. This framework can realize the perception of this category without collecting new categories of millimeter-wave radar signals for training, which can reduce data collection and training costs, broaden the application field of millimeter-wave radar, and promote the application of millimeter-wave radar perception from theory to practice.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. A vision-assisted millimeter-wave radar zero-sample behavior perception method, characterized in that: The following steps are involved: S1. Collect video data and millimeter-wave radar data containing various action categories at the same time, construct video datasets and millimeter-wave radar datasets respectively, and divide the millimeter-wave radar dataset into a training set and a test set, wherein the categories of the training set and the test set do not overlap; S2, fine-tune the pre-trained C3D network using the video dataset, extract the last fully connected layer features from the C3D network, and average the features of the same category as the global attributes related to vision; S3, convert the data of the video dataset into a time series of skeleton points, describe the coordinates of different key positions of the human body over time, obtain the time series changes of the angles of the connecting parts of the body according to the coordinates, and characterize the signal characteristics of different categories as local attributes related to vision; S4, merge the visual related attributes obtained in S2 and S3 to obtain the final attributes of each category; S5. Construct a three-channel convolutional neural network to extract the feature expression with strong recognition power in millimeter wave radar signals; The three-channel convolutional neural network includes three parallel multi-layer convolutional neural layers, and the distance-pitch angle and the distance-horizontal angle are downsampled along the time axis and input into the first and second multi-layer convolutional neural networks to generate corresponding feature maps, and the two feature maps are connected; at the same time, the third multi-layer convolutional neural network is used to generate a feature map for the distance-speed signal, and the feature map is fused with the connection results of the first and second multi-layer convolutional neural layers; finally, the fused feature map is input into the fourth multi-layer convolutional layer, and then through the fully connected layer to generate the feature expression of the millimeter wave radar; S6. Based on the training set, obtain the distance-speed, distance-horizontal angle, and distance-pitch angle based on the millimeter-wave radar signal, input them into the three-channel convolutional neural network, and obtain the feature expression of the millimeter-wave radar based on the training set; construct a loss function using the features of the millimeter-wave radar and the attributes of the corresponding category, obtain the projection relationship between the two, and train the parameters of the three-channel convolutional neural network; S7. Based on the test set, obtain the distance-speed, distance-horizontal angle, and distance-pitch angle based on the millimeter-wave radar signal, input them into the trained three-channel convolutional neural network, obtain the feature expression of the millimeter-wave radar based on the test set, and compare the features of the millimeter-wave radar with the attributes of each different category, and regard the category with the closest distance as the category of the millimeter-wave radar signal.

2. The method for zero-sample behavior perception of millimeter-wave radar based on vision assistance according to claim 1 is characterized in that: In S1, there are u types of action categories specified. Each action category contains a number of samples. At the same time, videos and millimeter-wave radar data containing these actions are collected to construct a video dataset and a millimeter-wave radar dataset ; among them, the video dataset is used to generate u action attributes, and then assist the millimeter-wave radar in zero-shot learning; the millimeter-wave radar dataset is divided, and the millimeter-wave radar data of v categories constitute the training set , where v < u; the data of the remaining u - v categories constitute the test set , and .

3. The method for zero-sample behavior perception of millimeter-wave radar based on vision assistance according to claim 2 is characterized in that: S2 is specifically: Assume that the C3D network is pre-trained on a large visual behavior recognition dataset, and the parameters of the C3D network are fine-tuned using the actual collected video dataset to obtain updated network parameters; then, the updated network parameters are used to Each sample in the dataset extracts the last fully connected layer features; assuming that No. The characteristics of a sample are expressed as , , then The average value of the feature of the class action It is expressed as: ， in, Indicates The number of class samples, .

4. The method for zero-sample behavior perception of millimeter-wave radar based on vision assistance according to claim 3 is characterized in that: S3 is specifically: The positions of the skeleton points generated from the video data are expressed as , respectively using The coordinates of the upper or lower limb joints are used to calculate the angles of the joints, thereby describing the changing process of different actions. The calculation formula is as follows: ， in, represents the angle at the tth moment; They represent the coordinates of the mth point, the nth point, and the oth point at time t, respectively. , , ,and ; There are 8 different combinations, resulting in 8 different angles. The 8 angle vectors at each moment are used as the features of that moment, and the T time series are constructed. The dimensional features are input into the convolutional neural network, and the features are extracted to describe the characteristics of the skeleton points of each sample changing over time, and the features of the same category are averaged to obtain the first Characteristics of class actions .

5. The method for zero-sample behavior perception of millimeter-wave radar based on vision assistance according to claim 4 is characterized in that: S4 is specifically: The attention mechanism is used to fuse the two visual attributes of video and skeleton points to obtain the final attributes of each category. : ， Among them, w represents the weight coefficient.

6. The method of zero-sample behavior perception of millimeter-wave radar based on vision assistance according to claim 2 is characterized in that: In S6, the constructed loss function is: ， in, Refers to the yth i categories, ; Indicates The semantic attributes of the training samples of the class, N represents the The number of training samples for each class, represents the cross entropy loss; Represents the characteristic expression of the i-th millimeter-wave radar signal; Represents projection error, mapping radar features to visual semantic attributes; represents the cross entropy loss for predicting activity types from radar features; represents the trade-off parameter between projection loss and prediction loss.

7. The method for zero-sample behavior perception of millimeter-wave radar based on vision assistance according to claim 6 is characterized in that: S7 is specifically: Based on the test set data, the distance-speed, distance-horizontal angle, and distance-pitch angle based on the millimeter-wave radar signal are obtained and input into the trained three-channel convolutional neural network to obtain the feature expression of the millimeter-wave radar based on the test set. Then, the cosine similarity is used to calculate the similarity between the millimeter-wave radar features and attributes. By comparing one by one, the closest attribute is selected, and the category corresponding to the attribute is regarded as the action category of the millimeter-wave radar data.

Images