Radar echo human body action recognition method based on transfer learning

Abstract

The invention discloses a radar echo human motion recognition method based on transfer learning, and relates to the field of artificial intelligence, and the method comprises the specific steps: obtaining motion echo signals, converting each motion echo signal into a corresponding micro-Doppler time-frequency characteristic pattern through short-time Fourier transform, and constructing a source domain sample set and a target domain sample set; constructing and training a deep convolutional neural network comprising a first feature extractor and a first label classifier; constructing and training an adversarial migration neural network comprising a second feature extractor and a first domain classifier; sequentially connecting the trained second feature extractor and the trained first label classifier in series to form a deep neural network; and obtaining the motion type of the motion echo signal to be identified by using a deep neural network. According to the method, the dependence on the data volume of the target domain is effectively reduced, cross-domain feature extraction and domain invariance are realized, and the generalization ability of the deep neural network in different scenes is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a method for recognizing human body actions from radar echoes based on transfer learning. Background Art

[0002] When a human body moves, the speeds and spatial positions of various limb parts change, and these changes cause changes in the target speed or radial distance in the echo signal. Therefore, based on ultra-wideband radar, the changes in the echo signal within a certain time range can be detected to achieve action detection and action information extraction. However, the acquisition standards, preprocessing methods, and background clutter of echo signals for different actions make the echo data distributions from different sources inconsistent and lack homogeneity, resulting in poor recognition accuracy of deep learning models on different echo data sets. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides a method for recognizing human body actions from radar echoes based on transfer learning.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for recognizing human body actions from radar echoes based on transfer learning constructs a deep neural network for recognizing the action types corresponding to action echo signals according to steps S1 to S4, and realizes the recognition of the action types corresponding to action echo signals according to step A:

[0006] Step S1: Obtain a preset number of human body action echo signals, each of which is known to correspond to an action type label. Preprocess each action echo signal, and use the short-time Fourier transform to convert each preprocessed action echo signal into a corresponding micro-Doppler time-frequency feature map. Divide each micro-Doppler time-frequency feature map into source domain data and target domain data according to a preset ratio to form a source domain sample set with a single micro-Doppler time-frequency feature map and its corresponding action type label as a sample, and a target domain sample set with a single micro-Doppler time-frequency feature map as a sample;

[0007] Step S2: Construct a deep convolutional neural network including a first feature extractor and a first label classifier, and train the deep convolutional neural network with each micro-Doppler time-frequency feature map in the source domain sample set as the input and the action type label corresponding to the micro-Doppler time-frequency feature map in the source domain sample set as the output;

[0008] Step S3: Construct an adversarial transfer neural network including a second feature extractor and a first domain classifier. Based on the same structures of the first feature extractor and the second feature extractor, and using the weights of the first feature extractor in the trained deep convolutional neural network as the initial weights of the second feature extractor, train the adversarial transfer neural network with the micro-Doppler time-frequency feature maps in the source domain sample set and the target domain sample set as inputs and the corresponding domain classification labels of the micro-Doppler time-frequency feature maps in the source domain sample set and the target domain sample set as outputs;

[0009] Step S4: Connect in series the second feature extractor in the trained adversarial transfer neural network and the first label classifier in the trained deep convolutional neural network to form a deep neural network with the micro-Doppler time-frequency feature map as the input and the corresponding action type label of the micro-Doppler time-frequency feature map as the output;

[0010] Step A: Use the short-time Fourier transform to convert the echo signal of the action to be recognized into the corresponding micro-Doppler time-frequency feature map, and input it into the deep neural network to output the action type of the echo signal of the action to be recognized.

[0011] Further, the preprocessing method in step S1 includes: direct wave cancellation, clutter suppression, and noise filtering.

[0012] Further, the input end of the first feature extractor constitutes the input end of the deep convolutional neural network, the input end of the first label classifier is connected to the output end of the first feature extractor, and the output end of the first label classifier constitutes the output end of the deep convolutional neural network;

[0013] The first feature extractor receives the input image and uses a sequentially connected convolutional stack layer, residual network layer, feature dimensionality reduction layer, and average pooling layer to obtain the deep features of the input image; the first label classifier includes a fully connected layer and a Softmax layer connected in sequence, which are used to obtain the probability distribution of the action type corresponding to the input image, and the classification result with the largest probability is taken as the recognition classification result.

[0014] Further, the first domain classifier includes two fully connected layers and a Softmax layer connected in sequence, which are used to obtain the domain classification label of the input sample and are connected to the second feature extractor through a gradient reversal layer to optimize the second feature extractor.

[0015] Further, during the training process of the deep convolutional neural network, the loss function is:

[0016]

[0017] where, is the loss function of the deep convolutional neural network, g is the first feature extractor, θg is the weight of the first feature extractor, c is the first label classifier, θ c is the weight of the first label classifier, x i is the i-th micro-Doppler time-frequency feature map, y i is the action type label corresponding to the i-th micro-Doppler time-frequency feature map.

[0018] Furthermore, during the training process of the adversarial transfer neural network, the loss function is:

[0019]

[0020] where is the loss function of the adversarial transfer neural network, g’ is the second feature extractor, θ g ' is the weight of the second feature extractor, d is the first domain classifier, θ d is the weight of the first domain classifier, x i is the i-th micro-Doppler time-frequency feature map, d i is the domain corresponding to the i-th micro-Doppler time-frequency feature map, i.e., the source domain or the target domain.

[0021] Beneficial effects brought by adopting the above technical solution:

[0022] The present invention uses an adversarial transfer neural network to homogenize the features of the source domain radar echo signal and the target domain radar echo signal, realizes cross-domain feature extraction and domain invariance, and improves the generalization ability of the model for radar echo human action recognition in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is the flow chart of the present invention;

[0024] Figure 2 is the network structure diagram of the first label classifier of the present invention;

[0025] Figure 3 is the learning curve of the correct rate of the application example of the present invention;

[0026] Figure 4 is the network structure diagram of the first domain classifier of the present invention;

[0027] Figure 5 is the flow chart of the adversarial learning of the present invention;

[0028] Figure 6 is the network structure diagram of the deep neural network of the present invention;

[0029] Figure 7 is the flow chart for recognizing the action echo signal of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0031] Reference Figure 1 , a radar echo human action recognition method based on transfer learning, constructs a deep neural network for recognizing the action type corresponding to the action echo signal according to steps S1 to S4, and realizes the recognition of the action type corresponding to the action echo signal according to step A:

[0032] Step S1, using an ultra-wideband radar, obtain various human action echo signals with a preset quantity and each known corresponding action type label respectively, preprocess each action echo signal, convert each preprocessed action echo signal into a corresponding micro-Doppler time-frequency feature map by using short-time Fourier transform, and divide each micro-Doppler time-frequency feature map into source domain data and target domain data according to a preset ratio, forming a source domain sample set with a single micro-Doppler time-frequency feature map and its corresponding action type label as samples, and a target domain sample set with a single micro-Doppler time-frequency feature map as samples;

[0033] Step S2, reference Figure 2 , construct a deep convolutional neural network including a first feature extractor and a first label classifier, and use each micro-Doppler time-frequency feature map in the source domain sample set as the input and the action type label corresponding to the micro-Doppler time-frequency feature map in the source domain sample set as the output to train the deep convolutional neural network; in this embodiment, the Xavier initialization method is used to initialize the parameters in the deep convolutional neural network, and the upsampling method is used to randomly increase the number of samples corresponding to the target action type during the training stage to make the number of samples in different categories balanced; under the optimal hyperparameter settings, the learning curves of the accuracy rate of supervised training and cross-domain training of the source domain data set are as Figure 3 shown. It can be seen from the figure that as the number of iterations increases, the accuracy rate curves on the training and validation sets also increase with the increase of the number of iterations and tend to be stable, that is, the settings of various parameters of the model are relatively reasonable, and there is no phenomenon of overfitting and underfitting;

[0034] Step S3, reference Figure 4 and Figure 5, construct an adversarial transfer neural network including a second feature extractor and a first domain classifier. Based on the same structures of the first feature extractor and the second feature extractor, and using the weights of the first feature extractor in the trained deep convolutional neural network as the initial weights of the second feature extractor, and randomly determining the initial weights of the first domain classifier, train the adversarial transfer neural network with the micro-Doppler time-frequency feature maps in the source domain sample set and the target domain sample set as inputs and the domain classification labels corresponding to the micro-Doppler time-frequency feature maps in the source domain sample set and the target domain sample set as outputs; in this embodiment, the first domain classifier is connected to the second feature extractor through a gradient reversal layer. When performing backpropagation gradient learning, multiply the gradient by a certain negative constant λ to enable the second feature extractor to learn domain-invariant general features. When the adversarial reaches an equilibrium point, the second feature extractor can extract the shared features of the source domain and the target domain from the input samples, ensuring that the feature distribution has domain invariance;

[0035] Step S4, refer to Figure 6 , sequentially connect in series the second feature extractor in the trained adversarial transfer neural network and the first label classifier in the trained deep convolutional neural network to form a deep neural network with the micro-Doppler time-frequency feature map as the input and the action type label corresponding to the micro-Doppler time-frequency feature map as the output;

[0036] Step A, refer to Figure 7 , use the short-time Fourier transform to convert the echo signal of the action to be recognized into the corresponding micro-Doppler time-frequency feature map, and input it into the deep neural network to output the action type of the echo signal of the action to be recognized; in this embodiment, the short-time Fourier transform selects the Hamming window as the window function, the number of overlapping samples in each segment accounts for 50% of the total number of samples, and the number of operation points of the Fourier transform is set to 1024.

[0037] Furthermore, the preprocessing method in Step S1 includes: direct wave cancellation, clutter suppression, and noise filtering.

[0038] Furthermore, the input end of the first feature extractor constitutes the input end of the deep convolutional neural network, the input end of the first label classifier is connected to the output end of the first feature extractor, and the output end of the first label classifier constitutes the output end of the deep convolutional neural network;

[0039] The first feature extractor receives the input image and uses a sequentially connected convolutional stacking layer, a residual network layer, a feature dimensionality reduction layer, and an average pooling layer to obtain the deep features of the input image; the first label classifier includes a fully connected layer and a Softmax layer connected in sequence, which is used to obtain the probability distribution of the action type corresponding to the input image, and take the classification result with the largest probability as the recognition classification result, where the number of neurons in the Softmax layer is the number of action echo signal types.

[0040] Furthermore, the first domain classifier includes two fully connected layers and a Softmax layer connected in sequence, which is used to obtain the domain classification label of the input sample and is connected to the second feature extractor through a gradient reversal layer to optimize the second feature extractor, where the number of neurons in the Softmax layer is 2.

[0041] Furthermore, during the training process of the deep convolutional neural network, the loss function is:

[0042]

[0043] where, is the loss function of the deep convolutional neural network, g is the first feature extractor, θ g is the weight of the first feature extractor, c is the first label classifier, θ c is the weight of the first label classifier, x i is the i-th micro-Doppler time-frequency feature map, y i is the action type label corresponding to the i-th micro-Doppler time-frequency feature map.

[0044] Furthermore, during the training process of the adversarial transfer neural network, the loss function is:

[0045]

[0046] where, is the loss function of the adversarial transfer neural network, g’ is the second feature extractor, θ g ' is the weight of the second feature extractor, d is the first domain classifier, θ d is the weight of the first domain classifier, x i is the i-th micro-Doppler time-frequency feature map, d i is the domain corresponding to the i-th micro-Doppler time-frequency feature map, that is, the sample domain or the target domain.

[0047] The above are only the preferred embodiments of the present invention, and do not impose any limitations on the present invention. Any simple modifications, changes, and equivalent structural changes made to the above embodiments according to the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for human action recognition based on radar echo using transfer learning, characterized in that: Construct a deep neural network for identifying the action type corresponding to the action echo signal according to steps S1 to S4, and identify the action type corresponding to the action echo signal according to step A: Step S1: Obtain a preset number of human action echo signals, each of which is known to correspond to an action type label. Preprocess each action echo signal respectively, use the short-time Fourier transform to convert each preprocessed action echo signal into a corresponding micro-Doppler time-frequency feature map, and divide each micro-Doppler time-frequency feature map into source domain data and target domain data according to a preset ratio, forming a source domain sample set with a single micro-Doppler time-frequency feature map and its corresponding action type label as samples, and a target domain sample set with a single micro-Doppler time-frequency feature map as samples; Step S2: Construct a deep convolutional neural network including a first feature extractor and a first label classifier, and use each micro-Doppler time-frequency feature map in the source domain sample set as the input and the action type label corresponding to the micro-Doppler time-frequency feature map in the source domain sample set as the output to train the deep convolutional neural network; Step S3: Construct an adversarial transfer neural network including a second feature extractor and a first domain classifier. Based on the same structure of the first feature extractor and the second feature extractor, and using the weights of the first feature extractor in the trained deep convolutional neural network as the initial weights of the second feature extractor, use the micro-Doppler time-frequency feature maps in the source domain sample set and the target domain sample set as the input and the domain classification labels corresponding to the micro-Doppler time-frequency feature maps in the source domain sample set and the target domain sample set as the output to train the adversarial transfer neural network; Step S4: Sequentially connect the second feature extractor in the trained adversarial transfer neural network and the first label classifier in the trained deep convolutional neural network to form a deep neural network with the micro-Doppler time-frequency feature map as the input and the action type label corresponding to the micro-Doppler time-frequency feature map as the output; Step A: Use the short-time Fourier transform to convert the action echo signal to be identified into a corresponding micro-Doppler time-frequency feature map, and input it into the deep neural network to output the action type of the action echo signal to be identified.

2. The method for radar echo human action recognition based on transfer learning according to claim 1, wherein The preprocessing method described in step S1 includes: direct wave elimination, clutter suppression, and noise filtering.

3. The method for identifying human body movements from radar echoes based on transfer learning according to claim 1, characterized in that The input end of the first feature extractor constitutes the input end of the deep convolutional neural network, the input end of the first label classifier is connected to the output end of the first feature extractor, and the output end of the first label classifier constitutes the output end of the deep convolutional neural network; The first feature extractor receives the input image and uses a sequentially connected convolutional stacking layer, residual network layer, feature dimension reduction layer, and average pooling layer to obtain the deep features of the input image; the first label classifier includes a fully connected layer and a Softmax layer connected in sequence, which is used to obtain the probability distribution of the action type corresponding to the input image, and take the classification result with the largest probability as the recognition classification result.

4. The method for radar echo human action recognition based on transfer learning according to claim 1, wherein, The first domain classifier includes two fully connected layers and a Softmax layer connected in sequence, and is used to obtain the domain classification label of the input sample, and is connected to the second feature extractor through a gradient reversal layer to optimize the second feature extractor.

5. The method for radar echo human action recognition based on transfer learning according to claim 1, characterized in that, During the training process of the deep convolutional neural network, the loss function is: Among them, is the loss function of the deep convolutional neural network, g is the first feature extractor, and θ g is the weight of the first feature extractor, c is the first label classifier, and θ c is the weight of the first label classifier, x i is the i-th micro-Doppler time-frequency feature map, and y i is the action type label corresponding to the i-th micro-Doppler time-frequency feature map.

6. The method for radar echo human action recognition based on transfer learning according to claim 1, characterized in that, During the training process of the adversarial transfer neural network, the loss function is: Among them, is the loss function of the adversarial transfer neural network, g’ is the second feature extractor, and θ g ' is the weight of the second feature extractor, d is the first domain classifier, and θ d is the weight of the first domain classifier, x i is the i-th micro-Doppler time-frequency feature map, and d i is the domain corresponding to the i-th micro-Doppler time-frequency feature map, that is, the sample domain or the target domain.

Images