Continuous human body activity identification method based on FMCW radar multi-domain feature fusion

Abstract

The invention provides a continuous human body activity identification method based on FMCW radar multi-domain feature fusion, and the method comprises the steps: processing original data of a radar through employing a fast Fourier transform algorithm and a multi-signal classification technology, and obtaining a distance-Doppler image and a distance-angle image; the radar data are mapped to a fractional domain through short-time fractional Fourier transform, and a fractional domain spectrogram is obtained; then realizing synchronous segmentation of three types of continuous activity sequence domain information through a variable window length STA / LTA continuous action detection algorithm; and finally, taking the segmented activity sample as the input of a multi-input multi-task continuous activity recognition model, extracting input features through a convolutional network and a bidirectional long-term memory network, introducing a CTC layer to fuse multi-input representation, calculating CTC loss of a prediction sequence and a real label, executing multi-task learning, and obtaining a human body action recognition result. According to the method, continuous human body actions can be segmented at high precision, three types of radar domain information are accurately fused, and a high-precision continuous human body action recognition task is realized.

Description

Technical Field

[0001] The invention relates to the field of wireless sensing technology, in particular to a continuous human activity recognition method based on FMCW radar multi-domain feature fusion. Background Art

[0002] With the continuous advancement of science and technology and the rapid development of society, intelligent monitoring systems are playing an increasingly important role in security, intelligent transportation, health care and other fields. In these application scenarios, human activity recognition technology is a crucial link. It can monitor and identify various human activities in real time, provide the system with necessary information, and further enhance the system's intelligence level and ability to respond to emergencies. Human activity recognition refers to the process of identifying and classifying the movement state of the human body in space using sensor technology and data processing methods. At present, human activity recognition has become an active research field, involving the intersection of multiple disciplines and technical fields such as computer vision, pattern recognition, and machine learning, and has a wide range of applications in daily life and various scientific research fields.

[0003] At present, there has been great progress in the research of human posture recognition in vision, sensors and other aspects at home and abroad. However, the existing posture recognition methods have certain limitations. Human posture recognition methods based on vision are easily blocked by obstacles and easily affected by light; human posture recognition methods based on sensors usually need to be worn on the body or installed in the environment, which may affect the user's comfort and naturalness. Among the technical means to achieve human activity recognition, radar technology has attracted much attention due to its unique advantages. Compared with sensor-based methods, radar does not need to be worn on the body or installed in the environment, avoiding interference and inconvenience to users. In addition, radar technology is not restricted by environmental conditions such as light, shadow and occlusion. It is suitable for activity recognition in complex environments, such as night and bad weather. It can work all day and night, with a long detection distance and high positioning accuracy and stability. Compared with vision-based methods, radar technology does not need to obtain image or video data, and can better protect personal privacy.

[0004] However, despite the many advantages of FMCW radar, there are still some challenges to achieve accurate human activity recognition. First, human activities are often complex and diverse, including various actions such as walking, standing, and sitting, so the recognition algorithm needs to have high accuracy and generalization ability. Secondly, radar signals are affected by various interference factors during propagation, such as multipath effects and noise interference, which will reduce the performance and stability of the recognition system. Traditional radar signal processing methods can often only extract information in a single field, such as time domain or frequency domain information, which limits the accuracy and robustness of the recognition algorithm. Especially in complex and changing environments, the lack of single domain information may lead to the failure of the recognition algorithm. Multi-domain information fusion technology based on FMCW radar signals can comprehensively utilize radar signal information in multiple aspects such as time domain, frequency domain and angle domain, thereby improving the accuracy and robustness of target recognition and tracking. In addition, in real life, human activities are continuous and time-variable. This means that human activities not only involve different actions and postures, but also include transitions, intersections and overlaps. This complexity brings a series of difficulties and challenges to activity segmentation and classification. Summary of the invention

[0005] The present invention proposes a continuous human activity recognition method based on FMCW radar multi-domain feature fusion, which can segment continuous human movements with high precision and accurately fuse three types of radar domain information to achieve high-precision continuous human movement recognition tasks.

[0006] The present invention adopts the following technical solutions.

[0007] A continuous human activity recognition method based on FMCW radar multi-domain feature fusion is proposed. The method firstly processes the raw data collected by the radar and obtains the range-Doppler image and the range-angle image by using the fast Fourier transform algorithm and the multiple signal classification technology, and maps the radar data to the fractional domain by using the short-time fractional Fourier transform to obtain the fractional domain spectrogram; then the variable window length STA / LTA continuous action detection algorithm is used to realize the synchronous segmentation of three types of continuous activity sequence domain information; finally, the segmented activity samples are used as the input of the multi-input multi-task continuous activity recognition model, the input features are extracted by the convolutional network and the bidirectional long-term memory network, the CTC layer is introduced to fuse the multi-input representation, and the CTC loss of the predicted sequence and the true label is calculated, and the multi-task learning is performed to obtain the human action recognition result.

[0008] The method comprises the following steps:

[0009] Step S1: Use FMCW radar to collect original human posture data, perform two-dimensional fast Fourier transform on the slow-time / fast-time data matrix of a single receiving channel of the radar system to obtain a range-Doppler image. Organize multiple receiving channels according to time delay and Doppler frequency, calculate the spatial covariance matrix of the received signal and the spatial spectrum distribution of each angle, and construct a range-angle image;

[0010] Step S2: applying the cell average constant false alarm rate method to the range-Doppler image to detect the range of motion; for the detected range interval, applying multiple short-time fractional Fourier transforms in the slow time dimension to obtain a fractional domain spectrum;

[0011] Step S3: performing activity detection based on variable window STA / LTA on the range-Doppler image, the range-angle image, and the fractional domain spectrogram containing continuous human activity information, segmenting the continuous human activity radar image to identify the starting point and end point of the human activity;

[0012] Step S4: Input the segmented human activity data into the convolutional neural network for feature extraction, then use the bidirectional long short-term memory layer to model the three-way features, and use the reshape and concatenate functions to merge the multi-way outputs. The fused high-dimensional human action tensor is output through the connection time series classification method to obtain the human activity classification result.

[0013] In step S1, the range-Doppler image and the range-angle image are obtained by the following specific steps: Step S11: After the radar receives the echo signal, it is sampled and a slow time / fast time data matrix is ​​formed; the slow time represents the time series of CPI, and the fast time represents the time sampling within each CPI; it is assumed that the data matrix is ​​X, and the size is M×N, where M represents the slow time dimension (the duration of CPI), and N represents the fast time dimension, that is, the number of time sampling within each CPI;

[0014] Before performing 2D FFT, a window function is applied to the data matrix to reduce the spectrum leakage caused by the boundary; after applying the window function, the data matrix X after window function processing is obtained. w , the formula is:

[0015] X w (i,j)=X(i,j)×w(i,j)

[0016] Among them, X w (i, j) represents the element of the data matrix after window function processing, X(i, j) represents the element of the original data matrix, and w(i, j) represents the value of the window function;

[0017] After the data matrix X is processed by the window function wPerform a 2D FFT to convert the time domain into the frequency domain and obtain the range-Doppler plane, which is represented as a range-Doppler image:

[0018] Y(u,v)=FFT2(X w )

[0019] Among them, Y(u,v) represents the result after 2D FFT, the size is M×N, u and v represent the index in the frequency domain;

[0020] In addition, the range-Doppler images of multiple CPIs are accumulated and averaged to enhance the target signal:

[0021]

[0022] Among them, Y avg (u,v) represents the result after cumulative averaging, N represents the number of accumulated frames, and Y(u,v) represents the range-Doppler image of each frame;

[0023] The processed and enhanced range-Doppler images are combined to form an RD video; each image represents the range-Doppler information of one CPI;

[0024] Step S12: The radar system collects echo signals through multiple receiving channels, each receiving channel corresponds to an antenna or array element, which is used to receive signals from different directions; next, for each CPI, the received signal data is processed; first, the data is organized according to delay and Doppler frequency to form a delay-Doppler matrix; for i receiving channels, the received signal data is represented as a complex sequence: [s1(i), s2(i), ..., s N (i)], where N represents the number of sampling points; these signals are organized according to delay and Doppler frequency to form a delay-Doppler matrix D i , whose elements can be expressed as

[0025] D i (t,f)=s i (t,f)

[0026] The rows of the matrix correspond to different angles, and the columns correspond to different Doppler frequencies; each element represents the received signal strength at a specific angle and Doppler frequency;

[0027] Based on the delay-Doppler matrix, the spatial covariance matrix R of the received signal is calculated; the spatial covariance matrix describes the signal correlation between different receiving channels; the delay-Doppler matrices of all receiving channels are spliced ​​by column to form a total delay-Doppler matrix D, whose size is N×M, where N represents the delay dimension and M represents the Doppler frequency dimension. The spatial covariance matrix R is obtained by calculating the conjugate transpose of D and the inner product of D:

[0028] R=D H D

[0029] Among them, D H represents the conjugate transpose of D;

[0030] Then, the spatial covariance matrix is ​​further subjected to eigendecomposition to extract its eigenvectors and eigenvalues: R(u k )=λ k u k

[0031] By calculating the spatial spectrum distribution of each angle, the arrival of the target at different angles is estimated. This is achieved using the multi-signal classification algorithm MUSIC, with the following formula:

[0032]

[0033] The MUSIC algorithm calculates the spatial spectrum distribution P(θ) at each angle θ through this formula; in this formula, a(θ) is the array manifold vector at angle θ, which describes the response pattern of the antenna array to a signal at a specific angle. K is the number of signals, and λ is k is the eigenvalue, u k is the corresponding eigenvector. The angular spectrum P(θ) represents the spatial distribution of the signal at different angles and is used to estimate the arrival angle of the target;

[0034] By combining the estimated angle spectrum with the range, a range-Doppler RA image is generated.

[0035] In step S12, by repeating the steps, each CPI is processed to obtain a series of RA images; these images are combined at a predetermined frame rate to form a RA video; and the changes in the angle and Doppler of the target, as well as the motion characteristics of the target, are observed by playing the RA video.

[0036] In step S2, the fractional domain spectrogram is obtained by the following specific steps:

[0037] Step S21: First, the cell average constant false alarm rate (CA-CFAR) method CA-CFAR is applied to the distance-Doppler map to detect the movement distance interval. For the detected distance interval, multiple short-time fractional Fourier transforms (STFrFT) are applied in the slow time dimension to obtain a fractional domain spectrogram.

[0038] Step S3 specifically includes the following steps:

[0039] Step S31: A variable window STA / LTA based activity detection algorithm is used to identify the start and end points of continuous motion. First, the absolute difference between the upper and lower envelopes of the time index is calculated based on the fractional domain spectrogram to create an absolute distance vector. The absolute distance i of each data record can be calculated as v i =|u i -l i |, where v i is the absolute distance vector, u i and l i They are upper and lower envelopes respectively;

[0040] Step S32: STA(t) and LTA(t) can be defined as the leading window and the lagging window of time t:

[0041]

[0042] Where T1 and T2 are the lengths of the short time window and the long time window respectively; the starting point of the detection activity is when the following conditions are met:

[0043]

[0044] Among them, σ1 and σ2 are predefined detection thresholds; similarly, if

[0045]

[0046] Where σ3 is the detection threshold of the stopping point;

[0047] In order to find the starting point, STA(t) needs to exceed the threshold σ1, which means that the activity must appear in the short window; at the same time, the average power ratio of the short window and the long window should be greater than σ2; so that in the presence of noise, the system will not be triggered unless the ratio exceeds σ2; ​​it also ensures that the stop point of the activity can be correctly detected, that is, when the activity disappears from the processing window and the ratio drops below the threshold σ2;

[0048] Step S33: Implement the variable window length of STA / LTA by introducing a mask; the mask is a zero vector with the same shape as the input vector; during the execution of the function, the index position under specific conditions will be set to 1 to indicate that the data at these positions has been processed or belongs to a specific category; when a specific event is detected, that is, when an activity occurs, the mask replaces 0 with 1 in the corresponding interval; create a zero vector mask with the same shape as the input vector and set its initial value to 0, which is used to mark a specific interval in the input vector after STA / LTA detection; then define a state variable with an initial value of 0, and by looping through the input vector, update the state variable and determine the start position and stop position according to the comparison of the long window value, the short window value, and the average power ratio of the short window and the long window with the above threshold, and mark the corresponding interval in the mask; Step S34: After using the variable window length STA / LTA detection algorithm to generate a detection mask to segment a single activity in the fractional domain spectrogram, the single activity in the range-Doppler video stream and the range-angle video stream also needs to be segmented synchronously; therefore, the demask algorithm is used to extract the interest interval marked by the specific mask matrix from the two-dimensional data matrix;

[0049] The demask algorithm accepts three input parameters: two-dimensional data matrices x and y, and a two-dimensional Boolean matrix mask used to indicate the data intervals of interest in x and y. The specific usage method is as follows: First, the algorithm initializes two empty lists x2 and y2 to collect the extracted data intervals; at the same time, a flag variable flag with an initial value set to 0 is introduced to track the starting point of the interval of interest; next, the algorithm traverses all rows and columns of the mask matrix; during the traversal process, if flag is 0 and the current position mask is 0, ignore the position and continue traversing; if flag is 1 and the current position mask is 1, and the number of consecutive values ​​1 from the starting point to the current position exceeds 2, it indicates the end of an interval of interest. At this time, the end position of the interval is recorded, and the corresponding x and y intervals are added to the x2 and y2 lists. When flag is 0 and the current position mask value is 1, set flag to 1 and set the current position as the starting point. On the contrary, if flag is 1 and the current position mask value is 0, reset flag to 0 and record the current position as the end position of the interval of interest. Finally, the algorithm returns arrays x2 and y2, containing the features and labels of the extracted data intervals, respectively.

[0050] Step S4 specifically includes the following steps:

[0051] Step S41: For the distance-angle RA and distance-Doppler RD information, first use 3D convolution and pooling layers to extract the spatiotemporal features of the video; the convolution layer applies 3D convolution operations to each time step of the input video sequence to extract the spatiotemporal features of the video; similarly, for the fractional domain spectrogram, two-dimensional convolution and pooling are used to extract spatiotemporal features. The outputs of the three convolution layers are then processed into the specified shape through the reshape function. The features are processed through the fully connected layer and the Dropout layer; then, the bidirectional LSTM layer is applied to the input sequence;

[0052] Step S42: After extracting the input features, the model uses the concatenate function to merge the penultimate layer outputs of the three-way input into a tensor, and then connects the merged tensor to an output layer with five categories and a softmax activation function through a fully connected layer;

[0053] Step S43: Finally, a CTC layer is defined to calculate the CTC loss and added after the output layer; the calculation formula of the CTC loss function is as follows:

[0054] Given an input sequence X and a corresponding label sequence Y, define Y′ as the label sequence with repeated characters and blank characters removed, that is, Y′={y|y∈Y,y≠blank}. Then the CTC loss is:

[0055] Loss CTC = -∑log(P(Y′|X))

[0056] Among them, P(Y′|X) represents the probability of predicting the output sequence to be Y′ given the input sequence X. During the training process, CTC uses a dynamic programming algorithm to calculate the conditional probability of the model output sequence and compares it with the actual output sequence to calculate the loss function. By minimizing the loss function, the model learns the correspondence between the input sequence and the output sequence, thereby making accurate sequence labeling predictions.

[0057] The recognition method collects continuous human motion radar data sets and performs human motion detection in the following way: a millimeter-wave radar is placed in front of the subject, and according to the different motion habits of each subject, an angle deviation within about 120° of the total field of view is allowed when the subject faces the millimeter-wave radar; the target of the collection is human motion, and the reflection cross-sectional area RCS of the human body facing the millimeter-wave radar is 0.5-1m 2 , making the distance detection range applicable to this recognition method 5-15 meters.

[0058] The recognition method collects five human body movements, namely, punching, walking, standing, sitting down, and standing up, and asks the subjects to perform corresponding movement demonstrations in a random combination of the five movements at a distance of 5 meters in front of the FMCW radar for data collection and movement detection.

[0059] Compared with the prior art, the present invention and its solution perform algorithmic analysis and processing on the original signal of FMCW radar, segment continuous human motion with high precision based on the activity detection algorithm with variable window length STA / LTA, and accurately fuse three types of radar domain information by building a multi-input multi-task learning neural network, thereby realizing the task of continuous human motion recognition with high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0061] Attached Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention;

[0062] Attached Figure 2 is a visualization schematic diagram of a method for forming multi-domain data of an FMCW radar according to an embodiment of the present invention;

[0063] Attached Figure 3 is a visualization diagram of a human motion data set according to an embodiment of the present invention;

[0064] Attached Figure 4 is a schematic diagram of a continuous action segmentation process according to an embodiment of the present invention;

[0065] Attached Figure 5 is a schematic diagram of a human posture recognition model according to an embodiment of the present invention;

[0066] Attached Figure 6 Schematic diagram of a feature extraction network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] As shown in the figure, a continuous human activity recognition method based on FMCW radar multi-domain feature fusion is described. The method first processes the raw data collected by the radar and obtains the range-Doppler image and the range-angle image by using the fast Fourier transform algorithm and the multiple signal classification technology, and maps the radar data to the fractional domain by using the short-time fractional Fourier transform to obtain the fractional domain spectrogram; then, the variable window length STA / LTA continuous action detection algorithm is used to realize the synchronous segmentation of the three types of continuous activity sequence domain information; finally, the segmented activity samples are used as the input of the multi-input multi-task continuous activity recognition model, and the input features are extracted through the convolutional network and the bidirectional long-term memory network. The CTC layer is introduced to fuse the multi-input representation, and the CTC loss of the predicted sequence and the true label is calculated to perform multi-task learning and obtain the human action recognition result.

[0068] The method comprises the following steps:

[0069] Step S1: Use FMCW radar to collect original human posture data, perform two-dimensional fast Fourier transform on the slow-time / fast-time data matrix of a single receiving channel of the radar system to obtain a range-Doppler image. Organize multiple receiving channels according to time delay and Doppler frequency, calculate the spatial covariance matrix of the received signal and the spatial spectrum distribution of each angle, and construct a range-angle image;

[0070] Step S2: applying the cell average constant false alarm rate method to the range-Doppler image to detect the range of motion; for the detected range interval, applying multiple short-time fractional Fourier transforms in the slow time dimension to obtain a fractional domain spectrum;

[0071] Step S3: performing activity detection based on variable window STA / LTA on the range-Doppler image, the range-angle image, and the fractional domain spectrogram containing continuous human activity information, segmenting the continuous human activity radar image to identify the starting point and end point of the human activity;

[0072] Step S4: Input the segmented human activity data into the convolutional neural network for feature extraction, then use the bidirectional long short-term memory layer to model the three-way features, and use the reshape and concatenate functions to merge the multi-way outputs. The fused high-dimensional human action tensor is output through the connection time series classification method to obtain the human activity classification result.

[0073] In step S1, the range-Doppler image and the range-angle image are obtained by the following specific steps: Step S11: After the radar receives the echo signal, it is sampled and a slow time / fast time data matrix is ​​formed; the slow time represents the time series of CPI, and the fast time represents the time sampling within each CPI; it is assumed that the data matrix is ​​X, and the size is M×N, where M represents the slow time dimension (the duration of CPI), and N represents the fast time dimension, that is, the number of time sampling within each CPI;

[0074] Before performing 2D FFT, a window function is applied to the data matrix to reduce the spectrum leakage caused by the boundary; after applying the window function, the data matrix X after window function processing is obtained. w , the formula is:

[0075] X w (i,j)=X(i,j)×w(i,j)

[0076] Among them, X w (i, j) represents the element of the data matrix after window function processing, X(i, j) represents the element of the original data matrix, and w(i, j) represents the value of the window function;

[0077] After the data matrix X is processed by the window function w Perform a 2D FFT to convert the time domain into the frequency domain and obtain the range-Doppler plane, which is represented as a range-Doppler image:

[0078] Y(u,v)=FFT2(X w )

[0079] Among them, Y(u,v) represents the result after 2D FFT, the size is M×N, and u and v represent the index in the frequency domain;

[0080] In addition, the range-Doppler images of multiple CPIs are accumulated and averaged to enhance the target signal:

[0081]

[0082] Among them, Y avg (u,v) represents the result after cumulative averaging, N represents the number of accumulated frames, and Y(u,v) represents the range-Doppler image of each frame;

[0083] The processed and enhanced range-Doppler images are combined to form an RD video; each image represents the range-Doppler information of one CPI;

[0084] Step S12: The radar system collects echo signals through multiple receiving channels, each receiving channel corresponds to an antenna or array element, which is used to receive signals from different directions; next, for each CPI, the received signal data is processed; first, the data is organized according to delay and Doppler frequency to form a delay-Doppler matrix; for i receiving channels, the received signal data is represented as a complex sequence: [s1(i), s2(i), ..., s N (i)], where N represents the number of sampling points; these signals are organized according to delay and Doppler frequency to form a delay-Doppler matrix D i , whose elements can be expressed as

[0085] D i (t,f)=s i (t,f)

[0086] The rows of the matrix correspond to different angles, and the columns correspond to different Doppler frequencies; each element represents the received signal strength at a specific angle and Doppler frequency;

[0087] Based on the delay-Doppler matrix, the spatial covariance matrix R of the received signal is calculated; the spatial covariance matrix describes the signal correlation between different receiving channels; the delay-Doppler matrices of all receiving channels are spliced ​​by column to form a total delay-Doppler matrix D, whose size is N×M, where N represents the delay dimension and M represents the Doppler frequency dimension. The spatial covariance matrix R is obtained by calculating the conjugate transpose of D and the inner product of D:

[0088] R=D H D

[0089] Among them, D H represents the conjugate transpose of D;

[0090] Then, the spatial covariance matrix is ​​further subjected to eigendecomposition to extract its eigenvectors and eigenvalues: R(u k )=λ k u k

[0091] By calculating the spatial spectrum distribution of each angle, the arrival of the target at different angles is estimated. This is achieved using the multi-signal classification algorithm MUSIC, with the following formula:

[0092]

[0093] The MUSIC algorithm calculates the spatial spectrum distribution P(θ) at each angle θ through this formula; in this formula, a(θ) is the array manifold vector at angle θ, which describes the response pattern of the antenna array to a signal at a specific angle. K is the number of signals, and λ is k is the eigenvalue, u k is the corresponding eigenvector. The angular spectrum P(θ) represents the spatial distribution of the signal at different angles and is used to estimate the arrival angle of the target;

[0094] By combining the estimated angle spectrum with the range, a range-Doppler RA image is generated.

[0095] In step S12, by repeating the steps, each CPI is processed to obtain a series of RA images; these images are combined at a predetermined frame rate to form a RA video; and the changes in the angle and Doppler of the target, as well as the motion characteristics of the target, are observed by playing the RA video.

[0096] In step S2, the fractional domain spectrogram is obtained by the following specific steps:

[0097] Step S21: First, the cell average constant false alarm rate (CA-CFAR) method CA-CFAR is applied to the distance-Doppler map to detect the movement distance interval. For the detected distance interval, multiple short-time fractional Fourier transforms (STFrFT) are applied in the slow time dimension to obtain a fractional domain spectrogram.

[0098] Step S3 specifically includes the following steps:

[0099] Step S31: A variable window STA / LTA based activity detection algorithm is used to identify the start and end points of continuous motion. First, the absolute difference between the upper and lower envelopes of the time index is calculated based on the fractional domain spectrogram to create an absolute distance vector. The absolute distance i of each data record can be calculated as v i =|u i -l i |, where v i is the absolute distance vector, u i and l i They are upper and lower envelopes respectively;

[0100] Step S32: STA(t) and LTA(t) can be defined as the leading window and the lagging window of time t:

[0101]

[0102] Where T1 and T2 are the lengths of the short time window and the long time window respectively; the starting point of the detection activity is when the following conditions are met:

[0103]

[0104] Among them, σ1 and σ2 are predefined detection thresholds; similarly, if

[0105]

[0106] Where σ3 is the detection threshold of the stopping point;

[0107] In order to find the starting point, STA(t) needs to exceed the threshold σ1, which means that the activity must appear in the short window; at the same time, the average power ratio of the short window and the long window should be greater than σ2; so that in the presence of noise, the system will not be triggered unless the ratio exceeds σ2; ​​it also ensures that the stop point of the activity can be correctly detected, that is, when the activity disappears from the processing window and the ratio drops below the threshold σ2;

[0108] Step S33: Implement the variable window length of STA / LTA by introducing a mask; the mask is a zero vector with the same shape as the input vector; during the execution of the function, the index position under specific conditions will be set to 1 to indicate that the data at these positions has been processed or belongs to a specific category; when a specific event is detected, that is, when an activity occurs, the mask replaces 0 with 1 in the corresponding interval; create a zero vector mask with the same shape as the input vector and set its initial value to 0, which is used to mark a specific interval in the input vector after STA / LTA detection; then define a state variable with an initial value of 0, and by looping through the input vector, update the state variable and determine the start position and stop position according to the comparison of the long window value, the short window value, and the average power ratio of the short window and the long window with the above threshold, and mark the corresponding interval in the mask; Step S34: After using the variable window length STA / LTA detection algorithm to generate a detection mask to segment a single activity in the fractional domain spectrogram, the single activity in the range-Doppler video stream and the range-angle video stream also needs to be segmented synchronously; therefore, the demask algorithm is used to extract the interest interval marked by the specific mask matrix from the two-dimensional data matrix;

[0109] The demask algorithm accepts three input parameters: two-dimensional data matrices x and y, and a two-dimensional Boolean matrix mask used to indicate the data intervals of interest in x and y. The specific usage method is as follows: First, the algorithm initializes two empty lists x2 and y2 to collect the extracted data intervals; at the same time, a flag variable flag with an initial value set to 0 is introduced to track the starting point of the interval of interest; next, the algorithm traverses all rows and columns of the mask matrix; during the traversal process, if flag is 0 and the current position mask is 0, ignore the position and continue traversing; if flag is 1 and the current position mask is 1, and the number of consecutive values ​​1 from the starting point to the current position exceeds 2, it indicates the end of an interval of interest. At this time, the end position of the interval is recorded, and the corresponding x and y intervals are added to the x2 and y2 lists. When flag is 0 and the current position mask value is 1, set flag to 1 and set the current position as the starting point. On the contrary, if flag is 1 and the current position mask value is 0, reset flag to 0 and record the current position as the end position of the interval of interest. Finally, the algorithm returns arrays x2 and y2, which contain the features and labels of the extracted data intervals, respectively.

[0110] Step S4 specifically includes the following steps:

[0111] Step S41: For the distance-angle RA and distance-Doppler RD information, first use 3D convolution and pooling layers to extract the spatiotemporal features of the video; the convolution layer applies 3D convolution operations to each time step of the input video sequence to extract the spatiotemporal features of the video; similarly, for the fractional domain spectrogram, two-dimensional convolution and pooling are used to extract spatiotemporal features. The outputs of the three convolution layers are then processed into the specified shape through the reshape function. The features are processed through the fully connected layer and the Dropout layer; then, the bidirectional LSTM layer is applied to the input sequence;

[0112] Step S42: After extracting the input features, the model uses the concatenate function to merge the penultimate layer outputs of the three-way input into a tensor, and then connects the merged tensor to an output layer with five categories and a softmax activation function through a fully connected layer;

[0113] Step S43: Finally, a CTC layer is defined to calculate the CTC loss and added after the output layer; the calculation formula of the CTC loss function is as follows:

[0114] Given an input sequence X and a corresponding label sequence Y, define Y′ as the label sequence with repeated characters and blank characters removed, that is, Y′={y|y∈Y,y≠blank}. Then the CTC loss is:

[0115] Loss CTC = -∑log(P(Y′|X))

[0116] Among them, P(Y′|X) represents the probability of predicting the output sequence to be Y′ given the input sequence X. During the training process, CTC uses a dynamic programming algorithm to calculate the conditional probability of the model output sequence and compares it with the actual output sequence to calculate the loss function. By minimizing the loss function, the model learns the correspondence between the input sequence and the output sequence, thereby making accurate sequence labeling predictions.

[0117] The recognition method collects continuous human motion radar data sets and performs human motion detection in the following way: a millimeter-wave radar is placed in front of the subject, and according to the different motion habits of each subject, an angle deviation within about 120° of the total field of view is allowed when the subject faces the millimeter-wave radar; the target of the collection is human motion, and the reflection cross-sectional area RCS of the human body facing the millimeter-wave radar is 0.5-1m 2 , making the distance detection range applicable to this recognition method 5-15 meters.

[0118] The recognition method collects five human body movements, namely, punching, walking, standing, sitting down, and standing up, and asks the subjects to perform corresponding movement demonstrations in a random combination of the five movements at a distance of 5 meters in front of the FMCW radar for data collection and movement detection.

[0119] In this example, when collecting continuous human motion radar data sets and performing human motion detection, the millimeter-wave radar is placed in front of the subject. At the same time, due to the different motion habits of each subject, a certain angle deviation is allowed when facing the millimeter-wave radar. The IWR1642 Boost development board we use has a horizontal field of view of about ±60° (total field of view of about 120°), which means that the subject is allowed to have a motion angle deviation within the above range.

[0120] Since the target collected by the radar is human movement, and the reflection cross-sectional area (RCS) of the human body facing the millimeter-wave radar is usually 0.5-1m 2 , so the solution is applicable to a distance detection range of 5-15 meters.

[0121] In the embodiment of the present scheme, taking five human body movements of punching, walking, standing, sitting down, and standing up as examples, the subjects are asked to perform corresponding action demonstrations in a random combination of the above five actions (e.g., punching - standing up - walking - standing up) at 5 meters in front of the FMCW radar for data collection and action detection.

[0122] Example:

[0123] like Figure 1 As shown, this embodiment provides a human body posture recognition method based on FMCW radar signals.

[0124] This embodiment includes the following specific steps:

[0125] Step 1: Take five human actions, such as punching, walking, standing, sitting down, and standing up, as examples, and ask the subjects to demonstrate with a specific action combination (e.g. punching-standing-walking-standing up). Use frequency modulated continuous wave (FMCW) radar to collect ADC data of the original human posture, and then perform corresponding data analysis, and perform two-dimensional fast Fourier transform on the slow time / fast time data matrix of a single receiving channel of the radar system to obtain the range-Doppler image. Organize multiple receiving channels according to time delay and Doppler frequency, calculate the spatial covariance matrix of the received signal and the spatial spectrum distribution of each angle, and construct the range-angle image.

[0126] Step 2: Apply the cell average constant false alarm rate method to the range-Doppler image to detect the range interval of motion. For the detected range interval, apply multiple short-time fractional Fourier transforms in the slow time dimension to obtain the fractional domain spectrum.

[0127] Step 3: Perform activity detection based on variable window STA / LTA on the range-Doppler image, range-angle image, and fractional domain spectrogram containing continuous human activity information, and segment the continuous human activity radar image to identify the starting point and end point of the human activity.

[0128] Step 4: Input the segmented human activity data into the convolutional neural network for feature extraction, then use the bidirectional long short-term memory layer to model the three-way features, and use the reshape and concatenate functions to merge the multi-way outputs. The fused high-dimensional human action tensor is output through the connection time series classification method to obtain the human activity classification result.

[0129] The specific implementation process in step 1 is as follows:

[0130] Step 11: When collecting human body movements, we selected observation data containing 200 frames of radar emission signals, with a total duration of about 8.0 seconds, and set four types of activity combinations as input activity sequences. The specific activity combinations are (1) sit down - punch - stand up - walk, (2) punch - stand up - walk - stand up, (3) stand - sit down - punch - stand up, (4) stand up - walk - sit down - punch. Four experimenters were invited to collect activity data, and each person performed each type of continuous activity 100 times. A total of 1,600 activity sequence samples were collected in the experiment, of which 400 samples were collected for each activity.

[0131] Step 12: Perform signal processing on the human motion information collected by the radar, such as Figure 2 First, the range-Doppler image is obtained by performing a two-dimensional fast Fourier transform (2DFFT) on the slow-time / fast-time data matrix of a single radar channel.

[0132] Step 13: Organize multiple receiving channels according to delay and Doppler frequency to form a delay-Doppler matrix. Based on the delay-Doppler matrix, calculate the spatial covariance matrix R of the received signal, and calculate the spatial spectrum distribution P(θ) of each angle through the multiple signal classification (MUSIC) algorithm. By combining the estimated angle spectrum with the distance, a range-Doppler (RA) image is generated.

[0133] The specific implementation process in step 2 is as follows:

[0134] Step 21: For the obtained distance-Doppler map, the Cell Average-Constant False Alarm Rate (CA-CFAR) method is first applied to detect the range of motion distance. Then, the slow time dimension of the detected motion distance interval is subjected to a short-time fractional Fourier transform to obtain a fractional domain spectrogram. Finally, the distance-Doppler map, distance-angle map, and fractional domain spectrogram are combined to obtain a schematic diagram of the human motion data set as shown below: Figure 3 shown.

[0135] like Figure 4 As shown, the specific implementation process in step 3 is as follows:

[0136] Step 31: Calculate the absolute difference between the upper and lower envelopes of the time index according to the score domain spectrogram to create an absolute distance vector. And determine the starting point and end point of each human action according to the starting threshold point of the action and the STA / LTA value of the continuous action.

[0137] Step 32: Then, the variable window length of STA / LTA is realized by introducing a mask. The algorithm creates a zero vector mask with the same shape as the input vector. When a specific event (such as activity) is detected, the mask value of the corresponding interval is changed from 0 to 1, thereby quickly marking the event interval. In STA / LTA, based on the comparison of the average power ratio of the short window and the long window with the threshold, the input vector is looped, the state variables are updated, and the start and end positions of the event are marked, and finally the detected event interval is recorded in the mask.

[0138] Step S34: After using the variable window length STA / LTA detection algorithm to generate a detection mask to segment out a single activity in the fractional domain spectrogram, it is also necessary to synchronously segment the single activities in the range-Doppler video stream and the range-angle video stream. The demask algorithm is used to extract the interval of interest marked by a specific mask matrix from the two-dimensional data matrix. The algorithm extracts the features and labels marked as the interval of interest by traversing the data matrix and the Boolean matrix. Two empty lists are initialized to store the results, and a flag variable is used to track the interval status. During the traversal process, whether to enter or exit the interval of interest is determined according to the value of the Boolean matrix, and the continuous data that meets the conditions is extracted, and finally a list containing the features and labels of the interval of interest is returned.

[0139] like Figure 5 As shown, the specific implementation process in step 4 is as follows:

[0140] Step 41: For the range-angle (RA) and range-Doppler (RD) information, the spatiotemporal features are extracted through 6 3D convolutional layers and 4 pooling layers, where the convolution kernel size is 3×3×3, the output channels are gradually increased, and the pooling layer is downsampled to retain key information. To retain the time dimension features, the first pooling kernel is 1×2×2, and the rest are 2×2×2. The fractional domain spectrogram uses 4 2D convolutional layers and pooling layers (kernel sizes are 3×3 and 2×2, respectively), and the convolution outputs are 64, 128, 256, and 512, respectively, and then processed by reshape, fully connected layer, Dropout, and bidirectional LSTM. The corresponding feature extraction network is as follows: Figure 6 shown.

[0141] Step 42: The distance-Doppler map, distance-angle map, and fractional domain spectrum map have been compressed into corresponding high-dimensional tensors after passing through the feature extraction network. At this time, in order to reduce the complexity of the network, the three features are spliced ​​and fused features are obtained. Finally, the fused features are input into the subsequent continuous time classification layer and the actual action label for CTC loss calculation, and the gradient back propagation is used to optimize the network model so that the model's predicted output converges to the correct label. The CTC loss is defined as:

[0142] Given an input sequence X and a corresponding label sequence Y, define Y′ as the label sequence with repeated characters and blank characters removed, that is, Y′={y|y∈Y,y≠blank}. Then the CTC loss is:

[0143] Loss CTC = -∑log(P(Y′|X))

[0144] Among them, P(Y′|X) represents the probability of predicting the output sequence to be Y′ given the input sequence X.

[0145] The present invention and solution perform algorithmic analysis and processing on the original signal of FMCW radar, segment continuous human motion with high precision based on the activity detection algorithm of variable window length STA / LTA, and accurately fuse three types of radar domain information by building a multi-input multi-task learning neural network to achieve high-precision continuous human motion recognition tasks. In order to verify the role of the bidirectional LSTM layer and CTC layer in the human posture recognition model, experiments were conducted using the human motion data segmented by the STA / LTA action detection algorithm based on variable window length, and the results are shown in Table 1.

[0146] Table 1

[0147]

[0148] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of continuous human activity recognition methods based on FMCW radar multi-domain feature fusion under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A continuous human activity recognition method based on FMCW radar multi-domain feature fusion, characterized by: The method firstly processes the raw data collected by the radar by using the fast Fourier transform algorithm and the multiple signal classification technology to obtain the range-Doppler image and the range-angle image, and maps the radar data to the fractional domain by using the short-time fractional Fourier transform to obtain the fractional domain spectrogram; Then, the variable window length STA / LTA continuous action detection algorithm is used to achieve the synchronous segmentation of the three types of continuous activity sequence domain information; finally, the segmented activity samples are used as the input of the multi-input multi-task continuous activity recognition model, and the input features are extracted through the convolutional network and the bidirectional long-term memory network. The CTC layer is introduced to fuse the multi-input representation, and the CTC loss of the predicted sequence and the true label is calculated. Multi-task learning is performed to obtain the human action recognition results.

2. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 1 is characterized in that: The method comprises the following steps: Step S1: Use FMCW radar to collect original human posture data, perform two-dimensional fast Fourier transform on the slow-time / fast-time data matrix of a single receiving channel of the radar system to obtain a range-Doppler image. Organize multiple receiving channels according to time delay and Doppler frequency, calculate the spatial covariance matrix of the received signal and the spatial spectrum distribution of each angle, and construct a range-angle image; Step S2: applying the cell average constant false alarm rate method to the range-Doppler image to detect the range of motion; for the detected range interval, applying multiple short-time fractional Fourier transforms in the slow time dimension to obtain a fractional domain spectrum; Step S3: performing activity detection based on variable window STA / LTA on the range-Doppler image, the range-angle image, and the fractional domain spectrogram containing continuous human activity information, segmenting the continuous human activity radar image to identify the starting point and end point of the human activity; Step S4: Input the segmented human activity data into the convolutional neural network for feature extraction, then use the bidirectional long short-term memory layer to model the three-way features, and use the reshape and concatenate functions to merge the multi-way outputs. The fused high-dimensional human action tensor is output through the connection time series classification method to obtain the human activity classification result.

3. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 2 is characterized in that: In step S1, the range-Doppler image and the range-angle image are obtained by the following specific steps: Step S11: After the radar receives the echo signal, it samples it and forms a slow time / fast time data matrix; the slow time represents the time series of CPI, and the fast time represents the time sampling within each CPI; assuming that the data matrix is ​​X, the size is M×N, where M represents the slow time dimension, that is, the duration of CPI, and N represents the fast time dimension, that is, the number of time sampling within each CPI; Before performing 2D FFT, a window function is applied to the data matrix to reduce the spectrum leakage caused by the boundary; after applying the window function, the data matrix X after window function processing is obtained. w , the formula is: X w (i,j)=X(i,j)×w(i,j) Among them, X w (i, j) represents the element of the data matrix after window function processing, X(i, j) represents the element of the original data matrix, and w(i, j) represents the value of the window function; After the data matrix X is processed by the window function w Perform a 2D FFT to convert the time domain into the frequency domain and obtain the range-Doppler plane, which is represented as a range-Doppler image: Y(u,v)=FFT2(X w ) Among them, Y(u,v) represents the result after 2D FFT, the size is M×N, and u and v represent the index in the frequency domain; In addition, the range-Doppler images of multiple CPIs are accumulated and averaged to enhance the target signal: Among them, Y avg (u,v) represents the result after cumulative averaging, N represents the number of accumulated frames, and Y(u,v) represents the range-Doppler image of each frame; The processed and enhanced range-Doppler images are combined to form an RD video; each image represents the range-Doppler information of one CPI; Step S12: The radar system collects echo signals through multiple receiving channels, each receiving channel corresponds to an antenna or array element, which is used to receive signals from different directions; next, for each CPI, the received signal data is processed; first, the data is organized according to delay and Doppler frequency to form a delay-Doppler matrix; for i receiving channels, the received signal data is represented as a complex sequence: [s1(i), s2(i), ..., s N (i)], where N represents the number of sampling points; these signals are organized according to delay and Doppler frequency to form a delay-Doppler matrix D i , whose elements can be expressed as D i (t,f)=s i (t,f) The rows of the matrix correspond to different angles, and the columns correspond to different Doppler frequencies; each element represents the received signal strength at a specific angle and Doppler frequency; Based on the delay-Doppler matrix, the spatial covariance matrix R of the received signal is calculated; the spatial covariance matrix describes the signal correlation between different receiving channels; the delay-Doppler matrices of all receiving channels are spliced ​​by column to form a total delay-Doppler matrix D, whose size is N×M, where N represents the delay dimension and M represents the Doppler frequency dimension. The spatial covariance matrix R is obtained by calculating the conjugate transpose of D and the inner product of D: R=D H D Among them, D H represents the conjugate transpose of D; Then, the spatial covariance matrix is ​​further subjected to eigendecomposition to extract its eigenvectors and eigenvalues: R(u k )=λ k u k By calculating the spatial spectrum distribution of each angle, the arrival of the target at different angles is estimated. This is achieved using the multi-signal classification algorithm MUSIC, with the following formula: The MUSIC algorithm calculates the spatial spectrum distribution P(θ) at each angle θ through this formula; in this formula, a(θ) is the array manifold vector at angle θ, which describes the response pattern of the antenna array to a signal at a specific angle. K is the number of signals, and λ is k is the eigenvalue, u k is the corresponding eigenvector. The angular spectrum P(θ) represents the spatial distribution of the signal at different angles and is used to estimate the arrival angle of the target; By combining the estimated angle spectrum with the range, a range-Doppler RA image is generated.

4. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 3 is characterized by: In step S12, by repeating the steps, each CPI is processed to obtain a series of RA images; these images are combined at a predetermined frame rate to form a RA video; and the changes in the angle and Doppler of the target, as well as the motion characteristics of the target, are observed by playing the RA video.

5. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 2 is characterized in that: In step S2, the fractional domain spectrogram is obtained by the following specific steps: Step S21: First, the range of motion distance is detected by applying the cell average constant false alarm rate method CA-CFAR on the range-Doppler map. For the detected distance interval, multiple short-time fractional Fourier transforms STFrFT are applied in the slow time dimension to obtain the fractional domain spectrogram.

6. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 2 is characterized in that: Step S3 specifically includes the following steps: Step S31: A variable window STA / LTA based activity detection algorithm is used to identify the start and end points of continuous motion. First, the absolute difference between the upper and lower envelopes of the time index is calculated based on the fractional domain spectrogram to create an absolute distance vector. The absolute distance i of each data record can be calculated as v i =|u i -l i |, where v i is the absolute distance vector, u i and l i They are upper and lower envelopes respectively; Step S32: STA(t) and LTA(t) can be defined as the leading window and the lagging window of time t: Where T1 and T2 are the lengths of the short time window and the long time window respectively; the starting point of the detection activity is when the following conditions are met: Among them, σ1 and σ2 are predefined detection thresholds; similarly, if Where σ3 is the detection threshold of the stopping point; In order to find the starting point, STA(t) needs to exceed the threshold σ1, which means that the activity must appear in the short window; at the same time, the average power ratio of the short window and the long window should be greater than σ2; so that in the presence of noise, the system will not be triggered unless the ratio exceeds σ2; ​​it also ensures that the stop point of the activity can be correctly detected, that is, when the activity disappears from the processing window and the ratio drops below the threshold σ2; Step S33: Implement the variable window length of STA / LTA by introducing a mask; the mask is a zero vector with the same shape as the input vector; during the execution of the function, the index position under specific conditions will be set to 1 to indicate that the data at these positions have been processed or belong to a specific category; when a specific event is detected, that is, when an activity occurs, the mask replaces 0 with 1 in the corresponding interval; create a zero vector mask with the same shape as the input vector and set its initial value to 0, which is used to mark a specific interval in the input vector after STA / LTA detection; then define a state variable with an initial value of 0, and by looping through the input vector, compare the long window value, the short window value, and the average power ratio of the short window and the long window with the size of the above threshold, update the state variable and determine the start position and the stop position, and mark the corresponding interval in the mask; Step S34: After using the variable window length STA / LTA detection algorithm to generate a detection mask to segment out individual activities in the fractional domain spectrogram, individual activities in the range-Doppler video stream and the range-angle video stream also need to be segmented synchronously; therefore, the demask algorithm is used to extract the interval of interest identified by a specific mask matrix from the two-dimensional data matrix.

7. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 6 is characterized in that: The demask algorithm accepts three input parameters: two-dimensional data matrices x and y, and a two-dimensional Boolean matrix mask used to indicate the data intervals of interest in x and y. The specific usage method is as follows: First, the algorithm initializes two empty lists x2 and y2 to collect the extracted data intervals; at the same time, a flag variable flag with an initial value set to 0 is introduced to track the starting point of the interval of interest; next, the algorithm traverses all rows and columns of the mask matrix; during the traversal process, if flag is 0 and the current position mask is 0, ignore the position and continue traversing; if flag is 1 and the current position mask is 1, and the number of consecutive values ​​1 from the starting point to the current position exceeds 2, it indicates the end of an interval of interest. At this time, the end position of the interval is recorded, and the corresponding x and y intervals are added to the x2 and y2 lists. When flag is 0 and the current position mask value is 1, set flag to 1 and set the current position as the starting point. On the contrary, if flag is 1 and the current position mask value is 0, reset flag to 0 and record the current position as the end position of the interval of interest. Finally, the algorithm returns arrays x2 and y2, which contain the features and labels of the extracted data intervals, respectively.

8. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 2 is characterized by: Step S4 specifically includes the following steps: Step S41: For the distance-angle RA and distance-Doppler RD information, first use 3D convolution and pooling layers to extract the spatiotemporal features of the video; the convolution layer applies 3D convolution operations to each time step of the input video sequence to extract the spatiotemporal features of the video; similarly, for the fractional domain spectrogram, two-dimensional convolution and pooling are used to extract spatiotemporal features. The outputs of the three convolution layers are then processed into the specified shape through the reshape function. The features are processed through the fully connected layer and the Dropout layer; then, the bidirectional LSTM layer is applied to the input sequence; Step S42: After extracting the input features, the model uses the concatenate function to merge the penultimate layer outputs of the three-way input into a tensor, and then connects the merged tensor to an output layer with five categories and a softmax activation function through a fully connected layer; Step S43: Finally, a CTC layer is defined to calculate the CTC loss and added after the output layer; the calculation formula of the CTC loss function is as follows: Given an input sequence X and a corresponding label sequence Y, define Y′ as the label sequence with repeated characters and blank characters removed, that is, Y′={y|y∈Y,y≠blank}. Then the CTC loss is: Loss CTC =-∑log(P(Y′|X)) Among them, P(Y′|X) represents the probability of predicting the output sequence to be Y′ given the input sequence X. During the training process, CTC uses a dynamic programming algorithm to calculate the conditional probability of the model output sequence and compares it with the actual output sequence to calculate the loss function. By minimizing the loss function, the model learns the correspondence between the input sequence and the output sequence, thereby making accurate sequence labeling predictions.

9. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 2 is characterized by: When the recognition method collects a continuous human motion radar data set and performs human motion detection, the method is as follows: a millimeter wave radar is placed directly in front of the subject, and according to the different motion habits of each subject, an angle deviation within about 120° of the total field of view is allowed when the subject faces the millimeter wave radar; The target of the collection is human body movement. Assume that the reflection cross-sectional area RCS of the human body facing the millimeter-wave radar is 0.5-1m 2 , making the distance detection range applicable to this recognition method 5-15 meters.

10. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 9 is characterized in that: The recognition method collects five human body movements, namely, punching, walking, standing, sitting down, and standing up, and asks the subjects to perform corresponding movement demonstrations in a random combination of the five movements at a distance of 5 meters in front of the FMCW radar for data collection and movement detection.

Images