Continuous human activity recognition method based on multi-domain feature fusion of FMCW radar

Abstract

The application proposes a continuous human activity recognition method based on FMCW radar multi-domain feature fusion. Firstly, the fast Fourier transform algorithm and the multiple signal classification technology are used to process and obtain the range-Doppler image and range-angle image from the original radar data, and the radar data is mapped to the fractional domain by using the short-time fractional Fourier transform to obtain the fractional domain spectrum. Then, through the variable window length STA / LTA continuous action detection algorithm, the synchronous segmentation of three types of continuous activity sequence domain information is realized. Finally, the segmented activity samples are taken as the input of the multi-input multi-task continuous activity recognition model, the input features are extracted through the convolution network and the bidirectional long-term memory network, the multi-input representation is introduced by the CTC layer, the CTC loss of the predicted sequence and the real label is calculated, the multi-task learning is performed, and the human action recognition result is obtained. The application can accurately segment continuous human actions, accurately fuse three types of radar domain information, and realize high-precision continuous human action recognition task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless sensing, and particularly to a continuous human activity recognition method based on FMCW radar multi-domain feature fusion. BACKGROUND

[0002] With the continuous progress of technology and the rapid development of society, intelligent monitoring systems play an increasingly important role in security, intelligent transportation, healthcare and other fields. In these application scenarios, human activity recognition technology is crucial, as it can monitor and identify various human activities in real time, providing necessary information for the system and further improving the system's intelligence level and ability to respond to emergencies. Human activity recognition refers to the process of using sensor technology and data processing methods to identify and classify the movement state of the human body in space. Currently, human activity recognition has become an active research field, involving the intersection of computer vision, pattern recognition, machine learning and other disciplines and technical fields, and has a wide range of applications in daily life and various scientific research fields.

[0003] Currently, there has been great development in the field of human posture recognition in terms of vision and sensors. However, existing posture recognition methods have certain limitations. Vision-based human posture recognition methods are easily obstructed by obstacles and are susceptible to light; sensor-based human posture recognition methods usually need to be worn on the body or installed in the environment, which may affect the user's comfort and naturalness. Among the technical means for realizing human activity recognition, radar technology has attracted attention due to its unique advantages. Compared to sensor-based methods, radar does not need to be worn on the body or installed in the environment, avoiding interference and inconvenience to users. Moreover, radar technology is not limited by environmental conditions such as light, shadow and obstruction, and is suitable for activity recognition in complex environments, such as at night or in bad weather, and can work all day long, with a long detection range and high positioning accuracy and stability. Compared to vision-based methods, radar technology does not require image or video data, and can better protect personal privacy.

[0004] However, despite the many advantages of FMCW radar, there are still some challenges to achieving accurate human activity recognition. First, human activities often have complex diversity, including various actions such as walking, standing, sitting, etc., so the recognition algorithm needs to have high accuracy and generalization ability. Second, radar signals are affected by various interference factors during transmission, such as multipath effect, noise interference, etc., which can reduce the performance and stability of the recognition system. Traditional radar signal processing methods can only extract information in a single domain, such as time domain or frequency domain, which limits the accuracy and robustness of the recognition algorithm. Especially in complex and variable environments, the lack of single-domain information may cause the recognition algorithm to fail. Multi-domain information fusion technology based on FMCW radar signals can integrate information in time domain, frequency domain, and angle domain, etc., to improve the accuracy and robustness of target recognition and tracking. In addition, in real life, human activities are continuous and time-varying. This means that human activities not only involve different actions and postures, but also include transitions, intersections, and overlaps. This complexity brings a series of difficulties and challenges to activity segmentation and classification. SUMMARY

[0005] The present application proposes a continuous human activity recognition method based on FMCW radar multi-domain feature fusion, which can accurately segment continuous human actions and accurately fuse three types of radar domain information to achieve high-precision continuous human action recognition tasks.

[0006] The present application adopts the following technical solutions.

[0007] The continuous human activity recognition method based on FMCW radar multi-domain feature fusion first processes and obtains the range-Doppler image and range-angle image from the original data collected by the radar through the fast Fourier transform algorithm and multiple signal classification technology, and maps the radar data to the fractional domain using the short-time fractional Fourier transform to obtain the fractional domain spectrum. Then, through the variable window length STA / LTA continuous action detection algorithm, the synchronous segmentation of three types of continuous activity sequence domain information is realized. Finally, the segmented activity samples are used as the input of the multi-input multi-task continuous activity recognition model, the input features are extracted through the convolution network and the bidirectional long short-term memory network, the multi-input representation is introduced through the CTC layer, and the CTC loss of the predicted sequence and the real label is calculated. The multi-task learning is performed and the human action recognition result is obtained.

[0008] The method includes the following steps:

[0009] Step S1: Collect raw human posture data using FMCW radar, and perform two-dimensional fast Fourier transform on the slow-time / fast-time data matrix of a single receiving channel of the radar system to obtain a range-Doppler image. Organize multiple receiving channels according to time delay and Doppler frequency, calculate the spatial covariance matrix of the received signal and the spatial spectrum distribution of each angle, and construct a range-angle image;

[0010] Step S2: Apply the cell averaging constant false alarm rate method to the range-Doppler image to detect the range interval of motion; for the detected range interval, apply multiple short-time fractional Fourier transforms in the slow-time dimension to obtain a fractional domain spectrum image;

[0011] Step S3: Perform activity detection based on a variable window STA / LTA on the range-Doppler image, the range-angle image, and the fractional domain spectrum image containing continuous human activity information, and segment the continuous human activity radar image to identify the start and end points of human activity;

[0012] Step S4: Input the segmented human activity data into a convolutional neural network for feature extraction, then model the three-channel features using a bidirectional long short-term memory layer, and merge the multi-channel outputs using reshape and concatenate functions. Fuse the high-dimensional human motion tensor through a connection temporal classification method to obtain human activity classification results.

[0013] In step S1, the range-Doppler image and the range-angle image are obtained through the following specific steps: Step S11: After the radar receives the echo signal, it is sampled and forms a slow-time / fast-time data matrix; the slow-time represents the time sequence of CPI, and the fast-time represents the time sampling within each CPI; assume that the data matrix is X, with a size of MxN, where M represents the slow-time dimension (the duration of CPI), and N represents the fast-time dimension, i.e. the number of time samples within each CPI;

[0014] Before performing 2D FFT, apply a window function to the data matrix to reduce the spectral leakage caused by the boundary; after applying the window function, the window function processed data matrix X w is obtained, and the formula is:

[0015] X w (i,j) = X(i,j) x w(i,j)

[0016] where X w (i,j) represents an element of the window function processed data matrix, X(i,j) represents an element of the original data matrix, and w(i,j) represents the value of the window function;

[0017] Perform 2D FFT on the window function processed data matrix X wA 2D FFT is performed to convert the time domain to the frequency domain, resulting in a range-Doppler plane, denoted as a range-Doppler image:

[0018] Y(u,v) = FFT2(X w )

[0019] where Y(u,v) denotes the result after 2D FFT, with size M x N, and u and v denote the indices in the frequency domain;

[0020] In addition, the range-Doppler images of multiple CPIs are cumulatively averaged to enhance the target signal:

[0021]

[0022] where Y avg (u,v) denotes the result after cumulative averaging, and N denotes the number of frames for accumulation, and Y(u,v) denotes the range-Doppler image of each frame;

[0023] The processed and enhanced range-Doppler images are combined together to form an RD video; each image represents the range-Doppler information of one CPI;

[0024] Step S12: The radar system collects echo signals through multiple receiving channels, each receiving channel corresponding to an antenna or array element, for receiving signals from different directions; next, for each CPI, the received signal data is processed; first, the data is organized according to time delay and Doppler frequency to form a time delay-Doppler matrix; for i receiving channels, the received signal data is represented as a complex number sequence: [s1(i), s2(i),..., s N (i)], where N denotes the number of sampling points; these signals are organized according to time delay and Doppler frequency to form a time delay-Doppler matrix D i , the elements of which can be represented as

[0025] D i (t,f) = s i (t,f)

[0026] The rows of the matrix correspond to different angles, and the columns correspond to different Doppler frequencies; each element represents the received signal strength at a specific angle and Doppler frequency;

[0027] Based on the delay-Doppler matrix, a spatial covariance matrix R of the received signal is calculated; the spatial covariance matrix describes the signal correlation between different receiving channels; the delay-Doppler matrix of all receiving channels is spliced according to the column to form a total delay-Doppler matrix D, and the size of D is N x M, wherein N represents the delay dimension, and M represents the Doppler frequency dimension. By calculating the inner product of the conjugate transpose of D and D, the spatial covariance matrix R is obtained:

[0028] R = D H D

[0029] Wherein, D H represents the conjugate transpose of D;

[0030] Subsequently, the spatial covariance matrix is further processed by eigenvalue decomposition to extract the eigenvectors and eigenvalues: R(u k ) = λ k u k

[0031] By calculating the spatial spectrum distribution of each angle, the arrival of the target at different angles is estimated. The multiple signal classification algorithm (MUSIC) is used to realize it, and the formula is as follows:

[0032]

[0033] The MUSIC algorithm calculates the spatial spectrum distribution P(θ) of each angle θ through the formula; in the formula, a(θ) is the array manifold vector at angle θ, which describes the response mode of the antenna array to the signal at a specific angle. K is the number of signals, λ k is the eigenvalue, and u k is the corresponding eigenvector. The angle spectrum P(θ) represents the spatial distribution of the signal at different angles, which is used to estimate the arrival angle of the target;

[0034] By combining the estimated angle spectrum with the distance, a range-Doppler (RA) image is generated.

[0035] In step S12, by repeatedly performing the steps, each CPI is processed to obtain a series of RA images; these images are combined together according to a predetermined frame rate to form an RA video; by playing the RA video, the change of the target in angle and Doppler, and the motion characteristics of the target are observed.

[0036] In step S2, the fractional domain spectrum is obtained by the following specific steps:

[0037] Step S21: Firstly, the CA-CFAR method is used to detect the range of moving distance on the distance-Doppler map. For the detected distance range, the STFrFT is applied in the slow time dimension, i.e. the fractional domain spectrum is obtained.

[0038] Step S3 specifically comprises the following steps:

[0039] Step S31: An activity detection algorithm based on variable window STA / LTA is used to identify the start and end points of continuous motion; firstly, the absolute difference between the upper and lower envelopes in the time index is calculated according to the fractional domain spectrum to create an absolute distance vector; the absolute distance i of each data record can be calculated as i = |u i - l i |, wherein v i is the absolute distance vector, u i and l i are the upper and lower envelopes respectively;

[0040] Step S32: STA(t) and LTA(t) can be defined as the leading window and the lagging window at time t:

[0041]

[0042] wherein T1 and T2 are the lengths of the short time window and the long time window respectively; when the following conditions are met, the start point of the activity is detected:

[0043]

[0044] wherein σ1 and σ2 are predefined detection thresholds; similarly, if

[0045]

[0046] wherein σ3 is the detection threshold of the stop point;

[0047] In order to find the start point, STA(t) needs to exceed the threshold σ1, which means that the activity must appear within the short window; at the same time, the average power ratio of the short window and the long window should be greater than σ2; so that in the presence of noise, the system will not be triggered unless the ratio exceeds σ2; at the same time, it also ensures that the stop point of the activity can be correctly detected, i.e. when the activity disappears from the processing window, the ratio drops below the threshold σ2;

[0048] Step S33: Variable window length of STA / LTA is realized by introducing a mask, which is a zero vector with the same shape as the input vector. During the function execution, the index positions under certain conditions will be set to 1, indicating that the data in these positions has been processed or belongs to a certain category. When a certain event is detected, i.e., activity occurs, the mask will replace 0 with 1 in the corresponding interval. A zero vector mask with the same shape as the input vector is created and set to 0 initially, which is used to mark the specific interval in the input vector after STA / LTA detection. Then a state variable with an initial value of 0 is defined, and the input vector is traversed by loop. According to the size comparison of the long window value, the short window value and the average power ratio of the short window and the long window with the threshold value, the state variable is updated and the start position and the stop position are judged, and the corresponding interval is marked in the mask. Step S34: After using the variable window length STA / LTA detection algorithm to generate the detection mask to segment the single activity in the score domain spectrum diagram, the single activity in the range-doppler video stream and the range-angle video stream also needs to be segmented synchronously. Therefore, the demask algorithm is used to extract the interested interval identified by the specific mask matrix from the two-dimensional data matrix.

[0049] The demask algorithm accepts three input parameters: two-dimensional data matrix x and y, and two-dimensional boolean matrix mask used to indicate the data interval of interest in x and y. The specific method is as follows: first, the algorithm initializes two empty lists x2 and y2 to collect the extracted data interval; at the same time, a flag variable flag with an initial value of 0 is introduced to track the starting point of the interested interval; next, the algorithm traverses all rows and columns of the mask matrix; during the traversal, if flag is 0 and the current position mask is 0, the position is ignored and the traversal continues; if flag is 1 and the current position mask is 1, and the number of consecutive values of 1 from the starting point to the current position is more than 2, it indicates the end of an interested interval, at which time the end position of the interval is recorded, and the corresponding x and y interval is added to the x2 and y2 lists. When flag is 0 and the current position mask value is 1, flag is set to 1, and the current position is set as the starting point. Conversely, if flag is 1 and the current position mask value is 0, flag is reset to 0, and the current position is recorded as the end position of the interested interval. Finally, the algorithm returns arrays x2 and y2, which contain the features and labels of the extracted data interval, respectively.

[0050] Step S4 specifically includes the following steps:

[0051] Step S41: For the range-angle (RA) and range-doppler (RD) information, first use 3D convolution and pooling layers to extract the spatio-temporal features of the video; the convolution layer applies a 3D convolution operation to each time step of the input video sequence, thereby extracting the spatio-temporal features of the video; similarly, for the fractional domain spectrogram, the spatio-temporal features are extracted by two-dimensional convolution and pooling. Then the outputs of the three convolution layers are processed into a specified shape by the reshape function. And the features are processed through the fully connected layer and the Dropout layer; then, a bidirectional LSTM layer is applied to the input sequence;

[0052] Step S42: After the model extracts the input features, the concatenate function is used to merge the outputs of the second-to-last layer of the three inputs into a tensor, and then the merged tensor is connected to an output layer with five categories and a softmax activation function through a fully connected layer;

[0053] Step S43: Finally, a CTC layer is defined to calculate the CTC loss and is added after the output layer; the calculation formula of the CTC loss function is as follows:

[0054] Given an input sequence X and the corresponding label sequence Y, define Y' as the label sequence with repeated characters and blank characters removed, i.e. Y' = {y | y Y, y ≠ blank}. Then the CTC loss is:

[0055] Loss CTC =-∑log(P(Y'|X))

[0056] Where P(Y'|X) represents the probability of predicting the output sequence Y' given the input sequence X; during training, CTC uses a dynamic programming algorithm to calculate the conditional probability of the model output sequence and compares it with the true output sequence to calculate the loss function; by minimizing the loss function, the model learns the correspondence between the input sequence and the output sequence, thereby making accurate sequence labeling predictions.

[0057] The recognition method collects continuous human action radar data sets and performs human action detection, and the method is: placing the millimeter wave radar in front of the subject, allowing an angular offset within a total field of view of about 120° for each subject's different action habits when the subject faces the millimeter wave radar; the target is human action, and the reflection cross-sectional area (RCS) of the human body facing the millimeter wave radar is 0.5-1 m 2 , so that the distance detection range of the recognition method is 5-15 meters.

[0058] The recognition method collects five human body movements: punching, walking, standing, sitting, and standing up. Subjects are asked to perform corresponding actions in random combinations of these five movements at a distance of 5 meters directly in front of the FMCW radar for data collection and action detection.

[0059] Compared with existing technologies, this invention and solution perform algorithmic analysis and processing on the raw FMCW radar signal, accurately segment continuous human movements based on the activity detection algorithm with variable window length STA / LTA, and accurately fuse three types of radar domain information by building a multi-input multi-task learning neural network to achieve high-precision continuous human movement recognition. Attached Figure Description

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0061] Appendix Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention;

[0062] Appendix Figure 2 This is a visual schematic diagram of the FMCW radar multi-domain data formation method according to an embodiment of the present invention;

[0063] Appendix Figure 3 This is a schematic diagram of the human motion dataset visualization according to an embodiment of the present invention;

[0064] Appendix Figure 4 This is a schematic diagram of the continuous action segmentation process according to an embodiment of the present invention;

[0065] Appendix Figure 5 This is a schematic diagram of a human posture recognition model according to an embodiment of the present invention;

[0066] Appendix Figure 6 This is a schematic diagram of the feature extraction network according to an embodiment of the present invention. Detailed Implementation

[0067] As shown in the figure, a continuous human activity recognition method based on multi-domain feature fusion of FMCW radar is proposed. The method first processes the raw radar data to obtain range-Doppler and range-angle images using a Fast Fourier Transform (FFT) algorithm and multiple signal classification technology. Then, a Short-Time Fractional Fourier Transform (SFT) is used to map the radar data onto the fractional domain, resulting in a fractional domain spectrogram. Next, a Variable Window Length (STA / LTA) continuous action detection algorithm is used to simultaneously segment the three types of continuous activity sequence information. Finally, the segmented activity samples are used as input to a multi-input, multi-task continuous activity recognition model. Input features are extracted using convolutional networks and bidirectional long-term memory networks. A CTC layer is introduced to fuse the multi-input representation, and the CTC loss between the predicted sequence and the true label is calculated. Multi-task learning is then performed to obtain the human action recognition result.

[0068] The method comprises the following steps:

[0069] Step S1: Collecting original human posture data using FMCW radar, performing two-dimensional fast Fourier transform on the slow-time / fast-time data matrix of a single receiving channel of the radar system to obtain a range-Doppler image. Organize multiple receiving channels according to time delay and Doppler frequency, calculate the spatial covariance matrix of the received signal and the spatial spectrum distribution of each angle, and construct a range-angle image;

[0070] Step S2: Applying the cell averaging constant false alarm rate method to the range-Doppler image to detect the range interval of motion; for the detected range interval, applying multiple short-time fractional Fourier transform in the slow-time dimension to obtain a fractional domain spectrum image;

[0071] Step S3: Performing activity detection based on a variable window STA / LTA on the range-Doppler image, the range-angle image, and the fractional domain spectrum image containing continuous human activity information, segmenting the continuous human activity radar image to identify the start and end points of human activity;

[0072] Step S4: Inputting the segmented human activity data into a convolutional neural network for feature extraction, then modeling the three-way features using a bidirectional long short-term memory layer, and merging the multi-way outputs using reshape and concatenate functions. Merge the high-dimensional human motion tensor through the connection time series classification method to obtain the human activity classification result.

[0073] In step S1, the range-Doppler image and the range-angle image are obtained through the following specific steps: Step S11: After the radar receives the echo signal, it is sampled and forms a slow-time / fast-time data matrix; the slow-time represents the time sequence of CPI, and the fast-time represents the time sampling within each CPI; assume that the data matrix is X, with a size of MxN, where M represents the slow-time dimension (the duration of CPI), and N represents the fast-time dimension, i.e. the number of time samples within each CPI;

[0074] Before performing 2D FFT, apply a window function to the data matrix to reduce the spectral leakage caused by the boundary; after applying the window function, the window function processed data matrix X w is obtained, and the formula is:

[0075] X w (i,j)=X(i,j)×w(i,j)

[0076] wherein X w (i,j) represents the elements of the window function processed data matrix, X(i,j) represents the elements of the original data matrix, and w(i,j) represents the value of the window function;

[0077] The data matrix X after window function processing w Perform 2D FFT to convert the time domain to the frequency domain to obtain the range-Doppler plane, denoted as a range-Doppler image:

[0078] Y(u,v) = FFT2(X w )

[0079] where Y(u,v) represents the result after 2D FFT, with a size of MxN, and u and v represent the indexes in the frequency domain;

[0080] In addition, the range-Doppler images of multiple CPIs are cumulatively averaged to enhance the target signal:

[0081]

[0082] where Y avg (u,v) represents the result after cumulative averaging, N represents the number of frames for accumulation, and Y(u,v) represents the range-Doppler image of each frame;

[0083] The processed and enhanced range-Doppler images are combined together to form an RD video; each image represents the range-Doppler information of one CPI;

[0084] Step S12: The radar system collects echo signals through multiple receiving channels, each receiving channel corresponding to an antenna or array element, for receiving signals from different directions; next, for each CPI, the received signal data is processed; first, the data is organized according to time delay and Doppler frequency to form a time-delay-Doppler matrix; for i receiving channels, the received signal data is represented as a complex number sequence: [s1(i), s2(i),..., s N (i)], where N represents the number of sampling points; these signals are organized according to time delay and Doppler frequency to form a time-delay-Doppler matrix D i , whose elements can be represented as

[0085] D i (t,f) = s i (t,f)

[0086] The rows of the matrix correspond to different angles, and the columns correspond to different Doppler frequencies; each element represents the received signal strength at a specific angle and Doppler frequency;

[0087] Based on the delay-Doppler matrix, a spatial covariance matrix R of the received signal is calculated; the spatial covariance matrix describes the signal correlation between different receiving channels; the delay-Doppler matrix of all receiving channels is spliced according to the column to form a total delay-Doppler matrix D, and the size of D is N x M, wherein N represents the delay dimension, and M represents the Doppler frequency dimension. By calculating the inner product of the conjugate transpose of D and D, the spatial covariance matrix R is obtained:

[0088] R = D H D

[0089] Wherein, D H represents the conjugate transpose of D;

[0090] Subsequently, the spatial covariance matrix is further processed by eigenvalue decomposition to extract the eigenvectors and eigenvalues: R(u k ) = λ k u k

[0091] By calculating the spatial spectrum distribution of each angle, the arrival of the target at different angles is estimated. The multiple signal classification algorithm (MUSIC) is used to realize it, and the formula is as follows:

[0092]

[0093] The MUSIC algorithm calculates the spatial spectrum distribution P(θ) of each angle θ through the formula; in the formula, a(θ) is the array manifold vector at angle θ, which describes the response mode of the antenna array to the signal at a specific angle. K is the number of signals, λ k is the eigenvalue, and u k is the corresponding eigenvector. The angle spectrum P(θ) represents the spatial distribution of the signal at different angles, which is used to estimate the arrival angle of the target;

[0094] By combining the estimated angle spectrum with the distance, a range-Doppler (RA) image is generated.

[0095] In step S12, by repeatedly performing the steps, each CPI is processed to obtain a series of RA images; these images are combined together according to a predetermined frame rate to form an RA video; by playing the RA video, the change of the target in angle and Doppler, and the motion characteristics of the target are observed.

[0096] In step S2, the fractional domain spectrum is obtained by the following specific steps:

[0097] Step S21: Firstly, the CA-CFAR method is used to detect the range of moving distance on the distance-Doppler map. For the detected distance range, the STFrFT is applied in the slow time dimension, i.e. the fractional domain spectrum is obtained.

[0098] Step S3 specifically comprises the following steps:

[0099] Step S31: An activity detection algorithm based on variable window STA / LTA is used to identify the start and end points of continuous motion; firstly, the absolute difference between the upper and lower envelopes in the time index is calculated according to the fractional domain spectrum to create an absolute distance vector; the absolute distance i of each data record can be calculated as i = |u i - l i |, wherein v i is the absolute distance vector, u i and l i are the upper and lower envelopes, respectively;

[0100] Step S32: STA(t) and LTA(t) can be defined as the leading window and the lagging window at time t:

[0101]

[0102] wherein T1 and T2 are the lengths of the short time window and the long time window, respectively; the start point of the activity is detected when the following conditions are met:

[0103]

[0104] wherein σ1 and σ2 are predefined detection thresholds; similarly, if

[0105]

[0106] wherein σ3 is the detection threshold of the stop point;

[0107] In order to find the start point, STA(t) needs to exceed the threshold σ1, which means that the activity must appear within the short window; at the same time, the average power ratio of the short window and the long window should be greater than σ2; so that in the presence of noise, the system will not be triggered unless the ratio exceeds σ2; at the same time, it also ensures that the stop point of the activity can be correctly detected, i.e. when the activity disappears from the processing window, the ratio drops below the threshold σ2;

[0108] Step S33: Variable window length of STA / LTA is realized by introducing a mask, which is a zero vector with the same shape as the input vector. During the function execution, the index positions under certain conditions will be set to 1, indicating that the data in these positions has been processed or belongs to a certain category. When a certain event is detected, i.e., activity occurs, the mask will replace 0 with 1 in the corresponding interval. A zero vector mask with the same shape as the input vector is created and set to 0 initially, which is used to mark the specific interval in the input vector after STA / LTA detection. Then a state variable with an initial value of 0 is defined, and the input vector is traversed by loop. According to the size comparison of the long window value, the short window value and the average power ratio of the short window and the long window with the above threshold value, the state variable is updated and the start position and stop position are judged, and the corresponding interval is marked in the mask. Step S34: After using the variable window length STA / LTA detection algorithm to generate the detection mask to segment the single activity in the score domain spectrum diagram, the single activity in the range-doppler video stream and the range-angle video stream also needs to be segmented synchronously. Therefore, the demask algorithm is used to extract the interested interval identified by the specific mask matrix from the two-dimensional data matrix.

[0109] The demask algorithm accepts three input parameters: two-dimensional data matrix x and y, and two-dimensional boolean matrix mask used to indicate the data interval of interest in x and y. The specific method is as follows: first, the algorithm initializes two empty lists x2 and y2 to collect the extracted data interval; at the same time, a flag variable flag with an initial value of 0 is introduced to track the starting point of the interested interval; next, the algorithm traverses all rows and columns of the mask matrix; during the traversal, if flag is 0 and the current position mask is 0, ignore this position and continue traversal; if flag is 1 and the current position mask is 1, and the number of consecutive values of 1 from the starting point to the current position is more than 2, it indicates the end of an interested interval, at this time the end position of the interval is recorded, and the corresponding x and y interval is added to the x2 and y2 lists. When flag is 0 and the current position mask value is 1, set flag to 1 and set the current position as the starting point. Conversely, if flag is 1 and the current position mask value is 0, reset flag to 0 and record the current position as the end position of the interested interval. Finally, the algorithm returns arrays x2 and y2, which contain the features and labels of the extracted data interval, respectively.

[0110] Step S4 specifically includes the following steps:

[0111] Step S41: For distance-angle (RA) and distance-Doppler (RD) information, 3D convolutional and pooling layers are first used to extract the spatiotemporal features of the video. The convolutional layers apply 3D convolution operations to each time step of the input video sequence to extract the spatiotemporal features. Similarly, for the fractional domain spectrogram, spatiotemporal features are extracted through 2D convolution and pooling. The outputs of the three convolutional layers are then processed into a specified shape using the reshape function. The features are then processed through fully connected layers and Dropout layers. Finally, a bidirectional LSTM layer is applied to the input sequence.

[0112] Step S42: After the model extracts the input features, it uses the concatenate function to merge the outputs of the penultimate layer of the three inputs into a tensor, and then connects the merged tensor to an output layer with five classes and a softmax activation function through a fully connected layer.

[0113] Step S43: Finally, a CTC layer is defined to calculate the CTC loss and added after the output layer; the formula for calculating the CTC loss function is as follows:

[0114] Given an input sequence X and a corresponding label sequence Y, define Y′ as the label sequence after removing duplicate characters and whitespace characters, i.e., Y′={y|y∈Y,y≠blank}. Then the CTC loss is:

[0115] Loss CTC =-∑log(P(Y′|X))

[0116] Wherein, P(Y′|X) represents the probability of predicting the output sequence as Y′ given the input sequence X; during the training process, CTC uses a dynamic programming algorithm to calculate the conditional probability of the model's output sequence and compares it with the true output sequence to calculate the loss function; by minimizing the loss function, the model learns the correspondence between the input sequence and the output sequence, thereby performing accurate sequence labeling prediction.

[0117] The aforementioned recognition method collects continuous human motion radar datasets and performs human motion detection as follows: A millimeter-wave radar is placed directly in front of the subject. Considering the different movement habits of each subject, an angular offset of approximately 120° is allowed when the subject faces the millimeter-wave radar. The target collected is human motion, and the radar cross-section (RCS) of the human body facing the millimeter-wave radar is assumed to be 0.5-1m². 2 This makes the detection range applicable to this identification method 5-15 meters.

[0118] The recognition method collects five human body movements: punching, walking, standing, sitting, and standing up. Subjects are asked to perform corresponding actions in random combinations of these five movements at a distance of 5 meters directly in front of the FMCW radar for data collection and action detection.

[0119] In this example, when collecting continuous human motion radar datasets and performing human motion detection, the millimeter-wave radar was placed directly in front of the subject. Since each subject has different movement habits, a certain angular offset is allowed when facing the millimeter-wave radar. The IWR1642 Boost development board we used has a horizontal field of view of approximately ±60° (total field of view approximately 120°), meaning that subject movement angular offsets within this range are permissible.

[0120] Since the radar collects data on human movement, and the radar cross-section (RCS) of a person facing the millimeter-wave radar is typically 0.5-1m. 2 Therefore, the applicable detection range of this scheme is 5-15 meters.

[0121] In an embodiment of this scheme, taking five human actions as examples, such as punching, walking, standing, sitting down, and standing up, the subject is asked to perform the corresponding actions at a distance of 5 meters directly in front of the FMCW radar using a random combination of the above five actions (e.g., punching-standing up-walking-standing), and data collection and action detection are carried out.

[0122] Example:

[0123] like Figure 1 As shown, this embodiment provides a human posture recognition method based on FMCW radar signals.

[0124] This embodiment includes the following specific steps:

[0125] Step 1: Using five human body movements—fist-waving, walking, standing, sitting, and standing up—as examples, have the subject demonstrate specific combinations of movements (e.g., fist-waving—standing up—walking—standing). ADC data of the raw human posture is acquired using a Frequency Modulated Continuous Wave (FMCW) radar, followed by corresponding data analysis. A two-dimensional Fast Fourier Transform is performed on the slow-time / fast-time data matrix of a single receiving channel of the radar system to obtain a range-Doppler image. Multiple receiving channels are organized according to time delay and Doppler frequency, and the spatial covariance matrix of the received signal and the spatial spectral distribution at each angle are calculated to construct a range-angle image.

[0126] Step 2: Apply the cell mean constant false alarm rate method to the range-Doppler image to detect intervals of motion distance. For each detected distance interval, apply multiple short-time fractional Fourier transforms in the slow time dimension to obtain a fractional domain spectrum.

[0127] Step 3: Perform activity detection based on variable window STA / LTA on the range-Doppler image, range-angle image, and fractional domain spectral map containing continuous human activity information, segment the continuous human activity radar image to identify the start and end points of human activity.

[0128] Step 4: Input the segmented human activity data into a convolutional neural network for feature extraction. Then, use a bidirectional long short-term memory layer to model the three feature streams, and use the reshape and concatenate functions to merge the multiple outputs. Output the fused high-dimensional human motion tensor through a connection-based temporal classification method to obtain the human activity classification result.

[0129] The specific implementation process in step 1 is as follows:

[0130] Step 11: When collecting human movements, observation data containing 200 frames of radar transmission signals was selected, with a total duration of approximately 8.0 seconds. Four types of activity combinations were set as input activity sequences. The specific activity combinations were (1) sit down—pump fist—stand up—walk, (2) pump fist—stand up—walk—stand, (3) stand—sit down—pump fist—stand up, (4) stand up—walk—sit down—pump fist. Four participants were invited to collect activity data, and each participant performed each type of continuous activity 100 times. A total of 1600 activity sequence samples were collected in the experiment, of which 400 samples were collected for each type of activity.

[0131] Step 12: Process the human motion information collected by the radar, such as... Figure 2 As shown, firstly, a range-Doppler image is obtained by performing a two-dimensional fast Fourier transform (2DFFT) on the slow-time / fast-time data matrix of a single radar channel.

[0132] Step 13: Organize multiple receiving channels according to time delay and Doppler frequency to form a time-delay-Doppler matrix. Based on the time-delay-Doppler matrix, calculate the spatial covariance matrix R of the received signal, and calculate the spatial spectral distribution P(θ) for each angle using the Multi-Signal Classification (MUSIC) algorithm. By combining the estimated angular spectrum with the range, a range-Doppler (RA) image is generated.

[0133] The specific implementation process in step 2 is as follows:

[0134] Step 21: For the obtained distance-Doppler map, first apply the Cell Averaging-Constant False Alarm Rate (CA-CFAR) method to detect the intervals of motion distance. Then, perform a short-time fractional Fourier transform on the slow-time dimension of the detected motion distance intervals to obtain the fractional-domain spectrogram. Finally, combine the distance-Doppler map, distance-angle map, and fractional-domain spectrogram to obtain a schematic diagram of the human motion dataset, as shown below. Figure 3 As shown.

[0135] like Figure 4 As shown, the specific implementation process in step 3 is as follows:

[0136] Step 31: Calculate the absolute difference between the upper and lower envelopes of the time index based on the fractional domain spectrogram to create an absolute distance vector. Then, determine the start and end points of each human movement based on the starting threshold point of the action and the STA / LTA values ​​of continuous actions.

[0137] Step 32: Then, the variable window length of STA / LTA is implemented by introducing a mask. This algorithm creates a zero-vector mask with the same shape as the input vector. When a specific event (such as an event occurring) is detected, the mask value for the corresponding interval is changed from 0 to 1, thus quickly marking the event interval. In STA / LTA, based on the comparison between the average power ratio of the short and long windows and the threshold, the input vector is iterated, the state variables are updated, and the start and end positions of the event are marked. Finally, the detected event interval is recorded in the mask.

[0138] Step S34: After generating a detection mask using the variable window length STA / LTA detection algorithm to segment individual activities in the fractional domain spectral map, it is also necessary to simultaneously segment individual activities in the range-Doppler video stream and the range-angle video stream. The demask algorithm is used to extract regions of interest (ROIs) identified by a specific mask matrix from the two-dimensional data matrix. This algorithm extracts features and labels marked as ROIs by traversing the data matrix and Boolean matrix. Two empty lists are initialized to store the results, and a flag variable is used to track the ROI status. During the traversal, the value of the Boolean matrix is ​​used to determine whether the ROI has been entered or exited, extracting continuous data that meets the conditions, and finally returning a list containing the ROI features and labels.

[0139] like Figure 5 As shown, the specific implementation process in step 4 is as follows:

[0140] Step 41: For range-angle (RA) and range-doppler (RD) information, spatio-temporal features are extracted by 6 3D convolutional layers and 4 pooling layers, where the convolution kernel size is 3x3x3, the output channels are gradually increased, and the pooling layers reduce the sampling to retain key information. To retain the time dimension features, the first pooling kernel is 1x2x2, and the rest are 2x2x2. The fractional domain spectrogram uses 4 2D convolutional layers and pooling layers (kernel sizes are 3x3 and 2x2, respectively), and the convolution outputs are 64, 128, 256, and 512, respectively, followed by reshape, fully connected layers, Dropout, and bidirectional LSTM processing. The corresponding feature extraction network is shown in FIG. 8. Figure 6

[0141] Step 42: After passing through the feature extraction network, the range-doppler graph, range-angle graph, and fractional domain spectrogram are compressed into corresponding high-dimensional tensors. To reduce network complexity, the features of the three are spliced and the fused features are obtained. Finally, the fused features are input into the subsequent continuous time sequence classification layer and the actual action label to calculate the CTC loss. Gradient backpropagation is used to optimize the network model, so that the model's prediction output converges to the correct label. The CTC loss is defined as:

[0142] Given an input sequence X and the corresponding label sequence Y, define Y' as the label sequence with repeated characters and blank characters removed, i.e., Y' = {y | y e Y, y ≠ blank}. Then the CTC loss is:

[0143] Loss CTC = -∑log(P(Y'|X))

[0144] Where P(Y'|X) represents the probability of the predicted output sequence being Y' given the input sequence X.

[0145] The present application and scheme perform algorithm analysis and processing on the original signal of FMCW radar, use the variable window length STA / LTA activity detection algorithm to accurately segment continuous human body actions, and accurately fuse the three types of radar domain information through the multi-input multi-task learning neural network, to realize the high-precision continuous human body action recognition task. To verify the role of the bidirectional LSTM layer and the CTC layer in the human posture recognition model, experiments are conducted on the human action data segmented by the variable window length-based STA / LTA action detection algorithm, and the results are shown in Table 1.

[0146] Table 1

[0147]

[0148] ​The patent is not limited to the above best embodiment, and anyone can derive other various forms of continuous human activity recognition methods based on FMCW radar multi-domain feature fusion under the inspiration of the patent. Any equivalent changes and modifications made within the scope of the patent application shall be covered by the patent.

Claims

1. A continuous human activity recognition method based on multi-domain feature fusion of FMCW radar, characterized in that: The method first processes the raw data acquired by the radar and obtains range-Doppler images and range-angle images by using the fast Fourier transform algorithm and multiple signal classification technology. Then, it uses short-time fractional Fourier transform to map the radar data onto the fractional domain to obtain the fractional domain spectrum. Then, the variable window length STA / LTA continuous action detection algorithm is used to achieve synchronous segmentation of the three types of continuous activity sequence domain information. Finally, the segmented activity samples are used as input to the multi-input multi-task continuous activity recognition model. Input features are extracted through convolutional networks and bidirectional long-term memory networks. A CTC layer is introduced to fuse multi-input representations, and the CTC loss between the predicted sequence and the real label is calculated. Multi-task learning is performed to obtain human action recognition results. The method includes the following steps: Step S1: Use FMCW radar to collect raw human posture data, perform two-dimensional fast Fourier transform on the slow time / fast time data matrix of a single receiving channel of the radar system to obtain a range-Doppler image; organize multiple receiving channels according to time delay and Doppler frequency, calculate the spatial covariance matrix of the received signal and the spatial spectrum distribution of each angle, and construct a range-angle image. Step S2: Apply the cell mean constant false alarm rate method to the range-Doppler image to detect the range of motion distance; for the detected range, apply multiple short-time fractional Fourier transforms in the slow time dimension to obtain the fractional domain spectrum. Step S3: Perform activity detection based on variable window STA / LTA on the range-Doppler image, range-angle image, and fractional domain spectrogram containing continuous human activity information, segment the continuous human activity radar image, and identify the start and end points of human activity; Step S4: Input the segmented human activity data into a convolutional neural network for feature extraction, then use a bidirectional long short-term memory layer to model the three-way features, and use the reshape and concatenate functions to merge the multiple outputs; output the fused high-dimensional human action tensor through a connection-based temporal classification method to obtain the human activity classification result.

2. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 1, characterized in that: In step S1, the distance-Doppler image and the distance-angle image are obtained through the following specific steps: Step S11: After the radar receives the echo signal, it samples it and forms a slow-time / fast-time data matrix; slow time represents the time series of CPI, while fast time represents the time sampling within each CPI; assuming the data matrix is ​​X and the size is M×N, where M represents the slow-time dimension, i.e. the duration of CPI, and N represents the fast-time dimension, i.e. the number of time samples within each CPI. Before performing the 2D FFT, a window function is applied to the data matrix to reduce spectral leakage caused by boundaries; after applying the window function, the data matrix X is obtained after window function processing. w The formula is: X w (i,j)=X(i,j)×w(i,j) Among them, X w (i,j) represents the elements of the data matrix after processing by the window function, X(i,j) represents the elements of the original data matrix, and w(i,j) represents the value of the window function; The data matrix X after window function processing w Performing a 2D FFT converts the time domain to the frequency domain, yielding the range-Doppler plane, which is represented as a range-Doppler image: Y(u,v)=FFT2(X w ) Where Y(u,v) represents the result after 2D FFT, with a size of M×N, and u and v represent the indices in the frequency domain; In addition, cumulative averaging of distance-Doppler images from multiple CPIs enhances the target signal. Among them, Y avg (u,v) represents the result after cumulative averaging, N represents the cumulative number of frames, and Y(u,v) represents the distance-Doppler image of each frame; The processed and enhanced range-Doppler images are combined to form an RD video; each image represents range-Doppler information at one CPI. Step S12: The radar system acquires echo signals through multiple receiving channels. Each receiving channel corresponds to an antenna or array element for receiving signals from different directions. Next, for each CPI, the received signal data is processed. First, the data is organized according to time delay and Doppler frequency to form a time delay-Doppler matrix. For i receiving channels, the received signal data is represented as a complex sequence: [s1(i), s2(i), ..., s N (i)], where N represents the number of sampling points; these signals are organized according to time delay and Doppler frequency to form a time delay-Doppler matrix D. i Its element is represented as D i (t,f)=s i (t,f) The rows of this matrix correspond to different angles, and the columns correspond to different Doppler frequencies; each element represents the received signal strength at a specific angle and Doppler frequency. Based on the time-delay-Doppler matrix, the spatial covariance matrix R of the received signal is calculated; the spatial covariance matrix describes the signal correlation between different receiving channels; the time-delay-Doppler matrices of all receiving channels are concatenated column-wise to form a total time-delay-Doppler matrix D, which has a size of N×M, where N represents the time delay dimension and M represents the Doppler frequency dimension; the spatial covariance matrix R is obtained by calculating the inner product of the conjugate transpose of D and D: R=D H D Among them, D H Represents the conjugate transpose of D; Subsequently, the spatial covariance matrix is ​​further subjected to eigenvalue decomposition to extract its eigenvectors and eigenvalues: R(u k )=λ k u k By calculating the spatial spectral distribution at each angle, the arrival status of the target at different angles is estimated; this is achieved using the MUSIC multi-signal classification algorithm, as shown in the following formula: The MUSIC algorithm calculates the spatial spectral distribution P(θ) for each angle θ using this formula; in this formula, a(θ) is the array manifold vector at angle θ, which describes the response pattern of the antenna array to a signal at a specific angle; K is the number of signals, and λ... k It is an eigenvalue, u k It is the corresponding feature vector; the angle spectrum P(θ) represents the spatial distribution of the signal at different angles, and is used to estimate the arrival angle of the target; By combining the estimated angular spectrum with the distance, a range-Doppler RA image is generated.

3. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 2, characterized in that: In step S12, each CPI is processed by repeating the steps to obtain a series of RA images; these images are combined together according to a predetermined frame rate to form an RA video; the changes in angle and Doppler of the target, as well as the motion characteristics of the target, are observed by playing the RA video.

4. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 1, characterized in that: In step S2, the fractional domain spectrum is obtained through the following specific steps: Step S21: First, the interval of motion distance is detected by using the Cell Mean Constant False Alarm Rate (CA-CFAR) method applied to the distance-Doppler image; for the detected distance interval, multiple Short Time Fractional Fourier Transforms (STFrFT) are applied in the slow time dimension to obtain the fractional domain spectrum.

5. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S31: An activity detection algorithm based on variable window STA / LTA is used to identify the start and end points of continuous motion. First, the absolute difference between the upper and lower envelopes of the time index is calculated based on the fractional domain spectrogram to create an absolute distance vector; the absolute distance i of each data record is calculated as v. i =|u i -l i |, where v i Let u be the absolute distance vector. i and l i These are the upper and lower envelopes, respectively; Step S32: STA(t) and LTA(t) are defined as the leading window and lag window at time t: In the formula, T1 and T2 are the lengths of the short-time window and the long-time window, respectively; the starting point of the detection activity is determined when the following conditions are met: Where σ1 and σ2 are predefined detection thresholds; similarly, if Where σ3 is the detection threshold for the stopping point; To find the starting point, STA(t) needs to exceed the threshold σ1, which means that the activity must appear within the short window; at the same time, the average power ratio of the short window and the long window should be greater than σ2; so that the system will not be triggered in the presence of noise unless the ratio exceeds σ2; ​​and also to ensure that the stopping point of the activity can be correctly detected, that is, when the activity disappears from the processing window and the ratio drops below the threshold σ2. Step S33: Implement the variable window length of STA / LTA by introducing a mask; this mask is a zero vector with the same shape as the input vector; during function execution, the index positions under specific conditions will be set to 1 to indicate that the data at these positions has been processed or belongs to a specific category; when a specific event is detected, i.e., when an activity occurs, the mask replaces 0 with 1 in the corresponding interval; create a zero vector mask with the same shape as the input vector and set its initial value to 0, which is used to mark specific intervals in the input vector after STA / LTA detection; then define a state variable with an initial value of 0, and update the state variable and determine the start and stop positions by iterating through the input vector, comparing the long window value, short window value, and the average power ratio of the short and long windows with the above threshold, and marking the corresponding intervals in the mask; Step S34: After generating a detection mask using the variable window length STA / LTA detection algorithm to segment individual activities in the fractional domain spectral map, individual activities in the range-Doppler video stream and the range-angle video stream also need to be segmented synchronously; therefore, the demask algorithm is used to extract the regions of interest identified by a specific mask matrix from the two-dimensional data matrix.

6. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 5, characterized in that: The demask algorithm accepts three input parameters: two-dimensional data matrices x and y, and a two-dimensional Boolean matrix mask indicating the data intervals of interest within x and y. The specific usage is as follows: First, the algorithm initializes two empty lists x2 and y2 to collect the extracted data intervals; simultaneously, a flag variable initialized to 0 is introduced to track the starting point of the interest interval; next, the algorithm iterates through all rows and columns of the mask matrix; during the traversal, if flag is 0 and the current position of the mask is 0, the position is ignored and the traversal continues; if flag is 1 and the current position of the mask is 1, the algorithm continues; if flag is 1 and the current position of the mask is 0, the algorithm continues; if flag is 1 and the current position of the mask is 0, the algorithm continues; if flag is 1 and the current position of the mask is 0, the algorithm continues; if flag is 1 and the current position of the mask is 0, the algorithm continues; if flag is 1 and the current position of the mask is 0, the algorithm continues; if flag is 1 and the current position of the mask is 1 ... The algorithm sets the mask to 1. If the number of consecutive 1s from the starting point to the current position exceeds two, it indicates the end of an interest interval. At this time, the end position of the interval is recorded, and the corresponding x and y intervals are added to the x2 and y2 lists. When the flag is 0 and the current position mask value is 1, the flag is set to 1, and the current position is set as the starting point. Conversely, if the flag is 1 and the current position mask value is 0, the flag is reset to 0, and the current position is recorded as the end position of the interest interval. Finally, the algorithm returns arrays x2 and y2, which contain the features and labels of the extracted data intervals, respectively.

7. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 1, characterized in that: Step S4 specifically includes the following steps: Step S41: For distance-angle (RA) and distance-Doppler (RD) information, 3D convolution and pooling layers are first used to extract the spatiotemporal features of the video; the convolutional layers apply 3D convolution operations to each time step of the input video sequence to extract the spatiotemporal features of the video; similarly, for the fractional domain spectrogram, spatiotemporal features are extracted through two-dimensional convolution and pooling; then the outputs of the three convolutional layers are processed into a specified shape using the reshape function; and the features are processed through fully connected layers and Dropout layers; finally, a bidirectional LSTM layer is applied to the input sequence. Step S42: After the model extracts the input features, it uses the concatenate function to merge the outputs of the penultimate layer of the three inputs into a tensor, and then connects the merged tensor to an output layer with five classes and a softmax activation function through a fully connected layer. Step S43: Finally, a CTC layer is defined to calculate the CTC loss and added after the output layer; the formula for calculating the CTC loss function is as follows: Given an input sequence X and a corresponding label sequence Y, define Y′ as the label sequence after removing duplicate characters and whitespace characters, i.e., Y′={y|y∈Y,y≠blank}; then the CTC loss is: Loss CTC =-∑log(P(Y′|X)) Wherein, P(Y′|X) represents the probability of predicting the output sequence as Y′ given the input sequence X; during the training process, CTC uses a dynamic programming algorithm to calculate the conditional probability of the model's output sequence and compares it with the true output sequence to calculate the loss function; by minimizing the loss function, the model learns the correspondence between the input sequence and the output sequence, thereby performing accurate sequence labeling prediction.

8. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 1, characterized in that: When the recognition method collects continuous human motion radar dataset and performs human motion detection, the method is as follows: place the millimeter-wave radar directly in front of the subject. Considering the different movement habits of each subject, the subject is allowed to have an angular offset within 120° of the total field of view when facing the millimeter-wave radar. The target data collected is human motion. The radar cross-section (RCS) of the human body facing the millimeter-wave radar is assumed to be 0.5-1m. 2 This makes the detection range applicable to this identification method 5-15 meters.

9. The continuous human activity recognition method based on FMCW radar multi-domain feature fusion according to claim 8, characterized in that: The recognition method collects five human body movements: punching, walking, standing, sitting, and standing up. Subjects are asked to perform corresponding actions in random combinations of these five movements at a distance of 5 meters directly in front of the FMCW radar for data collection and action detection.

Images