Human fall detection method based on feature fusion of lightweight network

Abstract

The application belongs to the field of radar signal processing, and particularly relates to a feature fusion human body fall detection method based on a lightweight network; the method comprises collecting human body action echo signals by using an FMCW radar; a time-distance Doppler graph and a micro-Doppler graph are obtained respectively; a first lightweight network is used to extract a first Doppler feature from the time-distance Doppler graph; a second lightweight network is used to extract a second Doppler feature from the micro-Doppler graph; a splicing module is used to generate splicing features from the first Doppler feature and the second Doppler feature; a self-attention module is used to generate fusion features from the splicing features according to channels; a GRU-based time sequence feature network is used to extract space-time features from the fusion features; and a classifier is used to perform prediction processing on the space-time features to obtain human body fall detection results. The application can fully extract fusion features related to human body falls, thereby improving the accuracy of human body fall detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of radar signal processing, and particularly relates to a feature fusion human fall detection method based on a lightweight network. BACKGROUND

[0002] Fall detection using frequency modulated continuous wave radar mainly includes three parts of radar echo signal collection and preprocessing, feature extraction and classification, and the key of fall detection is to process the micro-Doppler feature corresponding to human action from radar sampling data. In order to realize higher fall detection accuracy, most of the fall detection networks are deep convolutional neural networks with large parameters and large amount of calculation, which are not convenient to carry on hardware. In some real-time requirement high scene, it is not applicable.

[0003] The existing fall detection method mainly uses CNN and some parameter-heavy network models such as some CNNs to detect falls. Although such network models achieve good detection accuracy, they are not suitable for some real-time radar fall detection scenarios due to their large size. SUMMARY

[0004] In view of the above problems, a feature fusion fall detection method based on a lightweight network is proposed, which uses a lightweight network model combined with a feature fusion module to realize feature extraction of human fall action. The research work of the present application is dedicated to solving the problem of large network model parameters and high complexity in FMCW radar human fall detection, so as to realize lightweight fall detection network model under the premise of ensuring fall detection accuracy.

[0005] A feature fusion human fall detection method based on a lightweight network, the method comprising:

[0006] Collecting human action echo signals using FMCW radar;

[0007] Preprocessing the human action echo signals to obtain time-distance Doppler maps and micro-Doppler maps respectively;

[0008] Using a first lightweight network to extract features from the time-distance Doppler maps to extract first Doppler features;

[0009] Using a second lightweight network to extract features from the micro-Doppler maps to extract second Doppler features;

[0010] Using a splicing module to splice the first Doppler features and the second Doppler features to generate spliced features;

[0011] Using a self-attention module to fuse the spliced features according to the channel to generate fused features;

[0012] The fusion features are extracted by using a GRU-based time sequence feature network.

[0013] The spatiotemporal features are predicted by using a classifier to obtain a human fall detection result.

[0014] The present application has the following advantages:

[0015] 1. The present application adopts a double-branch feature extraction network structure to ensure sufficient extraction of spatial information of human motion features.

[0016] 2. The present application uses a lightweight network model MobilenetV3 to extract features from distance-time graphs and micro-Doppler graphs, then uses a self-attention mechanism to further fuse the extracted features, and then uses a GRU network to further extract time sequence information, thereby ensuring the accuracy of human fall detection while using a lightweight network model.

[0017] 3. The present application improves the traditional differential evolution algorithm by using adaptive mutation factors and adaptive crossover factors, which can find individuals that are globally optimal in the early stage and individuals that are locally optimal in the later stage, thereby improving the accuracy of weight parameters in the GRU-based time sequence feature network and improving the accuracy of human fall detection. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a feature fusion human fall detection method flowchart based on a lightweight network according to an embodiment of the present application;

[0019] Figure 2 is a lightweight network structure schematic diagram according to an embodiment of the present application;

[0020] Figure 3 is a self-attention module structure schematic diagram according to an embodiment of the present application;

[0021] Figure 4 is a GRU-based time sequence feature network structure diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0023] Figure 1It is a feature fusion human fall detection method flow chart based on a lightweight network according to an embodiment of the present application; as shown in the figure, the method comprises: Figure 1

[0024] 101, collecting human action echo signals by using FMCW radar;

[0025] In the embodiment of the present application, the collected radar echo human action data set is first preprocessed into a distance-time graph and a micro-Doppler graph. The radar echo signal of human action is a time-varying non-stationary signal, and the echo of the radar contains the signals of multiple reflection points on the human body. Each time point has an echo component. Short Fast Fourier Transform (STFT) is the most commonly used time-frequency transform in radar fall detection. In order to avoid the mutual influence between multiple signal components of time-varying signals, STFT is used for time-frequency analysis of radar echo signals. In the process of processing time-varying non-stationary signals, the signal is first windowed and a segment of signal in the window is approximated as a stationary signal, then FFT is performed on the signal in the window to obtain the frequency distribution in the segment, and the window is moved along the slow time dimension of the signal and overlaps with the previous signal by a certain proportion, so that the frequency distribution of the target echo signal with time can be obtained, and thus the time-frequency spectrum of the target signal can be obtained through time-frequency transform.

[0026] Let the original signal be S(t), then the STFT of S(t) is:

[0027]

[0028] In the formula, w(t) represents the window function, the length of which affects the time and frequency resolution. The greater the window length, the smaller the time resolution, and the greater the frequency resolution. Once the length of the window is determined, the time and frequency resolution are also fixed. τ is the time delay, f is the frequency, and the discrete form of STFT is:

[0029]

[0030] In the formula, S(k) represents the representation form of each segment of S(t), m represents the sampling point, n represents the frequency sampling point, w(k) represents the discrete window function, T represents the time sampling interval, and f s represents the frequency sampling interval.

[0031] 102, preprocessing the human action echo signal to obtain a time-distance Doppler graph and a micro-Doppler graph, respectively;

[0032] ​Based on the above analysis, in the embodiment of the application, the process of obtaining the time-distance Doppler graph includes performing two-dimensional fast Fourier transform on the human action echo signal in the fast time dimension, ignoring the influence of the Doppler frequency, extracting the abscissa frequency corresponding to the spectral peak from each frame of human action echo data to obtain the distance information of the human target; performing two-dimensional fast Fourier transform on the human action echo signal in the slow time dimension, considering the influence of the target motion speed, calculating the Doppler frequency, and calculating the speed information of the human target; combining the distance information of the human target and the speed information of the human target to obtain the range Doppler graph; and arranging the multiple frames of range Doppler graphs corresponding to a single human action in time sequence to form the time-distance Doppler graph.

[0033] Based on the above analysis, in the embodiment of the application, the process of obtaining the micro-Doppler graph includes performing STFT transform on the human action radar echo data. In the process of processing the human action echo signal, the signal is first windowed and a segment of signal in the window is approximated as a stationary signal, then the frequency distribution in the segment is obtained by performing FFT on the signal in the window, and the window is moved along the slow time dimension of the signal and overlaps with the previous signal by a certain proportion, so that the frequency distribution of the target echo signal with time is obtained, thereby obtaining the micro-Doppler graph of the target signal through time-frequency transform.

[0034] 103. performing feature extraction on the time-distance Doppler graph by using a first lightweight network to extract first Doppler features;

[0035] 104. performing feature extraction on the micro-Doppler graph by using a second lightweight network to extract second Doppler features;

[0036] In the embodiment of the application, the first lightweight network and the second lightweight network can adopt the same type of network, for example, can be a MobilenetV3 network, a Shufflenet network or a SqueezeNet network, etc.

[0037] Take the Mobilenet network as an example, the MobileNet series is a lightweight network model proposed by the Google team in 2017, aiming to greatly reduce the size of the model and speed up the operation of the model while not sacrificing the performance of the model too much. MobileNetV1 was first proposed, and its design idea is to replace the regular convolution in the VGG network model with a depth separable convolution, which reduces the parameter quantity and calculation quantity of the model while improving the speed of the model on the basis of ensuring the performance of the model. However, the depth convolution has poor effect and is only suitable for the low layer of the network. Therefore, MobileNetV2 was proposed, which introduced a linear bottleneck and an inverted residual structure on the basis of MobileNetV1, and improved the efficiency of the layer structure by utilizing the low rank property of the problem. The inverted residual block in MobileNetV2 is different from the standard residual block in ResNet, which first compresses, then extracts features by convolution, and finally expands the channel structure. The inverted residual block of MobileNetV2 adopts the structure of first expanding the channel, then extracting features by convolution, and finally compressing, and in order to solve the information loss problem caused by low-dimensional ReLU, the last ReLU function is replaced by a linear function. MobileNetV3 adds an attention mechanism module on the basis of MobileNetV1 and MobileNetV2. The feature of MobileNetV3 is that it has fewer parameters and less calculation, and is more suitable for scenarios with limited storage space and power consumption.

[0038] As shown in Figure 2 The MobilenetV3 network includes an input layer, an intermediate layer and an output layer, the intermediate layer includes a convolution layer Con3_s2, a bneck layer, an average pooling layer and a full connection layer. Among them, the bneck layer is internally replaced by a depth separable convolution to realize the lightweight of MobilenetV3, and the average pooling layer reduces the dimension of the extracted features by downsampling.

[0039] Based on the above analysis, two MobilenetV3 networks are used to extract Doppler features of time-distance Doppler maps and micro-Doppler maps respectively.

[0040] 105, using a splicing module to perform feature splicing on the first Doppler feature and the second Doppler feature to generate spliced features;

[0041] In the embodiment of the application, the feature splicing of the first Doppler feature and the second Doppler feature includes corresponding splicing of the first Doppler feature and the second Doppler feature according to the same channel.

[0042] 106, using a self-attention module to perform feature fusion on the spliced features according to the channel to generate fused features;

[0043] In this embodiment of the invention, the feature fusion of the spliced ​​features according to channels using the self-attention module includes concatenating the spliced ​​features of each channel to reshape a channel feature map; multiplying the channel feature map with its transpose feature map to obtain an inter-channel correlation matrix; processing the inter-channel correlation matrix to obtain an inter-pixel correlation matrix; applying the transpose of the inter-pixel correlation matrix to the channel feature map to reshape an attention feature map; and performing a weighted summation of the spliced ​​features and the attention feature map to generate a fused feature.

[0044] After the splicing features are determined, the multi-channel feature map is input into the self-attention-based feature fusion module, which is as follows: Figure 3 The self-attention module shown uses labels, values, and query structures derived from the extracted action feature map. The module's computational flow is as follows, assuming the input concatenated features are:

[0045] X∈R h×w×c

[0046] In the formula, h×w is the size of the feature map, and c is the number of channels in the feature map. By connecting and reshaping the data from each channel, we obtain:

[0047] X c ∈R n×c

[0048] In the formula, n = h × w. X c Rather than transpose Multiplying them together, we obtain the inter-channel correlation matrix M:

[0049]

[0050] Then, Softmax processing is performed on each column element of M. Let M be an example. ij If the elements are M, then the processed inter-pixel correlation matrix N∈R c×c The value is:

[0051]

[0052] Applying it to the feature matrix yields the attention spectrum as follows:

[0053] X c1 =X c ·N T

[0054] Transform the fused attention spectrum to its original size:

[0055] X c1 ∈R n×c →X c2 ∈R h×w×c

[0056] Finally, the original feature map is combined with the attention spectrum using a weighted summation method. This avoids the numerical attenuation of the original feature map caused by direct multiplication, which would make the output feature map information completely dependent on the attention spectrum information. The weighted summation formula is as follows:

[0057] X f =αX c2 +X

[0058] In the formula, α is a learnable parameter, whose main function is to control the ratio between the two.

[0059] The above outlines all the steps of the attention module. It generates an attention matrix using its own input data, then applies this matrix to the input data, adding output attention data. The parameter α controls the proportion of attention in the output matrix, allowing the convolutional neural network to adjust the proportion of this mechanism during training, adaptively controlling the module's function. The attention mechanism primarily targets the inter-channel information of the input matrix, enabling better fusion of feature channels between action data features, extracting more useful feature information, and improving network performance.

[0060] 107. Use a GRU-based temporal feature network to extract spatiotemporal features from the fused features;

[0061] In this embodiment of the invention, a temporal feature network based on GRU is established. GRU (Gated Recurrent Unit) is a recurrent neural network that extracts information features based on existing historical data for prediction. Since GRU is a variant of LSTM, it can also solve the long dependency problem in RNNs (Recurrent Neural Networks). It has an internal mechanism called "gates." The GRU network incorporates hidden states, update gates, and reset gates. These gates determine which important data needs to be retained and which needs to be deleted. This more complex information transmission method effectively overcomes the shortcomings of traditional RNNs. Therefore, it can be used to predict the future actions of human nodes based on their historical action data.

[0062] By constructing a gated recurrent unit network, the network takes the current fused feature and the hidden state transmitted from the previous fused feature as input, and outputs the spatiotemporal features and hidden state at the current time. The gated recurrent unit network obtains the gated state based on the hidden state transmitted from the fused feature at the next time and the fused feature input at the current time. The gated state is updated through forward propagation, and the weight parameters to be trained are determined. The partial derivatives of the loss function with respect to each weight parameter are obtained through the backpropagation algorithm, and the weight parameters are updated until training is completed.

[0063] The input-output structure of a GRU-based temporal feature network is as follows: Figure 4 As shown. x t h represents a current input, which in this method is the fused feature at the current time step. t-1 This represents the hidden state passed down from the previous node, which contains relevant information about the previous node. Combined with x... t and h t-1 GRU will obtain the output y of the currently hidden node. t and the hidden state h passed to the next node t Here, the nodes correspond to time points.

[0064] Since GRU networks involve extensive training processes, and the selection of weight parameters significantly impacts model accuracy, potentially leading to convergence issues, this invention uses the weight parameters of the gated recurrent unit network as individuals in the differential evolution algorithm population. The population is initialized; the fitness value of each individual with weight is calculated; it is determined whether the termination condition or the maximum number of generations has been reached; if so, evolution stops, and the best individual is output as the optimal value of the weight parameters; otherwise, evolution continues; the individuals with weight are mutated and crossoverdine to obtain an intermediate population; individuals are selected from the previous generation and the intermediate population to obtain a new generation population, and the fitness value of each individual with weight in the new generation population is calculated until evolution stops.

[0065] In a preferred embodiment of the present invention, the fitness function is set as follows:

[0066]

[0067] Where f(i) represents the fitness value of the i-th individual with weight; N is the number of output neurons in the GRU network. Let be the true value corresponding to the i-th weight in the k-th layer output neuron. This is the predicted value corresponding to the i-th weight in the k-th layer output neuron. Each training process will be accompanied by a set of weight parameters.

[0068] In a preferred embodiment of the present invention, in order to further improve the prediction effect, the present invention uses an adaptive mutation factor to perform mutation operation on the weight individuals; and uses an adaptive crossover factor to perform crossover operation on the weight individuals.

[0069] The adaptive mutation factor is expressed as:

[0070]

[0071] Among them, X k X is the variant factor in the kth generation; max X is the largest variable factor;min e is the minimum mutation factor; e is the current evolutionary number; e max This represents the maximum number of evolutions.

[0072] The adaptive crossover factor is expressed as:

[0073]

[0074] Among them, V k V is the crossover factor of the kth generation; max V is the maximum crossover factor; min It is the minimum crossover factor.

[0075] In this embodiment, a strong global optimization capability is required in the early stages of algorithm evolution, while the entire population should have a stronger local search capability in the later stages. Therefore, an adaptive mutation factor and an adaptive crossover factor are used to improve the differential evolution algorithm, so that the factors are continuously updated with the number of iterations. This helps to balance the global search capability and local search capability of the differential evolution algorithm, thereby improving the prediction accuracy and convergence speed of the GRU-based temporal feature network, and ultimately improving the accuracy of human fall detection.

[0076] 108. The spatiotemporal features are predicted using a classifier to obtain the human fall detection results.

[0077] In this embodiment of the invention, spatiotemporal features can be input into a classifier, such as a softmax layer, for classification and prediction to obtain the human fall detection result.

[0078] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A feature fusion method for human fall detection based on lightweight networks, characterized in that, The method includes: Using FMCW radar to collect human motion echo signals; The human motion echo signal is preprocessed to obtain time-distance Doppler images and micro-Doppler images, respectively; The time-distance Doppler image is used to extract features using a first lightweight network to extract the first Doppler features; The micro-Doppler image is used to extract features, and the second Doppler features are extracted. The first Doppler feature and the second Doppler feature are spliced ​​together using a splicing module to generate spliced ​​features; The spliced ​​features are fused according to channels using a self-attention module to generate fused features; The spatiotemporal features are extracted from the fused features using a GRU-based temporal feature network. The step of using a GRU-based temporal feature network to extract features from the fused features includes constructing a gated recurrent unit network (GRU), inputting the fused features at the current time step and the hidden state transmitted from the fused features at the previous time step, and outputting the spatiotemporal features and hidden state at the current time step; the GRU network obtains a gated state based on the hidden state transmitted from the fused features at the next time step and the fused features input at the current time step; the GRU network updates the gated state through forward propagation, determines the weight parameters to be trained, obtains the partial derivatives of the loss function with respect to each weight parameter through a backpropagation algorithm, and updates the weight parameters until training is complete; The training process of the GRU-based temporal feature network includes: initializing the population by using the weight parameters of the gated recurrent unit network as individuals in the differential evolution algorithm; calculating the fitness value of each individual in the population; determining whether the termination condition or the maximum number of generations has been reached; if so, stopping the evolution and outputting the best individual as the optimal value of the weight parameter; if not, continuing the evolution; performing mutation and crossover on the individuals to obtain an intermediate population; selecting individuals from the previous generation and the intermediate population to obtain a new generation population, and continuing to calculate the fitness value of each individual in the new generation population. The mutation and crossover of the weight individuals include mutating the weight individuals using an adaptive mutation factor and crossing the weight individuals using an adaptive crossover factor. The adaptive mutation factor is expressed as: Among them, X k X is the variant factor in the kth generation; max X is the largest variable factor; min e is the minimum mutation factor; e is the current evolutionary number; e max This represents the maximum number of evolutions. The adaptive crossover factor is expressed as: Among them, V k V is the crossover factor of the kth generation; max V is the maximum crossover factor; min It is the minimum crossover factor; The spatiotemporal features are used to predict and process the data to obtain the results of human fall detection.

2. The human fall detection method based on feature fusion using a lightweight network according to claim 1, characterized in that, The process of obtaining the time-distance Doppler map includes performing a two-dimensional fast Fourier transform on the human motion echo signal in a fast time dimension, ignoring the influence of the Doppler frequency, and extracting the horizontal axis frequency corresponding to the spectral peak of each frame of human motion echo data to obtain the distance information of the human target. A two-dimensional fast Fourier transform is performed on the human motion echo signal in a slow time dimension to consider the influence of the target's motion speed, and the Doppler frequency is calculated, as well as the velocity information of the human target. The distance information and velocity information of the human target are combined to obtain a range Doppler map. The multiple frames of range Doppler maps corresponding to a single human motion are arranged in chronological order to form a time-range Doppler map.

3. The human fall detection method based on feature fusion using a lightweight network according to claim 1, characterized in that, The process of obtaining the micro-Doppler image involves applying a STFT to the human motion echo signal. In processing the human motion echo signal, the STFT first windows the signal and approximates a segment of the signal within the window as a stationary signal. Then, it performs an FFT transformation on the signal within the window to obtain the frequency distribution of the signal within that segment. The window moves along the slow time dimension of the signal and overlaps with the signal within the previous window by a certain proportion to obtain the frequency distribution of the target echo signal over time. Thus, the micro-Doppler image of the target signal is obtained through time-frequency transformation.

4. The human fall detection method based on feature fusion using a lightweight network according to claim 1, characterized in that, The first lightweight network and the second lightweight network are the same type of lightweight network, including MobilenetV3 network, Shufflenet network or SqueezeNet network.

5. The human fall detection method based on feature fusion using a lightweight network according to claim 1, characterized in that, The feature stitching of the first Doppler feature and the second Doppler feature using the stitching module includes stitching the first Doppler feature and the second Doppler feature correspondingly according to the same channel.

6. The human fall detection method based on feature fusion using a lightweight network according to claim 1, characterized in that, The step of using a self-attention module to perform feature fusion on the spliced ​​features according to channels includes concatenating the spliced ​​features of each channel to reshape a channel feature map; multiplying the channel feature map with its transpose feature map to obtain an inter-channel correlation matrix; processing the inter-channel correlation matrix to obtain an inter-pixel correlation matrix; and applying the transpose of the inter-pixel correlation matrix to the channel feature map to reshape an attention feature map. The spliced ​​features and the attention feature map are weighted and summed to generate fused features.

Images