Human fall recognition method based on radar range Doppler map and IR-ST

Abstract

This paper proposes a method for human fall recognition based on radar range Doppler maps and IR‑ST. First, a 60GHz linear frequency modulated continuous wave radar is used to collect radar raw data for seven actions: sitting up, jumping, sitting still, stepping, squatting, bending over, and falling. Second, a mean velocity projection algorithm is used to extract the range Doppler map from the radar raw data. At the same time, the background noise in the range Doppler map is reduced to highlight important motion information. The range Doppler map extracted by the mean velocity projection algorithm is used to construct a fall dataset. Finally, a neural network combining an inverted residual convolution module and a sliding self-attention mechanism is designed for classification. The results show that the present invention comprehensively considers the frequency, amplitude, and periodic characteristics corresponding to human movements and range Doppler maps, can effectively classify different human movements in the same scene, monitor human behavior at home, and accurately identify falls.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to neural network processing and recognition of signals. Background Art

[0002] The results of the seventh national census show that my country has 264.02 million people aged 60 and over, accounting for 18.7% of the national population. Compared to the results of the sixth national census in 2010, the proportion of people aged 60 and over has increased by 5.44%. Because many elderly people suffer from conditions such as osteoporosis, falls often lead to injuries. According to the World Health Organization, falls are the second leading cause of unintentional death worldwide, with the largest proportion occurring among those aged 60 and over. Therefore, research on fall detection has positive practical significance.

[0003] In recent years, research on human falls has become a hot topic. Based on the different detection devices, these methods can be divided into two categories: those based on contact devices (accelerometers, pressure sensors, gyroscopes, etc.) and those based on non-contact devices (Wi-Fi, infrared array sensors, cameras, radar, etc.). Radar-based methods are gaining popularity due to their strong penetration, privacy protection, and immunity to light interference. Radar detection equipment is also gradually appearing on the market, showing broad market prospects.

[0004] Radar-based human fall detection methods can be divided into two categories: traditional machine learning and deep learning. Traditional machine learning methods convert radar echo data into feature spectra and manually extract effective features based on experience and domain knowledge. These methods then use statistically based machine learning algorithms to detect falls. Classification accuracy depends on the quality of feature extraction and is only applicable to specific situations. Deep learning methods, on the other hand, utilize a multi-hidden layer network model trained with extensive data to automatically extract deep features and perform action recognition tasks, demonstrating strong robustness and generalization capabilities.

[0005] Using deep learning to process radar data is the future, but its development still faces a series of bottlenecks that need to be overcome. Ma et al. used a network that fused a convolutional neural network with a long short-term memory network, achieving 95% accuracy in classifying six types of actions. Wang et al. used a convolutional neural network combined with a long short-term memory network, achieving 92.65% accuracy in classifying six types of actions. Currently, the mainstream neural network architectures are convolutional neural networks and long short-term memory networks. However, convolutional neural networks can only focus on local features and cannot capture global features, while long short-term memory networks take too long to train on long sequences of data. Summary of the Invention

[0006] The main technical problem solved by the present invention is to provide a human fall recognition method based on radar range Doppler map and IR-ST. It can use the original radar signal to extract the characteristics of the human body, including falls and seven daily movements, and design a lightweight neural network model (IR-ST) for movement classification, thereby improving the reliability of the home fall monitoring system.

[0007] In order to solve the above technical problems, the technical solution provided by the present invention is:

[0008] The human fall recognition method based on radar range Doppler map and IR-ST has the following steps:

[0009] Step 1: Use a 60GHz linear frequency modulated continuous wave radar to collect radar raw data for seven actions: sit-up, jump, sit still, step, squat, bend over, and fall;

[0010] Step 2: Use the mean velocity projection algorithm to extract the range Doppler map from the raw data, reduce background noise, and extract motion information;

[0011] Step 3: Design an IR-ST model that combines the inverted residual convolution structure with the window self-attention mechanism and uses the range Doppler map to classify human motions;

[0012] In step 1, seven actions are designed: sit-up, jump, sit still, step, squat, bend over, and fall. Multiple sets of raw data are collected for each action. The radar generates a linear frequency modulation signal through a synthesizer and transmits it through the transmitting antenna TX. The frequency of the signal will increase from 60GHz to 64GHz within a certain period. This period is called a chirp cycle. The frequency change within the chirp cycle is Expressed as:

[0013]

[0014] Where, is the starting frequency of the radar, is the slope of the frequency increase within a chirp period, which can be obtained from the following formula

[0015]

[0016] in is the bandwidth within the radar signal chirp period, is the duration of a chirp cycle, measured by cosine After function processing, it is transmitted by the transmitting antenna TX.

[0017] When the transmitted signal is reflected by the environment and the target, it is received by the receiving antenna RX. The received signal frequency Expressed as:

[0018]

[0019] Where, is the time delay, which can be expressed as follows:

[0020]

[0021] Where, is the distance between the radar and the object, The speed of light is achieved by mixing the transmitting signal and the receiving signal through a mixer to output an intermediate frequency signal, which converts two sinusoidal signals into and Input into the mixer, intermediate frequency signal is the difference between the instantaneous frequencies of the two input sinusoidal signals. The intermediate frequency signal can be expressed as follows:

[0022]

[0023] The intermediate frequency signal obtained through several chirp cycles constitutes a sampling cycle, and several sampling cycles constitute a set of original data;

[0024] The mean velocity projection algorithm used in step 2 extracts the range Doppler map from the original radar signal. The mean velocity projection algorithm consists of two parts: velocity projection algorithm and mean filtering.

[0025] The velocity projection algorithm can be divided into two parts: range Fourier transform and Doppler Fourier transform. The ordinate is composed of a sampling period. Since each frequency sweep period is very short, the velocity information in each chirp period can be ignored. The range Fourier transform refers to extracting the abscissa frequency corresponding to the spectrum peak of each chirp period on the ordinate to obtain the target distance information. The distance calculation formula is as follows:

[0026]

[0027] in is the speed of light, is the frequency sweep period, is the sweep bandwidth, is the frequency of the radar beat signal in motion or still conditions,

[0028] The horizontal axis of the range Doppler map is composed of multiple sampling periods. By stacking multiple frames of data for multiple chirp periods, the velocity information in the horizontal axis cannot be ignored.

[0029] Doppler Fourier transform refers to obtaining the frequency of the data on the horizontal axis, that is, the Doppler frequency. The velocity calculation formula is as follows:

[0030]

[0031] in is the Doppler frequency, is the center frequency of the chirp periodic signal, and the velocity projection algorithm is used to rearrange the radar original signal to obtain the range Doppler matrix;

[0032] The mean filtering method is expressed as follows:

[0033]

[0034] Where x is any value in the Doppler matrix, X is the range Doppler matrix, and the average value of the range Doppler function is obtained by the Average function. The function makes a judgment. If x is less than the matrix average, x is set to 0, otherwise x=x;

[0035] Converting the values ​​in the range-Doppler matrix into different colors can generate a range-Doppler map. Using this map, a radar fall detection dataset is constructed.

[0036] The IR-ST network construction steps in step 3 are as follows:

[0037] Step 1: The radar range-Doppler map is passed through five inverse residual convolution modules, where a shortcut branch is used to add the output of the second module to the third output. The shortcut branch means that when the input size and number of channels of the neural network module are the same, the input matrix and the output matrix of the model are added together to obtain a new matrix as the input of the next module.

[0038] Step 2: The extracted feature matrix is ​​input into the Patch Partion layer. The Patch Partion layer reduces the matrix size and increases the depth through the convolution kernel.

[0039] Step 3: Through the Linear Embedded layer, the Linear Embedded layer increases the dimension of the input matrix through the convolution kernel to increase the number of channels;

[0040] Step 4: Slide the self-attention mechanism module twice;

[0041] Step 5: Pass the output through the Patch Merging layer, which reduces the input size by half and doubles the number of channels;

[0042] Step 6: Through the Adaptive Pooling layer, the Adaptive Pooling layer uses the convolution kernel to convert the input matrix size into a 1×1 matrix;

[0043] Step 7: Output after processing through the fully connected layer. The fully connected layer: merges multiple convolutional layers into a fully connected layer. The goal of the fully connected layer is to flatten the high-level features learned by the convolutional layer.

[0044] The inverted residual convolution module: first, the input matrix is ​​increased in dimension by a point-by-point convolution layer, that is, the number of channels is increased, then the feature extraction is performed by a depth-wise convolution layer, and finally the dimension is reduced by a point-by-point convolution layer, and the number of channels is reduced;

[0045] Convolution layer: First, perform a convolution operation on each channel, then add the H convolution feature matrices obtained by processing to form a channel, and finally repeat the above operation H+1 times to convert the original H channel image into H+1 channel.

[0046] Depthwise convolution: A convolution kernel in this convolution layer is responsible for only one channel. After the output of this layer, the number of channels of the image remains unchanged.

[0047] Point-by-point convolution: Use a 1×1 convolution kernel to extract and add features from each channel of the input matrix, and control the increase or decrease of the number of channels in the output matrix by changing the number of convolution kernels.

[0048] The sliding self-attention mechanism module consists of two small modules, each of which has four parts: two LN layers, MSA module, and MLP module.

[0049] The LN layer performs a normalization operation on the input matrix. Each value in the matrix is ​​subtracted from the matrix mean and divided by the matrix standard deviation.

[0050] The MSA module is divided into a window-based W-MSA module and a sliding window-based SW-MSA module. During initialization, the W-MSA module divides the image into multiple modules, each of which uses a self-attention mechanism. As the network deepens, small modules are spliced ​​together to form larger modules, and the downsampling rate also increases with the network length.

[0051] The self-attention mechanism is expressed as follows:

[0052]

[0053] in For matching items, For key items, is the information extracted from the input by the self-attention mechanism, refers to The number of elements in The function will The calculated matrix is ​​normalized so that the sum of its elements is equal to 1, and finally multiplied by the information extracted from the input by the self-attention mechanism ;

[0054] MLP module, also known as multi-layer perceptron, is composed of input layer, hidden layer, and output layer, and the layers are fully connected.

[0055] The advantages of the present invention are:

[0056] 1. The original signal collected by the radar is processed by the RDMA algorithm to obtain a range Doppler map with lower background noise and more obvious motion characteristics.

[0057] 2. Design a new neural network that combines CNN and sliding self-attention mechanism for classification, and obtain a neural network model that takes into account accuracy, model size and computing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the present invention for identifying seven human body movements based on radar original signals.

[0059] Figure 2 Schematic diagram of the radar range Doppler matrix of the present invention.

[0060] Figure 3 These are seven human body movements and their corresponding range Doppler images of the present invention.

[0061] Figure 4 It is a schematic diagram of the working principle of the convolutional layer of the present invention.

[0062] Figure 5 This is the Patch Merging layer flow chart of the present invention.

[0063] Figure 6 It is the main structure of the IR-ST network model of the present invention.

[0064] Figure 7 This is the inverted residual network structure diagram of the present invention.

[0065] Figure 8 It is a schematic diagram of the common convolution operation principle of the present invention.

[0066] Figure 9 It is a schematic diagram of the depth convolution operation principle of the present invention.

[0067] Figure 10 It is a schematic diagram of the point-by-point convolution operation principle of the present invention.

[0068] Figure 11 It is the sliding self-attention mechanism module of the present invention.

[0069] Figure 12 This is the effect diagram of the W-MSA module segmentation picture of the present invention.

[0070] Figure 13 This is the SW-MSA module picture reorganization of the present invention.

[0071] Figure 14 It is a structural diagram of the MLP of the present invention.

[0072] Figure 15 It is a classification diagram of the prediction results of the present invention.

[0073] Figure 16 It is the classification result of the IR-ST neural network of the present invention.

[0074] Figure 17 This is a comparison chart of the accuracy and model size of the IR-ST neural network model of the present invention and other classifiers. DETAILED DESCRIPTION

[0075] The present invention will be further described in detail below with reference to the accompanying drawings.

[0076] This paper proposes a human fall motion classification method based on range-Doppler maps extracted from raw radar signals and a lightweight neural network. First, a 60GHz linear frequency-modulated continuous wave radar is used to collect radar raw data for seven motions: sit-up, jump, sit-still, step, squat, bend, and fall. Second, a mean velocity projection algorithm is used to extract range-Doppler maps from the raw radar data. This algorithm reduces background noise in the range-Doppler maps and highlights important motion information. A fall dataset is constructed from the range-Doppler maps extracted by the mean velocity projection algorithm. Finally, a neural network combining an inverted residual convolution module and a sliding self-attention mechanism is designed for classification. Results show that this method, by comprehensively considering the frequency, amplitude, and periodicity characteristics of human motion and range-Doppler maps, combined with a lightweight neural network approach, can effectively classify different human motions in the same scenario. The resulting neural network model balances accuracy and model size, demonstrating good accuracy and reliability in monitoring human behavior in the home and accurately identifying falls.

[0077] like Figure 1 As shown in the figure, the overall process of a human fall action classification method based on extracting the range Doppler map of the radar raw signal and a lightweight neural network is as follows:

[0078] Step 1: Use a 60GHz linear frequency modulation continuous wave radar to collect radar raw data for seven actions: "sit-up," "jump," "sit still," "step," "squat," "bend," and "fall." Each data set contains 100 sampling cycles, and each cycle contains 128 chirp cycles. The radar generates a linear frequency modulation signal through a synthesizer and transmits it from the transmitting antenna (TX). The signal frequency increases from 60GHz to 64GHz within a certain period, which is called a chirp cycle. The frequency change within a chirp cycle is It can be expressed as:

[0079]

[0080] Where, is the starting frequency of the radar, is the slope of the frequency increase within a chirp period, which can be obtained from the following formula

[0081]

[0082] in is the bandwidth within the radar signal chirp period, is the duration of one chirp cycle. After function processing, it is transmitted by the transmitting antenna (TX).

[0083] When the transmitted signal is reflected by the environment and the target, it is received by the receiving antenna (RX). The received signal frequency It can be expressed as:

[0084]

[0085] Where, is the time delay, which can be expressed as follows:

[0086]

[0087] Where, is the distance between the radar and the object, The signal transmission signal and the received signal are mixed by the mixer to output the intermediate frequency signal, which is the two sinusoidal signals. and Input into the mixer, intermediate frequency signal is the difference between the instantaneous frequencies of the two input sinusoidal signals. The intermediate frequency signal can be expressed as follows:

[0088]

[0089] In the present invention, the intermediate frequency signal obtained through 128 chirp cycles can constitute a sampling cycle, and can constitute a set of original data through 100 sampling cycles.

[0090] Step 2: If Figure 2 As shown, the present invention uses a mean velocity projection algorithm to extract a range Doppler map from the original signal of the radar. The mean velocity projection algorithm mainly consists of two parts: a velocity projection algorithm and a mean filter.

[0091] The velocity projection algorithm can be divided into two parts: the range Fourier transform and the Doppler Fourier transform. The vertical axis is composed of a sampling period. Since each frequency sweep period is very short, the velocity information in each chirp period can be ignored. The range Fourier transform extracts the horizontal axis frequency corresponding to the spectrum peak of each chirp period on the vertical axis to obtain the target distance information. The distance calculation formula is as follows:

[0092]

[0093] in is the speed of light, is the frequency sweep period, is the sweep bandwidth, is the frequency of the radar beat signal in motion or stationary conditions.

[0094] The horizontal axis of the range-Doppler plot is composed of multiple sampling periods. By accumulating multiple frames of data over multiple chirp periods, velocity information cannot be ignored in the horizontal axis. The Doppler Fourier transform is to obtain the frequency of the data on the horizontal axis, which is the Doppler frequency. The velocity calculation formula is as follows:

[0095]

[0096] in is the Doppler frequency, is the center frequency of the chirp periodic signal. The velocity projection algorithm is used to rearrange the original radar signal to obtain the range Doppler matrix.

[0097] However, there is a certain amount of environmental interference in the collected Doppler matrix. This application adopts a mean filtering method, which can be expressed as the following formula:

[0098]

[0099] Where x is any value in the Doppler matrix, and X is the range Doppler matrix. The average value of the range Doppler function is obtained by the Average function. The function makes a judgment. If x is less than the matrix average, x is set to 0, otherwise x=x.

[0100] The radar range-Doppler matrix is ​​obtained by processing the raw data using the mean velocity projection algorithm. The horizontal axis represents 100 sampling periods, and the vertical axis represents 128 chirps per sampling period. The range-Doppler map is obtained by converting the values ​​in the matrix into different colors.

[0101] In this paper, we designed seven human body movements, including "sitting-up" (a), "jumping" (b), "sitting still" (c), "stepping" (d), "squatting" (e), "bending" (f) and "falling" (g), with a total of 2297 range Doppler images.

[0102] like Figure 3 As shown in the figure, each action and its corresponding range Doppler map are listed from left to right. Figure 3 We can see that different human body movements have different range Doppler patterns. In the embodiment of the present invention, to simulate the interference in daily environment, each movement is completed in a daily laboratory environment, and the people in the laboratory can move freely during the measurement of data.

[0103] Step 3: Design a neural network (IR-ST) model that combines the inverted residual structure with the window self-attention mechanism and combines it with the range Doppler map to perform human motion classification.

[0104] The overall structure of the IR-ST network designed by the present invention is as follows:

[0105] The structure diagram of IR-ST is as follows Figure 6 shown.

[0106] Step 1: The radar range-Doppler map is passed through five inverted residual blocks, using a shortcut branch to add the output of the second block to the output of the third block. A shortcut branch involves adding the model's input matrix to its output matrix when the input size and number of channels of the neural network blocks are the same, generating a new matrix that serves as the input to the next block.

[0107] Step 2: Input the extracted feature matrix into the Patch Partion layer (module segmentation layer). The Patch Partion layer uses a convolution kernel with a size of 4×4, a stride of 4, and 48 channels to reduce the matrix size to 4 times the original size and increase the depth by 16 times.

[0108] The convolution kernel operation principle, such as Figure 4 As shown:

[0109] In the figure, a 3×3 convolution kernel is selected and multiplied with a 3×3 interval in the image pixel matrix, and then added to obtain a convolution eigenvalue. The convolution kernel is shifted to the right by one step, and the convolution kernel can be dot-multiplied with the new 3×3 interval and then added to obtain a new convolution eigenvalue. Similarly, all convolution eigenvalues ​​can be arranged into a convolution feature matrix, whose size N can be calculated by the following formula:

[0110]

[0111] in Represents the size of the input image, Represents the size of the convolution kernel, represents the stride, i.e., the length of one convolution kernel movement. The number of channels in the convolution feature matrix completed above is 1. When there are X convolution kernels performing the above operation, the output convolution feature matrix has X channels.

[0112] Step 3: Through the Linear Embedding layer, the Linear Embedding layer performs a dimensionality increase operation on the input matrix through a convolution kernel with a size of 1×1, a stride of 1, and a channel number of 96, increasing the number of channels to 96.

[0113] Step 4: Slide the self-attention mechanism module twice.

[0114] Step 5: Pass the output through the Patch Merging layer (module splicing layer). The function of the Patch Merging layer is to reduce the input size by half and double the number of channels.

[0115] The Patch Merging layer process is as follows Figure 5 As shown in , first divide the input matrix into 16 modules, label each module, and splice the modules with the same label together as a channel. Figure 5 There are four labels in , so the concatenated matrix has 4 channels. Then the normalization operation is performed. The normalization principle is as follows:

[0116]

[0117] in is the input matrix, is a matrix The average of is a matrix The variance of It is a very small value, and its main function is to prevent the denominator from being 0. Finally, the output matrix is ​​obtained by reducing the number of channels to half of the original number.

[0118] Step 6: Through the Adaptive Pooling layer, the Adaptive Pooling layer converts the input matrix size into a 1×1 matrix with 768 channels through a convolution kernel with a size of 7×7, a stride of 1, and 768 channels.

[0119] Step 7: Output after processing through the fully connected layer. Fully connected layer: Combine multiple convolutional layers into a fully connected layer. The goal of the fully connected layer is to flatten the high-level features learned by the convolutional layer. It combines all the characteristics of the input. The output vector of the fully connected layer It can be described as:

[0120]

[0121]

[0122] In the formula , and Respectively The input vector, weight matrix and bias vector of the fully connected layer.

[0123] The following introduces the two main modules of the inverted residual convolution and sliding self-attention mechanism in the first and fourth steps.

[0124] 1. Inverted residual convolution module: such as Figure 7 As shown in Figure 1. First, a point-wise convolutional layer of size 1 is used to increase the dimensionality of the input matrix, i.e., to increase the number of channels. Feature extraction is then performed through a 3×3 depthwise convolutional layer, and finally, a point-wise convolutional layer is used to reduce the dimensionality and number of channels.

[0125] Convolutional layer: such as Figure 8 As shown in the figure, the color image has three channels. First, a convolution operation is performed on each channel. Then, the three convolution feature matrices obtained are added together to form one channel. Finally, the above operation is repeated four times to convert the original three-channel image into a four-channel image.

[0126] Depthwise convolution, such as Figure 9 As shown in the figure, a convolution kernel in this convolution layer is responsible for only one channel. After the output of this layer, the number of channels of the image remains unchanged.

[0127] Point-wise convolution, such as Figure 10 As shown in the figure, a 1×1 convolution kernel is used to extract and add features from each channel of the input matrix, and the increase or decrease of the number of channels in the output matrix is ​​controlled by changing the number of convolution kernels.

[0128] Using depth + point-by-point convolution can reduce more computational complexity than using ordinary convolution operations. The formula for the ratio of depth + point-by-point convolution to ordinary convolution is as follows:

[0129]

[0130] In the formula represents the size of the convolution kernel, The size of its input matrix, is the number of channels of the input matrix, Indicates the number of convolution kernels. The present invention uses a depth convolution kernel size of 3×3, which can be deduced from the above formula 6. The theoretical calculation amount is equal to that of the ordinary convolution layer. .

[0131] Convolution operation principle: Figure 4As shown in the figure, a 3×3 convolution kernel is selected and multiplied with a 3×3 interval in the image pixel matrix and then added to obtain a convolution eigenvalue. The convolution kernel is shifted to the right by one step, and the convolution kernel can be dot-multiplied with the new 3×3 interval and then added to obtain a new convolution eigenvalue. Similarly, all convolution eigenvalues ​​can be arranged into a convolution feature matrix, and its size N can be calculated by formula 3: Represents the size of the input image, Represents the size of the convolution kernel, Represents the step size, which is the length of the convolution kernel movement once.

[0132] 2. Sliding self-attention mechanism module. Its structure is as follows Figure 11 As shown in the figure, the sliding self-attention mechanism module consists of two small modules. Each module has four parts: two LN (Layer Norm) layers, an MSA module, and an MLP module.

[0133] The LN layer performs normalization on the input matrix and the output matrix It can be expressed by Equation 4, where each value in the matrix is ​​subtracted from the matrix mean and divided by the matrix standard deviation.

[0134] The MSA module is divided into a window-based W-MSA module and a sliding window-based SW-MSA module. Figure 12 As shown in Figure 1, during initialization, the image is divided into multiple modules, and each module uses a separate self-attention mechanism. As the network deepens, small modules are spliced ​​together to form larger modules, and the downsampling rate also increases with the network length.

[0135] SW-MSA module. When using W-MSA, the attention mechanism cannot be used between windows, so that the two separated windows can be connected to each other, such as Figure 13 In b, the image is divided into 9 windows. To ensure that each window has the same size, the windows are arranged in Figure 13 Move c and d in the middle, splice windows 4, 6, 2, 8, 1, 3, 7, 9, and output the same Figure 13 The size of a in the figure remains consistent, and information interaction between different windows is achieved.

[0136] The self-attention mechanism can be expressed as follows:

[0137]

[0138] in For matching items, For key items, is the information extracted from the input by the self-attention mechanism. Refers to the number of elements in K. The function will The calculated matrix is ​​normalized so that the sum of its elements is equal to 1, and finally multiplied by the information V extracted from the input by the self-attention mechanism.

[0139] MLP module. MLP module is also called multi-layer perceptron, such as Figure 14 As shown:

[0140] The MLP module consists of an input layer, a hidden layer, and an output layer, and the layers are fully connected, for example Figure 14 Chinese elements It can be derived from the following formula:

[0141]

[0142] in 、 、 、 For input values, through nonlinear function The hidden layer obtained by adding the processed , similarly, the remaining elements can be obtained from the above formula.

[0143] As the IR-ST network was built, we collected training data for seven human actions: sit-up, jump, sit-still, step, squat, bend, and fall. GPUs were used to store the training data and perform training processing on high-performance computing units (HPCs).

[0144] The present invention uses three indicators, accuracy, recall, and true negative rate, to evaluate the training effect of the network. Accuracy refers to the ratio of correctly predicted actions to other actions predicted as the same action in the prediction situation. Recall refers to the ratio of correctly classified actions in the real situation. True negative rate refers to the ratio of images that are judged as not belonging to a certain action in the real situation. The calculation formula of the above indicators is shown as follows:

[0145]

[0146]

[0147] like Figure 15 As shown in the figure, TP means that both the true situation and the predicted situation are positive examples; FP means that the true situation is a negative example and the predicted situation is a positive example; FN means that the true situation is a positive example and the predicted situation is a negative example; TN means that both the true situation and the predicted situation are negative examples.

[0148] like Figure 16The output shows the probability of seven situations such as falling. Among them, the accuracy, recall rate and true negative rate of "standing still" and "bending over" all reach 100%; the recall rate of "squatting" and "sitting-up" respectively reaches 98.1% and 98.6%, and the accuracy and true negative rate are both 100%; the accuracy, recall rate and true negative rate of the "jumping" action are relatively poor, which are 98.2%, 96.4% and 99.8%; the accuracy, recall rate and true negative rate of the "stepping" action are 98.8%, 100% and 99.7%; the accuracy, recall rate and true negative rate of the "falling" action are 97.6%, 100% and 99.5% respectively. Through the above results, it can be concluded that the IR-ST neural network proposed in the present invention has achieved good results in recognition effect.

[0149] In order to verify the superiority of IR-ST neural network in classifying human fall motion using radar raw signals, this paper compares 9 different classification methods: LeNet, AlexNet, VGG16, MobiliNetV2, MobiliNetV3-s, MobiliNetV3-1, VIT-B / 16, Swin-T and IR-ST neural network. Figure 17 As can be seen, the IR-ST neural network approach designed in this paper is only 0.2% less accurate than the Swin-T network, while reducing the model size to 1 / 10th of the original. Furthermore, compared with the lightweight MobiliNetV2 and MobiliNetV3-s networks, the accuracy improves by approximately 2% and 10%, respectively, while slightly increasing the model size. The IR-ST neural network is a well-suited network for human fall motion classification, balancing accuracy and lightweightness.

Claims

1. A human fall recognition method based on radar range Doppler map and IR-ST, characterized in that: Here are the steps: Step 1: Use a 60GHz linear frequency modulated continuous wave radar to collect radar raw data for seven actions: sit-up, jump, sit still, step, squat, bend over, and fall; Step 2: Use the mean velocity projection algorithm to extract the range Doppler map from the raw data, reduce background noise, and extract motion information; Step 3: Design an IR-ST model that combines the inverted residual convolution structure with the window self-attention mechanism and uses the range Doppler map to classify human motions; In step 1, seven actions are designed: sit-up, jump, sit still, step, squat, bend over, and fall. Multiple sets of raw data are collected for each action. The radar generates a linear frequency modulation signal through a synthesizer and transmits it through the transmitting antenna TX. The frequency of the signal will increase from 60GHz to 64GHz within a certain period. This period is called a chirp cycle. The frequency change within the chirp cycle is Expressed as: Where, is the starting frequency of the radar, is the slope of the frequency increase within a chirp period, which is obtained from the following formula in is the bandwidth within the radar signal chirp period, is the duration of a chirp cycle, measured by cosine After function processing, it is transmitted by the transmitting antenna TX. When the transmitted signal is reflected by the environment and the target, it is received by the receiving antenna RX. The received signal frequency Expressed as: Where, is the time delay, which can be expressed as follows: Where, is the distance between the radar and the object, The speed of light is achieved by mixing the transmitting signal and the receiving signal through a mixer to output an intermediate frequency signal, which converts two sinusoidal signals into and Input into the mixer, intermediate frequency signal is the difference between the instantaneous frequencies of the two input sinusoidal signals. The intermediate frequency signal is expressed as follows: The intermediate frequency signal obtained through several chirp cycles constitutes a sampling cycle, and several sampling cycles constitute a set of original data; The mean velocity projection algorithm used in step 2 extracts the range Doppler map from the original radar signal. The mean velocity projection algorithm consists of two parts: velocity projection algorithm and mean filtering. The velocity projection algorithm is divided into two parts: range Fourier transform and Doppler Fourier transform. The ordinate is composed of a sampling period. Since each sweep period is very short, the velocity information in each chirp period is ignored. The range Fourier transform refers to extracting the abscissa frequency corresponding to the spectrum peak of each chirp period on the ordinate, that is, obtaining the distance information of the target. The distance calculation formula is as follows: in is the speed of light, is the frequency sweep period, is the sweep bandwidth, is the frequency of the radar beat signal in motion or still conditions, The horizontal axis of the range-Doppler map is composed of multiple sampling periods, which is achieved by stacking multiple frames of data for multiple chirp periods; Doppler Fourier transform refers to obtaining the frequency of the data on the horizontal axis, that is, the Doppler frequency. The velocity calculation formula is as follows: in is the Doppler frequency, is the center frequency of the chirp periodic signal, and the velocity projection algorithm is used to rearrange the radar original signal to obtain the range Doppler matrix; The mean filtering method is expressed as follows: Where x is any value in the Doppler matrix, X is the range Doppler matrix, and the average value of the range Doppler function is obtained by the Average function. The function makes a judgment. If x is less than the matrix average, x is set to 0, otherwise x=x; Convert the values ​​in the range-Doppler matrix into different colors to obtain a range-Doppler map. Use the range-Doppler map to construct a radar fall detection dataset. The IR-ST network construction steps in step 3 are as follows: Step 1: The radar range-Doppler map is passed through five inverse residual convolution modules, where a shortcut branch is used to add the output of the second module to the third output. The shortcut branch means that when the input size and number of channels of the neural network module are the same, the input matrix and the output matrix of the model are added together to obtain a new matrix as the input of the next module. Step 2: The extracted feature matrix is ​​input into the Patch Partion layer. The Patch Partion layer reduces the matrix size and increases the depth through the convolution kernel. Step 3: Through the Linear Embedded layer, the Linear Embedded layer increases the dimension of the input matrix through the convolution kernel to increase the number of channels; Step 4: Slide the self-attention mechanism module twice; Step 5: Pass the output through the Patch Merging layer, which reduces the input size by half and doubles the number of channels; Step 6: Through the Adaptive Pooling layer, the Adaptive Pooling layer uses the convolution kernel to convert the input matrix size into a 1×1 matrix; Step 7: Output after processing through the fully connected layer. The fully connected layer: merges multiple convolutional layers into a fully connected layer. The goal of the fully connected layer is to flatten the high-level features learned by the convolutional layer.

2. The human fall recognition method based on radar range Doppler map and IR-ST according to claim 1 is characterized in that: The inverted residual convolution module: first, the input matrix is ​​increased in dimension by a point-by-point convolution layer, that is, the number of channels is increased, then the feature extraction is performed by a depth-wise convolution layer, and finally the dimension is reduced by a point-by-point convolution layer, and the number of channels is reduced; Convolution layer: First, perform a convolution operation on each channel, then add the H convolution feature matrices obtained by processing to form a channel, and finally repeat the above operation H+1 times to convert the original H channel image into H+1 channel. Depthwise convolution: A convolution kernel in this convolution layer is responsible for only one channel. After the output of this layer, the number of channels of the image remains unchanged. Point-by-point convolution: Use a 1×1 convolution kernel to extract and add features from each channel of the input matrix, and control the increase or decrease of the number of channels in the output matrix by changing the number of convolution kernels.

3. The human fall recognition method based on radar range Doppler map and IR-ST according to claim 1 is characterized in that: The sliding self-attention mechanism module consists of two small modules, each of which has four parts: two LN layers, MSA module, and MLP module. The LN layer performs a normalization operation on the input matrix. Each value in the matrix is ​​subtracted from the matrix mean and divided by the matrix standard deviation. The MSA module is divided into a window-based W-MSA module and a sliding window-based SW-MSA module. During initialization, the W-MSA module divides the image into multiple modules, each of which uses a self-attention mechanism. As the network deepens, small modules are spliced ​​together to form larger modules, and the downsampling rate also increases with the network length. The self-attention mechanism is expressed as follows: in For matching items, For key items, is the information extracted from the input by the self-attention mechanism, refers to The number of elements in The function will The calculated matrix is ​​normalized so that the sum of its elements is equal to 1, and finally multiplied by the information extracted from the input by the self-attention mechanism ; MLP module, also known as multi-layer perceptron, is composed of input layer, hidden layer, and output layer, and the layers are fully connected.

Images