A human posture recognition method based on FMCW radar

Abstract

This invention provides a human posture recognition method based on FMCW radar. The method collects raw human posture data and environmental data using FMCW radar. A data preprocessing algorithm is then used to process the human posture data and environmental data to initially eliminate some interference. The preprocessed human posture data is then compared with the equalized environmental data to further eliminate interference from the environment, adapting to human posture recognition tasks in different environments. Finally, the compared data and its corresponding labels are fed into a neural network for training, ultimately enabling the recognition of unlabeled human postures. This method has the advantages of high accuracy and applicability to various application scenarios.

Description

Technical Field

[0001] This invention relates to the field of wireless sensing signal processing technology, and in particular to a human posture recognition method based on FMCW radar. Background Technology

[0002] In recent years, human posture recognition has played a significant role and has broad application prospects in fields such as security monitoring, human-computer interaction, safe driving, healthcare, and sports and fitness. Against the backdrop of my country's continuously growing population and deepening aging, the safety of elderly people living alone frequently prompts researchers to consider this issue. When elderly people living alone fall at home, they often do not receive timely medical attention because they are not detected in time. Assistive intelligent monitoring technology can improve this situation. At the same time, people's demand for intelligent services in production and daily life scenarios is constantly increasing. Most existing intelligent service technologies rely on voice control or manual commands sent through intelligent control devices. Posture recognition technology, however, requires no manual commands and can automatically detect the behavior of human targets. Based on various behaviors of the human body in different positions, it can then control intelligent service products to perform related tasks.

[0003] Currently, most research on human posture recognition technology is based on computer vision-based camera sensor solutions or wearable device solutions. However, both of these solutions have limitations. Computer vision solutions operate on an optical principle, and their performance is significantly affected in low-light conditions, such as at night when no lights are on. Furthermore, people are hesitant to accept computer vision-based solutions in more private settings like bedrooms and bathrooms due to concerns about privacy. Wearing devices is inconvenient in these situations, and falls are more likely, posing a threat to safety. Therefore, wearable device-based sensor technology is not suitable for safety monitoring in these scenarios.

[0004] FMCW radar, due to its operating principle of transmitting linear frequency modulated continuous wave signals, is unaffected by light conditions. Furthermore, its low transmission power and certain penetration capabilities allow it to be installed behind obstructions without creating the illusion of surveillance or compromising the privacy of those being monitored. Based on these advantages, FMCW radar has attracted increasing attention from researchers in the field of wireless sensing technology. In 2017, researchers at MIT developed a wall-mounted WiGait device capable of penetrating walls, primarily detecting the stride length and movement speed of moving individuals, and detecting falls. This device demonstrated the broad prospects of wireless devices in home surveillance. In 2018, Adeel Ahmad et al. used a MIMO-based linear frequency modulated continuous wave radar to detect multiple targets. In their experiments, researchers focused human echoes using range gate selection and beamforming techniques, effectively suppressing clutter. Experimental results showed that this method can effectively improve the localization performance of obstructed targets such as building ruins. Currently, existing human posture recognition technologies based on FMCW radar do not fully consider the impact of different environments on human posture recognition, resulting in low recognition accuracy. Therefore, research on methods that can accurately recognize human posture in various environments is necessary and has significant practical application value. Summary of the Invention

[0005] Therefore, the purpose of this invention is to address the problem that the accuracy of existing human posture recognition technologies based on FMCW radar is affected by different environments, and to provide a human posture recognition method based on FMCW radar. This method involves collecting human posture data and environmental data in the test space, performing data preprocessing, then canceling the equalized environmental data and human posture data, and finally feeding the cancellation result and its corresponding label into a neural network for training and learning, ultimately achieving the recognition of unlabeled human postures.

[0006] To achieve the above-mentioned objectives, this invention provides a human posture recognition method based on FMCW radar, the method comprising the following steps:

[0007] Step 1: For the space environment to be tested, collect raw data of human posture and environment using FMCW radar.

[0008] Step 2: Perform data preprocessing on the collected data;

[0009] Step 3: Perform time window equalization on the targetless environment data obtained through the above steps to obtain the equalized targetless environment data, which is denoted as the targetless environment equalization data.

[0010] Step 4: Perform data cancellation between the human posture data obtained in Step 2 and the targetless environment equalization data obtained in Step 3 to obtain human posture micro-Doppler spectrum feature data without environmental interference.

[0011] Step 5: Input the human body posture micro-Doppler spectrum feature data and its corresponding labels into the neural network for learning, training and posture recognition.

[0012] Furthermore, the human posture data and environmental data collected in step 1 are beat intermediate frequency (IF) signals that have undergone frequency mixing. The expression for the beat IF signal obtained after mixing the echo signal with the transmitted signal and passing it through a low-pass filter is as follows:

[0013] s if (t)=Acos2π[f c τ+(μτ-Δf d )t]

[0014] Where A is the proportionality coefficient, f c Let τ be the center frequency of the carrier wave, τ be the time delay between the radar transmitted signal and its echo signal, μ be the sweep slope of the linear frequency modulated continuous wave, and Δf be the center frequency of the carrier wave. d Let t be the Doppler frequency shift of the signal, and t be the signal transmission time.

[0015] Furthermore, the raw data in step 1 is the raw digital intermediate frequency signal data obtained after the beat intermediate frequency signal is sampled by an analog-to-digital converter (ADC).

[0016] Furthermore, step 2 specifically includes the following steps:

[0017] Step 2.1: Rearrange the data collected in Step 1 into the desired data format. Specifically, the data format is a three-dimensional data block consisting of slow time, fast time, and antenna. The FMCW radar used has multiple transmitting and receiving antennas. The number of transmitting antennas is m, and the number of receiving antennas is n. After rearranging the data, the received data of m*n antennas can be obtained.

[0018] Step 2.2: Perform antenna data equalization on the rearranged data to eliminate interference caused by the antenna end and obtain equalized received antenna data. The specific form of the equalized antenna data is a two-dimensional data matrix of slow time and fast time.

[0019] Step 2.3: Perform FFT on the equalized data in its distance dimension to obtain the target's distance information;

[0020] Step 2.4: Use MTI technology to process the signal, eliminate interference from static targets, and then perform spectral peak search on the two-dimensional data matrix to determine the range cell where the target is located;

[0021] Step 2.5: Based on the target's location and the time of the action, perform time window and distance window segmentation on the data. The size of the distance window is determined according to the distance unit where the target is located in Step 2.4. Within the time window of the action, distance units that exceed the target's location are segmented. The size of the time window is related to the start and end times of the action. By detecting the start and end times of the target's movement, the size of the time window is adaptively adjusted to perform time window segmentation on the data.

[0022] Step 2.6: Perform joint time-frequency analysis on the time dimension of the data to obtain the micro-Doppler feature data of the human target.

[0023] Furthermore, in step 3, the size of the time window in the targetless environment data is adaptively adjusted according to the time length after the short-time Fourier transform in step 2.6. The data in the targetless environment is cropped according to the size of the time window, and then each cropped data segment is balanced, and finally the balanced environment data is output.

[0024] Furthermore, in step 4, data cancellation involves using human posture data and targetless environment equalization data as two input signals. The human posture data is then processed according to the time window size in the targetless environment data equalization to obtain a micro-Doppler spectrum feature map of human posture without environmental interference.

[0025] Furthermore, step 5 specifically includes the following steps:

[0026] Step 5.1: Input the human posture micro-Doppler spectrum feature data without environmental interference obtained in Step 4 and its corresponding labels into the neural network for training and learning. The neural network is a residual neural network, and the detection result is multi-classified, including at least three categories of labels: walking or standing up, sitting down, and falling down.

[0027] Step 5.2: Identify unlabeled human posture data.

[0028] Furthermore, the unlabeled human posture data is identified by using a trained neural network to identify and judge the human posture data, providing corresponding posture recognition results, and sending corresponding instructions to the corresponding smart devices for processing subsequent tasks.

[0029] Furthermore, the processing of subsequent tasks includes at least:

[0030] If the posture recognition result is walking or standing up, turn on the corresponding lighting equipment near the walking or standing position;

[0031] If the posture recognition result is "sitting down", activate the corresponding smart device near the sitting position;

[0032] If the posture recognition result indicates a fall, an alarm will be triggered immediately.

[0033] Furthermore, the location is determined based on the location where the radar is installed.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] This invention provides a human posture recognition method based on FMCW radar. Before acquiring the human target posture, target-free environmental data is collected. Then, a data preprocessing algorithm processes the target posture data and environmental data separately to initially eliminate some interference. Next, the preprocessed human posture data and the equalized environmental data are canceled to further eliminate interference caused by the environment, thus adapting to human posture recognition tasks in different environments. Finally, the canceled data and its corresponding labels are fed into a neural network for training and learning, ultimately completing the recognition of unlabeled human postures with high accuracy and applicability to various application scenarios. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the overall process of a human posture recognition method based on FMCW radar provided in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the data preprocessing process provided in an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of human posture provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the FMCW radar data format provided in an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of FMCW radar transmitting chirp signals provided in an embodiment of the present invention.

[0042] Figure 6 This is a sample grayscale feature map of human posture data provided in the embodiments of the present invention. Detailed Implementation

[0043] The principles and features of the present invention are described below with reference to the accompanying drawings. The listed embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0044] Reference Figure 1 and Figure 2 This embodiment provides a human posture recognition method based on FMCW radar, the method including the following steps:

[0045] Step 1: For the test environment, collect raw data of human posture and environment using FMCW radar.

[0046] Step 2: Perform data preprocessing on the collected data.

[0047] Step 3: Perform time window equalization on the targetless environment data obtained through the above steps to obtain the equalized targetless environment data, which is denoted as the targetless environment equalization data.

[0048] Step 4: Perform data cancellation between the human posture data obtained in Step 2 and the targetless environment equalization data obtained in Step 3 to obtain human posture micro-Doppler spectrum feature data without environmental interference.

[0049] Step 5: Input the human body posture micro-Doppler spectrum feature data and its corresponding labels into the neural network for learning, training and posture recognition.

[0050] In this embodiment, the radar configuration parameters for collecting human posture data and environmental data in step 1 are consistent, and the collected human posture data and environmental data are beat intermediate frequency signals that have undergone frequency mixing processing. The human posture data uses five posture actions as examples: walking, standing still, standing up, sitting down, and falling. Schematic diagrams of several human postures for these five actions in this invention are shown below. Figure 3 As shown. Frequency-modulated continuous wave (FMCW) radar was used to collect human posture data and environmental data respectively. The beat intermediate frequency signal is obtained by mixing the echo signal and the transmitted signal and then passing it through a bandpass filter; its expression is:

[0051] s if (t)=Acos2π[f c τ+(μτ-Δf d )t]

[0052] Where A is the proportionality coefficient, f c Let τ be the center frequency of the carrier wave, τ be the time delay between the radar transmitted signal and its echo signal, μ be the sweep slope of the linear frequency modulated continuous wave, and Δf be the center frequency of the carrier wave. dLet t be the Doppler frequency shift of the signal, and t be the signal transmission time. The raw data is the original digital intermediate frequency signal data obtained after sampling the beat intermediate frequency signal by an analog-to-digital converter (ADC); subsequent data processing algorithms are all based on the raw data for in-depth processing.

[0053] Step 2 specifically includes the following steps:

[0054] Step 2.1: Rearrange the data collected in Step 1 into the desired data format. Specifically, the data format is a three-dimensional data block consisting of slow time, fast time, and antenna. The FMCW radar used for sampling has multiple transmitting and receiving antennas, with m transmitting antennas and n receiving antennas. After rearranging the data, m*n sets of received data from the antennas can be obtained. A schematic diagram of the rearranged radar data format in this embodiment is shown below. Figure 4 As shown, each column of the 3D data block represents a fast time dimension, and the number of cells in each column represents a data sampling point. Under the same radar configuration parameters, the more sampling points, the farther the radar can detect. Different columns represent different chirp numbers, and each frame of data consists of multiple chirp data, while the data in each page consists of multiple frames of data. Each page represents a set of data corresponding to a single transmit antenna and a single receive antenna, and different page numbers represent data received by different receive and transmit antennas. Figure 4 The data format diagram in the diagram is only for better explanation and intuitive display of the data arrangement, and does not represent the actual number of samples, chirps, frames, and antenna dimensions.

[0055] Step 2.2: Perform antenna data equalization on the rearranged data to eliminate interference caused by the antenna ends, obtaining equalized received antenna data. The specific form of the equalized antenna data is a two-dimensional data matrix of slow time versus fast time. A one-to-one transmit and receive antenna may be subject to external interference at some moment during transmission and reception, thus affecting the data. However, the possibility of interference to multiple transmit and receive antennas at the same time is extremely small. Therefore, this invention uses a group composed of different transmit and receive antennas as a unit of the antenna dimension, corresponding to... Figure 4 Different pages of the three-dimensional data block are used. Data from different antenna dimensions are averaged at their corresponding positions, ultimately outputting a two-dimensional data matrix containing only slow-time intervals and fast-time intervals. This effectively reduces interference caused by antenna transmission and reception.

[0056] Step 2.3: Perform an FFT on the equalized data along its range dimension to obtain the target's range information. A schematic diagram of the chirp signal emitted by the FMCW radar in this invention is shown below. Figure 5 As shown. The received raw data is the filtered and sampled intermediate frequency signal f. bDifferent intermediate frequency (IF) signals correspond to different propagation delays τ, while the electromagnetic waves emitted by the radar propagate in space at the speed of light c. Therefore, the target distance at different locations corresponds to different IF signal frequencies. This embodiment calculates the target distance information d by calculating the IF signal frequency. Furthermore, FFT can calculate the IF signal f caused by the target from the data. b The formula for calculating distance is:

[0057]

[0058] in μ is the frequency modulation slope of the radar transmitted signal, B is the bandwidth of the radar transmitted signal, T is the duration of one frame of FMCW radar data, and f is the frequency modulation slope of the radar transmitted signal. b For a target, the intermediate frequency signal is given, where c is the speed of light and d is the target distance information to be determined.

[0059] Step 2.4: Process the signal using MTI technology to eliminate interference from static targets, and then perform spectral peak search on the two-dimensional data matrix to determine the range cell where the target is located. MTI technology refers to Moving Target Indication. Because the electromagnetic waves emitted by the radar propagate at the speed of light, and the radar is stationary in the space environment, the range information obtained from an object stationary relative to the radar will have peak-to-peak values ​​at the same position in the range cell corresponding to different chirps. At the same time, the moving target is moving relative to the radar, and the peak-to-peak position of the range cell will change between different chirps. To address this, this invention uses MTI technology to subtract adjacent chirp signals, subtract the corresponding position of the second column data in the slow-time-fast-time two-dimensional matrix from the corresponding position of the first column data to obtain the first column output of the MTI data; subtract the corresponding position of the third column data from the corresponding position of the second column data to obtain the second column output of the MTI data; subsequent MTI operations are similar, and finally, a two-dimensional data matrix processed by MTI is output.

[0060] Step 2.5: Based on the target's location and the time of the action, the data is segmented into time and distance windows. The size of the distance window is determined according to the distance unit of the target in Step 2.4. Within the time window of the action, distance units exceeding the target's position are segmented. The size of the time window is related to the start and end times of the action. By detecting the start and end times of the target's movement, the size of the time window is adaptively adjusted, thus performing time window segmentation on the data. After MTI processing, the peak-to-peak value of the data matrix is ​​detected to understand the change in the target's position over time. The sizes of the distance and time windows can be adaptively adjusted according to the target's position interval, thereby achieving the effect of filtering out interference caused by signals without a target location or time interval.

[0061] Step 2.6: Perform joint time-frequency analysis on the time dimension of the data to obtain the micro-Doppler feature data of the human target. In this embodiment, a short-time Fourier transform (STFT) is performed on the slow time dimension of the data to obtain the micro-Doppler data features of the target.

[0062] In this embodiment, the size of the time window in the targetless environment data in step 3 corresponds to the time length after STFT in step 2.6. The data in the targetless environment is truncated according to the size of the time window. The formats of both data segments are adjusted to the same size, and then the average of each truncated data segment is calculated to output the final balanced environment data.

[0063] In step 4, data cancellation involves taking human posture data and targetless environment equalization data as two input signals, and performing data cancellation processing on the human posture data according to the time window size in the targetless environment data equalization to obtain the human posture micro-Doppler spectrum feature map without environmental interference.

[0064] Step 5 specifically includes the following steps:

[0065] Step 5.1: The human posture micro-Doppler spectrum feature data and their corresponding labels obtained in Step 4 without environmental interference are fed into the neural network for training. In this embodiment, the neural network used is a residual neural network (ResNet-50), which adds connections between different layers on top of a convolutional neural network, avoiding problems such as gradient explosion and gradient vanishing, and is able to extract deep information from the input data. This embodiment collects data information on five postures: walking, standing still, standing up, sitting down, and falling. The grayscale feature images of the human posture data samples in this embodiment are shown below. Figure 6 As shown.

[0066] Step 5.2: Identify unlabeled human posture data. After training the residual network, batches of unlabeled data are fed into the trained network for identification and judgment, and the network provides posture recognition results and sends corresponding instructions to smart devices, such as air conditioners, lights, televisions, and curtains. When a device receives the corresponding instruction, it will complete the pre-set function. For example, if a fall is detected, an alarm will be triggered immediately, and a designated contact person will be notified to handle the situation promptly; if the posture recognition result is walking or standing, the corresponding light will be turned on near the walking or standing location; if the posture recognition result is sitting, the corresponding smart device will be turned on near the location. The radar devices installed in different locations will be associated with the corresponding smart products in their respective functional areas to complete the tasks following radar posture recognition.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human posture recognition method based on FMCW radar, characterized by, The method includes: Step 1: For the space environment to be tested, collect raw data of human posture and environment using FMCW radar. Step 2: Perform data preprocessing on the collected data, specifically including the following steps: Step 2.1: Rearrange the data collected in Step 1 into the desired data format. Specifically, the data format is a three-dimensional data block consisting of slow time, fast time, and antenna. The FMCW radar used has multiple transmitting and receiving antennas. The number of transmitting antennas is m, and the number of receiving antennas is n. After rearranging the data, the received data of m*n antennas can be obtained. Step 2.2: Perform antenna data equalization on the rearranged data to eliminate interference caused by the antenna end and obtain equalized received antenna data. The specific form of the equalized antenna data is a two-dimensional data matrix of slow time and fast time. Step 2.3: Perform FFT on the equalized data in its distance dimension to obtain the target's distance information; Step 2.4: Use MTI technology to process the signal, eliminate interference from static targets, and then perform spectral peak search on the two-dimensional data matrix to determine the range cell where the target is located; Step 2.5: Based on the target's location and the time of the action, perform time window and distance window segmentation on the data. The size of the distance window is determined according to the distance unit where the target is located in Step 2.

4. Within the time window of the action, distance units that exceed the target's location are segmented. The size of the time window is related to the start and end times of the action. By detecting the start and end times of the target's movement, the size of the time window is adaptively adjusted to perform time window segmentation on the data. Step 2.6: Perform joint time-frequency analysis on the time dimension of the data to obtain the micro-Doppler feature data of the human target. Specifically, perform short-time Fourier transform (STFT) on the slow time dimension of the data to obtain the micro-Doppler data features of the target. Step 3: Perform time window equalization on the targetless environment data obtained through the above steps to obtain the equalized targetless environment data, which is denoted as the targetless environment equalization data. The size of the time window in the targetless environment data is adaptively adjusted according to the time length after the short-time Fourier transform in step 2.

6. According to the size of the time window, the data in the targetless environment is truncated, and then each segment of the truncated data is equalized, and finally the equalized environment data is output. Step 4: Perform data cancellation on the human posture data obtained in Step 2 and the targetless environment equalization data obtained in Step 3 to obtain human posture micro-Doppler spectrum feature data without environmental interference. Data cancellation is to use human posture data and targetless environment equalization data as two input signals, and perform data cancellation on the human posture data according to the time window size in the targetless environment data equalization to obtain human posture micro-Doppler spectrum feature map without environmental interference. Step 5: Input the human body posture micro-Doppler spectrum feature data and its corresponding labels into the neural network for learning, training and posture recognition. 2.The human posture recognition method based on FMCW radar according to claim 1, characterized in that, The human posture data and environmental data collected in step 1 are beat intermediate frequency (IF) signals that have undergone frequency mixing. The expression for the beat IF signal obtained after mixing the echo signal with the transmitted signal and passing it through a low-pass filter is as follows: Where A is the proportionality coefficient. The center frequency of the carrier wave. The time delay between the radar's transmitted signal and its echo signal. The sweep slope of the linear frequency modulated continuous wave. Doppler frequency shift of the signal t It refers to the signal transmission time.

3. The human posture recognition method based on FMCW radar according to claim 1, characterized in that, The raw data in step 1 is the raw digital intermediate frequency signal data obtained after the beat intermediate frequency signal is sampled by an analog-to-digital converter (ADC).

4. The human posture recognition method based on FMCW radar according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5.1: Input the human posture micro-Doppler spectrum feature data without environmental interference obtained in Step 4 and its corresponding labels into the neural network for training and learning. The neural network is a residual neural network, and the detection result is multi-classified, including at least three categories of labels: walking or standing up, sitting down, and falling down. Step 5.2: Identify unlabeled human posture data.

5. A human posture recognition method based on FMCW radar according to claim 4, characterized in that, The process involves identifying unlabeled human posture data, specifically by using a trained neural network to identify and judge the human posture data, providing corresponding posture recognition results, and sending corresponding instructions to the corresponding smart devices for processing of subsequent tasks.

6. The human posture recognition method based on FMCW radar according to claim 5, characterized in that, The processing of subsequent tasks includes at least: If the posture recognition result is walking or standing up, turn on the corresponding lighting equipment near the walking or standing position; If the posture recognition result is "sitting down", activate the corresponding smart device near the sitting position; If the posture recognition result indicates a fall, an alarm will be triggered immediately.

7. A human posture recognition method based on FMCW radar according to claim 6, characterized in that, The location is determined based on where the radar is installed.

Images