Radar Continuous Action Detection and Behavior Recognition Method for Smart Elderly Care Applications

Abstract

The present invention discloses a radar continuous action detection and behavior recognition method for intelligent elderly care applications, which comprises the following steps: 1) For the echo signals of common continuous behaviors of solitary elderly people collected by the radar, a micro-Doppler matrix is obtained through processing such as signal mixing, analog-to-digital conversion, matrix reorganization, static elimination, range-FFT, and Doppler-FFT; 2) A mapping factor is used to map the elements with relatively high amplitudes in the micro-Doppler matrix to visualize the micro-Doppler features; 3) A behavior detection model improved based on a visual target detection model is used to detect the obtained micro-Doppler map containing continuous action information, so as to obtain the types and start and end times of each action, and the human behaviors in the intelligent elderly care application scenarios are recognized according to the correlation between actions. This method can improve the imaging quality of the micro-Doppler map, effectively make up for the deficiencies in action segmentation, feature extraction, and behavior classification in previous recognition, and has more general applicability.

Description

Technical Field

[0001] The present invention relates to the technical fields of Internet of Things applications and intelligent elderly care technologies, and particularly to a radar continuous action detection and behavior recognition method for intelligent elderly care applications. Background Art

[0002] The degree of population aging in China is increasing day by day, and the issue of elderly care has become a major problem faced in the process of China's economic and social development. In this regard, the development of new intelligent and healthy elderly care technologies has become an inevitable trend for building an aging-friendly society.

[0003] As an important supporting technology in intelligent elderly care, action detection technology mainly relies on "wearable" and "non-wearable" devices. Among them, "wearable" devices require the monitored object to carry a dedicated sensor, lacking comfort and convenience, and are easily forgotten or rejected by the elderly. Therefore, "non-wearable" devices have a broader application scenario, which mainly includes visual sensors (such as cameras, infrared imaging), wireless sensors (such as WiFi), and radio frequency sensors (such as radars). Among them, infrared sensors are easily affected by environmental temperature, resulting in a decline in recognition performance and cannot be widely applied; video sensors are not applicable to application scenarios outside the line-of-sight range and in poor lighting conditions, and at the same time, this sensor is prone to leaking personal privacy and cannot be deployed in bedroom and bathroom environments, while for the elderly living alone, the bedroom and bathroom are the most needed scenarios for detection. To solve the above problems, behavior perception technologies based on WiFi and millimeter-wave radar have received attention from the academic and industrial communities. WiFi and millimeter-wave radar belong to passive perception technologies, with the characteristics of non-contact, not affected by light intensity, and can penetrate walls and some obstacles. However, the former has significant multipath effects in complex indoor environments, resulting in poor recognition effects. In contrast, the latter has higher recognition sensitivity, stronger robustness, and better privacy protection. Therefore, millimeter-wave radar has great functional advantages and good development prospects in monitoring human behavior. Using millimeter-wave radar to achieve behavior detection mainly includes two major technologies based on point cloud and micro-Doppler features. Among them, the behavior recognition technology based on point cloud can not only perform indoor positioning and tracking but also achieve behavior recognition through the captured point cloud information. However, in practical applications, this method has many challenges. For example, in a complex indoor environment, radar signals will be affected by multiple reflections of non-human targets such as furniture and walls, resulting in multipath effects. In this regard, background elimination methods need to be adopted to remove interference, but this may also lead to the loss of point cloud information.

[0004] Existing methods for identifying human actions using radar micro-Doppler features mainly use the method of directly mapping the micro-Doppler matrix to obtain an image-based micro-Doppler map. However, this method not only easily causes differences in the background colors of the obtained images for different samples, but also obvious noise exists in all regions of the images. The above defects will seriously affect the final performance of continuous action detection.

[0005] In addition, current behavior recognition technologies based on micro-Doppler features mainly focus on single behaviors, and there is relatively little research on continuous behavior recognition, mainly due to the following three difficulties in continuous behavior recognition:

[0006] (1) The action sequences in continuous behaviors are variable, so that current behavior detection technologies cannot be widely applied;

[0007] (2) The duration of each behavior in a continuous behavior sequence is uncertain, so that continuous behaviors are difficult to segment;

[0008] (3) It is difficult to fully extract the spatio-temporal features of continuous behavior spectrograms, so that the effects of current behavior detection technologies are not good. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide a radar continuous action detection and behavior recognition method for smart elderly care applications, which improves the imaging quality of the micro-Doppler map, effectively makes up for the deficiencies in action segmentation, feature extraction and behavior classification in previous recognition, and has more general applicability.

[0010] To achieve the above purpose, the present invention adopts the following technical solutions: A radar continuous action detection and behavior recognition method for smart elderly care applications, including the following steps:

[0011] Step 1: Use radar to collect human continuous behavior data and perform data preprocessing;

[0012] Step 2: Selectively image the micro-Doppler features using a dynamic feature visualization method;

[0013] Step 3: Use a behavior detection model improved based on a visual object detection model to detect the obtained micro-Doppler map containing continuous action information, so as to obtain the types and start and end times of each action, and identify the human behaviors in the smart elderly care application scenario according to the correlation between actions.

[0014] In a preferred embodiment, step 1 is specifically as follows: First, perform analog-to-digital conversion on the obtained original signal to obtain a digital signal; secondly, perform matrix reorganization on the digital signal and arrange it in the order of virtual receiving antennas; then, perform static cancellation on the reorganized three-dimensional matrix to eliminate the influence of background noise; finally, obtain the micro-Doppler matrix corresponding to the continuous behavior through range-FFT and Doppler-FFT.

[0015] In a preferred embodiment, collecting human continuous behavior data specifically means using a radar board to collect the echo signals of human continuous behaviors, collecting multiple continuous behavior sequences including various actions, where each behavior includes one or more atomic actions; the collected continuous behavior sequences include the following sequences:

[0016] (1) Jump - Walk - Drink water - Sit down - Stand up - Fall;

[0017] (2) Sit down - Stand up - Drink water - Walk - Jump - Fall;

[0018] (3) Walk - Sit down - Stand up - Drink water - Jump - Fall;

[0019] (4) Stretching exercise - Drink water - Walk - Sit down - Stand up - Fall;

[0020] (5) Sit down - Stand up - Stretching exercise - Walk - Drink water - Fall;

[0021] (6) Drink water - Walk - Stretching exercise - Sit down - Stand up - Fall;

[0022] (7) Tai chi - Jog - Stretching exercise - Cough - Fall - Wave for help;

[0023] (8) Cough - Jog - Pick up something - Stretching exercise - Fall - Wave for help;

[0024] (9) Pick up something - Jog - Jog - Stretching exercise - Fall - Wave for help;

[0025] (10) Stretching exercise - Pick up something - Jog - Cough - Sit down - Stand up.

[0026] In a preferred embodiment, step 2 is specifically as follows: First, take out the elements in the obtained micro-Doppler matrix column by column, and sort the taken-out elements in descending order of their magnitudes to obtain a column vector r'; secondly, select a mapping factor α, 0 < α < 100, retain the first α% of the elements of r' and set the remaining elements to zero; then, restore according to the positions of the elements of r' in the original micro-Doppler matrix to obtain a processed micro-Doppler matrix; finally, perform linear mapping of the color on the processed micro-Doppler matrix to obtain an image-based micro-Doppler map.

[0027] In a preferred embodiment, constructing the micro-Doppler matrix specifically involves obtaining the continuous-behavior micro-Doppler matrix through analog-to-digital conversion, matrix rearrangement, static elimination, and range-Doppler map (RDM) construction of the continuous behavior. The detailed content is described as follows:

[0028] The analog form obtained after the original signal collected by the radar passes through mixing and a low-pass filter is:

[0029]

[0030] First, the above analog signal is converted into a digital signal through analog-to-digital conversion, and then matrix rearrangement is performed;

[0031] According to the transceiver antenna order, the digital signal is rearranged into the form of a three-dimensional data matrix; First, the data corresponding to 4 receiving antennas in the aforementioned obtained two-dimensional matrix is separated; and the matrix is arranged in rows based on the original sampling points until the last chirp is completed; At this time, assuming that the nth sampling point of the mth chirp is expressed as:

[0032]

[0033] In the formula, λ represents the signal wavelength, R represents the distance between the human body and the radar, T f is the interval of ADC sampling in the fast time dimension, T s is the sampling interval in the slow time dimension, f b =(B / T c )τ; In addition, N r is the number of sampling points, and M is the total number of chirps transmitted during the behavior acquisition;

[0034] Since multiple receiving antennas are used, there is a relative distance between each receiving antenna, so each antenna will generate an additional phase shift; Here, taking 2 receiving antennas as an example, assuming the distance between the receiving antennas is d i , and the target human body is located at the position of θ angle in the front left of the radar. At this time, the phase change can be mathematically deduced as the following equation:

[0035]

[0036] Therefore, the nth sampling point of the mth chirp on the ith receiving antenna is expressed as:

[0037]

[0038] Finally, the 12 data matrices are spliced according to the order of virtual receiving antennas to obtain a three-dimensional data matrix; This three-dimensional data matrix contains the complete micro-motion information of the human target and will be used as the basis for constructing the subsequent micro-Doppler spectrogram;

[0039] When the radar system collects human behavior data, the echo signal carries interference generated by objects in the environment. First, the obtained three-dimensional matrix is processed separately according to the antenna order. Taking the i-th receiving antenna as an example, the row average value of the corresponding data matrix is calculated sequentially row by row, as shown in the formula:

[0040]

[0041] Among them, represents the value at the n-th sampling point of the m-th chirp on the i-th receiving antenna; Subsequently, each column of the original data matrix is subtracted from the average value matrix. Taking the m-th column as an example:

[0042]

[0043] Finally, the target data matrix after static elimination is obtained as Y'[n, m, i]; where n ∈ [1, N r , m ∈ [1, M], i ∈ [1, N a ;

[0044] The row and column dimensions of the target data matrix after static elimination need to be Fourier-transformed separately;

[0045] First, a Blackman window is added to the range dimension of the three-dimensional array to perform an N r point range FFT to obtain the range information of the target:

[0046]

[0047] Subsequently, a Hamming window is added to the velocity dimension to perform an N d point Doppler FFT to obtain the velocity information of the target:

[0048]

[0049] Finally, the zero-frequency component is shifted to the center of the spectrum, and the average value is taken for the antenna dimension to obtain the corresponding RDM, which is specifically expressed as:

[0050]

[0051] Project the obtained RDM onto the velocity dimension, and then sum them sequentially in the column direction according to the frame number order to obtain the micro-Doppler feature matrix containing the target motion information. Taking the q-th frame as an example, its calculation process is described as:

[0052]

[0053] In the formula, Denote the k-th sampling point of the (1+(q-1)p)-th chirp on the i-th receiving antenna; in addition, p ∈ [1, N d , q ∈ [1, M / N d .

[0054] In a preferred embodiment, step 3 is specifically as follows: calibrate a single behavior in the micro-Doppler spectrogram through a dedicated calibration tool to obtain the coordinates of the upper left corner and the lower right corner in the calibration box corresponding to each behavior, and then calculate the obtained coordinates of the upper left corner and the lower right corner into the center point coordinates and the width and height values; introduce a target detection model and add a coordinate attention mechanism CA on this basis to extract and fuse the temporal and spatial features of the continuous behavior spectrogram; improve the loss function and activation function to improve the detection performance to obtain a continuous behavior detection and recognition model; on this basis, divide the continuous behavior spectrogram dataset into a training set, a validation set, and a test set; first input the training set and the validation set into the improved target detection model; extract the features of the training set through the model, and the CA attention mechanism extracts and fuses the spatio-temporal domain features; obtain the model with the best effect through the validation set; finally, test the obtained model through the test set, so as to realize the segmentation and detection of continuous actions and identify human behaviors according to the correlation between actions

[0055] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses radar as a human continuous behavior recognition sensor, without the need to wear a dedicated sensor, improving the comfort of the elderly living alone. In addition, compared with conventional identification devices such as cameras, infrared sensors that are prone to infringe on the privacy of the elderly, and WiFi with low detection accuracy, it has better privacy protection, distance resolution, and the technical advantage of being able to penetrate walls

[0056] The behavior recognition technology based on radar mainly includes two types: point cloud and micro-Doppler features. Among them, the micro-Doppler feature has the advantage of being unaffected by multipath interference in the environment, which can improve the performance of behavior detection and recognition. Usually, human activities in real life generally exist as continuous behaviors. However, due to the uncertain behavior types and execution times, it is difficult to achieve segmentation and effective extraction of features, thus affecting the accuracy of behavior detection and recognition. Therefore, the present invention introduces target detection technology into the continuous behavior detection and recognition based on radar micro-Doppler features, and solves the deficiency of continuous behavior segmentation in the prior art by converting data processing into visual processing, thereby improving the convenience and accuracy of detection. On the basis of the original target detection model, in order to be close to the characteristics of the continuous behavior spectrogram, the present invention adds a CA attention mechanism to achieve the extraction and fusion of temporal and spatial features, further improving the accuracy of continuous behavior detection and recognition BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of the method of the present invention for a preferred embodiment of the present invention

[0058] Figure 2 Schematic diagram for constructing the spectrogram of the preferred embodiment of the present invention;

[0059] Figure 3 Partial schematic diagram of the three-dimensional target array of the preferred embodiment of the present invention;

[0060] Figure 4 Specific flowchart of the dynamic feature visualization of the preferred embodiment of the present invention;

[0061] Figure 5 Micro-Doppler spectrogram of the target walking in the preferred embodiment of the present invention;

[0062] Figure 6 Display of the ConYOLOv5s model in the preferred embodiment of the present invention;

[0063] Figure 7 Visualization display of the recognition result in the preferred embodiment of the present invention. Detailed implementation manners

[0064] The following further describes the present invention in conjunction with the accompanying drawings and embodiments.

[0065] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further descriptions of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0066] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "comprise" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0067] The present invention provides a radar continuous action detection and behavior recognition method for smart elderly care applications, which realizes the detection and recognition of common behaviors of solitary elderly people, including but not limited to walking, coughing, falling, waving for help, etc., and solves the defects of inconsistent background colors and noise in the micro-Doppler feature visualization images, the poor stability of existing single behavior recognition technologies, and the deficiencies of continuous behavior recognition systems in continuous behavior segmentation, detection, and recognition. The present invention mainly includes the following steps:

[0068] Step 1: Use a radar to collect human continuous behavior data and perform data preprocessing:

[0069] First, perform analog-to-digital conversion on the obtained original signal to obtain a digital signal; secondly, perform matrix reorganization on the digital signal and arrange it in the order of virtual receiving antennas; then, perform static cancellation on the reorganized three-dimensional matrix to eliminate the influence of background noise; finally, obtain the micro-Doppler matrix corresponding to the continuous behavior through range-FFT and Doppler-FFT.

[0070] Step 2: Selectively visualize the micro-Doppler features using the dynamic feature visualization method:

[0071] First, extract the elements in the obtained micro-Doppler matrix column by column, and sort the extracted elements in descending order of their magnitudes to obtain a column vector r'; secondly, select a mapping factor α (0 < α < 100), retain the first α% of the elements in r', and set the remaining elements to zero; then, restore according to the positions of the elements in r' in the original micro-Doppler matrix to obtain the processed micro-Doppler matrix; finally, perform linear mapping of the color on the processed micro-Doppler matrix to obtain the visualized micro-Doppler map.

[0072] Step 3: Use the ConYOLOv5s model improved based on the existing object detection model or use visual object detection models such as Faster-RCNN, YOLO series, and SSD to detect the obtained micro-Doppler map containing continuous action information, so as to obtain the types and start and end times of each action, and identify the human behaviors in the intelligent elderly care application scenario according to the correlation between actions:

[0073] Calibrate a single behavior in the micro-Doppler spectrogram through a dedicated calibration tool to obtain the coordinates of the upper left corner and the lower right corner in the calibration box corresponding to each behavior, and then calculate the obtained coordinates of the upper left corner and the lower right corner into the center point coordinates and width and height values. By introducing the YOLOv5s object detection model and adding the coordinate attention mechanism CA on this basis, the time and space features of the continuous behavior spectrogram are extracted and fused; by improving the loss function and activation function, the detection performance is improved; the continuous behavior detection and recognition model: ConYOLOv5s model is obtained through the above improvement steps. Divide the continuous behavior spectrogram dataset into a training set, a validation set, and a test set; first input the training set and the validation set into the ConYOLOv5s model; extract the features of the training set through the model, and the CA attention mechanism extracts and fuses the spatio-temporal domain features; obtain the model with the best effect through the validation set; finally, test the obtained model through the test set, thus realizing the segmentation, detection, and behavior recognition of continuous actions.

[0074] Different from the prior art, the above technical solution embeds human actions into the micro-Doppler features, uses the method of dynamic feature visualization to map the micro-Doppler matrix containing human motion information into a micro-Doppler map, and realizes continuous behavior segmentation, detection and recognition through a target detection algorithm. Among them, the use of the dynamic feature visualization method can effectively remove obvious noise points and highlight the micro-motion of the target by controlling the signal intensity range, unify the background color of the micro-Doppler map, and significantly improve the readability of the subsequent detection network for the spectrogram; the application of the target detection technology effectively solves the cumbersome process of realizing continuous behavior segmentation by sliding windows in the prior art. In addition, the technical solution can be applied to the scenario of human continuous behavior in the real situation. Most importantly, the technical solution can fully extract the spatio-temporal domain features of the continuous behavior spectrogram, and then obtain the behavior detection result of the target person.

[0075] Specifically, the radar emits electromagnetic waves through the transmitting antenna, and then receives the echo signal reflected by the target through the receiving antenna to capture the original signal containing information such as the target distance and speed. Among them, due to advantages such as not being affected by the multipath effect in the environment, the micro-Doppler effect based on radar is widely used in fields such as human perception.

[0076] Please refer to Figure 1 , this embodiment provides a radar continuous action detection and behavior recognition method for smart elderly care applications, including three steps: data acquisition, micro-Doppler matrix construction, dynamic feature visualization, and continuous behavior detection and recognition. The specific content is introduced as follows:

[0077] I. Data acquisition

[0078] Considering the common activities of the elderly living alone at home, the radar board is used to collect the echo signals of human continuous behaviors, and 10 continuous behavior sequences containing 12 actions are collected, where each sequence contains 6 behaviors:

[0079] (1) Jump - Walk - Drink water - Sit down - Stand up - Fall;

[0080] (2) Sit down - Stand up - Drink water - Walk - Jump - Fall;

[0081] (3) Walk - Sit down - Stand up - Drink water - Jump - Fall;

[0082] (4) Stretching exercise - Drink water - Walk - Sit down - Stand up - Fall;

[0083] (5) Sit down - Stand up - Stretching exercise - Walk - Drink water - Fall;

[0084] (6) Drink water - Walk - Stretching exercise - Sit down - Stand up - Fall;

[0085] (7) Tai chi - Jogging - Stretching exercise - Cough - Fall - Wave for help;

[0086] (8) Cough - trot - pick up something - stretch - fall - wave for help;

[0087] (9) Pick up something - trot - trot - stretch - fall - wave for help;

[0088] (10) Stretch - pick up something - trot - cough - sit down - stand up;

[0089] After the acquisition is completed, the processed continuous behavior data packets at the front - end are transmitted to the PC through the serial port.

[0090] II. Construction of Continuous Behavior Micro - Doppler Matrix

[0091] As Figure 2 shown, the continuous behavior micro - Doppler matrix is obtained through analog - to - digital conversion, matrix rearrangement, static cancellation, and range - Doppler map (RDM) construction. The detailed content is elaborated as follows:

[0092] The analog form obtained after the original signal collected by the radar passes through mixing and low - pass filtering is:

[0093]

[0094] The above - mentioned analog signal is first converted into a digital signal through analog - to - digital conversion, and then matrix rearrangement is carried out.

[0095] In order to extract the micro - Doppler information of the target, the digital signal needs to be rearranged into the form of a three - dimensional data matrix according to the order of the transmitting and receiving antennas. First, the data corresponding to 4 receiving antennas in the previously obtained two - dimensional matrix are separated; and the matrix is arranged in units of rows on the basis of the original sampling points until the last chirp is completed. At this time, assume that the nth sampling point of the mth chirp is expressed as:

[0096]

[0097] In the formula, λ represents the signal wavelength, R represents the distance between the human body and the radar, T f is the interval of ADC sampling in the fast - time dimension, T s is the sampling interval in the slow - time dimension, f b =(B / T c )τ. In addition, N r is the number of sampling points, and M is the total number of chirps transmitted during the behavior acquisition.

[0098] Since multiple receiving antennas are used in this paper and there is a relative distance between each receiving antenna, each antenna will generate an additional phase shift. Taking 2 receiving antennas as an example, assume that the distance between the receiving antennas is d i, the target human body is located at the position of θ angle in the front left of the radar. At this time, the phase change can be mathematically deduced as the following equation:

[0099]

[0100] Thus, the nth sampling point of the mth chirp on the ith receiving antenna can be expressed as:

[0101]

[0102] Finally, the 12 data matrices are spliced in the order of virtual receiving antennas to obtain a three-dimensional data matrix. This matrix contains the complete micro-motion information of the human target and will be used as the basis for constructing the micro-Doppler spectrogram in the follow-up.

[0103] When the radar system collects human behavior data, the echo signal carries the interference generated by the objects in the environment, such as walls, furniture, etc. In addition, the radar itself will also generate interference. The above-mentioned interferences are collectively referred to as static interferences. Sometimes their intensity approaches or even exceeds the target signal, which has a certain impact on the detection result. Therefore, this paper introduces the static cancellation technology. First, the obtained three-dimensional matrix is processed separately according to the antenna order. Here, taking the ith receiving antenna as an example, the row average value of the corresponding data matrix is calculated sequentially by rows, as shown in the formula:

[0104]

[0105] Among them, represents the value at the position of the nth sampling point of the mth chirp on the ith receiving antenna. Subsequently, each column of the original data matrix is subtracted from the average value matrix. Here, taking the mth column as an example:

[0106]

[0107] Finally, the target data matrix after static cancellation can be obtained as Y'[n, m, i]. Where n ∈ [1, N r , m ∈ [1, M], i ∈ [1, N a .

[0108] In order to construct the micro-Doppler matrix, it is necessary to perform Fourier transforms on the row and column dimensions of the target data matrix after static cancellation respectively. As Figure 3 shown, it presents the target data matrix in a certain frame, which contains three dimensions, namely the range dimension, the velocity dimension, and the antenna dimension corresponding to the phase centers of each antenna.

[0109] First, a Blackman window is added to the range dimension of the three-dimensional array and an N r -point range FFT is performed to obtain the range information of the target:

[0110]

[0111] Subsequently, a Hamming window is added in the velocity dimension to perform an N d -point Doppler FFT to obtain the velocity information of the target:

[0112]

[0113] Finally, the zero-frequency component is shifted to the center of the spectrum, and the average is taken over the antenna dimension to obtain the corresponding RDM, which can be specifically expressed as:

[0114]

[0115] Project the obtained RDM onto the velocity dimension, and then sum it up sequentially in the column direction according to the frame number sequence to obtain a micro-Doppler feature matrix containing the target motion information. Taking the q-th frame as an example here, its calculation process can be described as:

[0116]

[0117] In the formula, represents the k-th sampling point of the (1+(q-1)p)-th chirp on the i-th receiving antenna. In addition, p ∈ [1, N d , q ∈ [1, M / N d .

[0118] III. Visualization of Dynamic Features

[0119] To improve the imaging quality of the micro-Doppler features, color mapping is only performed on the matrix elements that change due to human motion. The specific process is as Figure 4 shown, where I = M / N d .

[0120] First, the elements of the above-obtained micro-Doppler matrix Y' are taken out column by column, and they are sorted in descending order of element magnitude to obtain a column vector r. Select a mapping factor α (0 < α < 100), retain the first α% of the elements of the column vector r, set the remaining elements to zero, and then restore them according to their positions in the micro-Doppler matrix Y' to obtain the processed micro-Doppler matrix Y pre '. Finally, the processed micro-Doppler matrix is mapped, as Figure 5 shown. Taking walking in continuous behavior as an example here, the horizontal axis in the figure represents the frame number, the vertical axis represents the frequency, and the main part of the curve represents the frequency change when the target human body moves relative to the radar. When the target approaches the radar, the frequency is negative; otherwise, it is positive. In addition, the lighter-colored peaks around the main part are the frequency changes generated by the swinging of the limbs during the movement of the torso.

[0121] IV. Method for Detecting and Recognizing Continuous Behaviors of Elderly People Living Alone Based on ConYOLOv5s

[0122] With the transformation of specific application scenarios, there is still much room for improvement in existing object detection models.

[0123] In the process of exploring the continuous behavior recognition of solitary elderly people, the present invention discovers that for the micro-Doppler spectrogram corresponding to continuous behaviors, the horizontal axis is time and the vertical axis is the frequency of the executed actions, and the two are inseparably related. In this regard, in order to ensure the comprehensive extraction and fusion of useful information, it is necessary to comprehensively consider the features in the horizontal and vertical directions of the spectrogram. To sum up, the present invention selects the YOLOv5s model and introduces a coordinate attention mechanism (CA) in the layer before the detection layer in the model. This module can capture the long-term dependencies in the current spatial direction and at the same time retain the accurate position information in the other spatial direction, which helps the network to more accurately locate the objects of interest and achieve sufficient feature extraction, thereby improving the detection performance of the model. At the same time, considering the importance of the loss function and activation function for deep learning models, the present invention also conducts corresponding exploratory experiments. Through experimental verification, the loss function of the model is replaced with SIoU, and the activation function SiLU in the CBS (Conv + Batch Normalization + SiLU) module is replaced with Mish. The model obtained in this way has the best recognition effect. The improved ConYOLOv5s model after the above improvements is as Figure 6 shown. Among them, the introduction of each improved module is as follows:

[0124] The CA mechanism can effectively extract the information of different channels and consider the encoding problem of spatial information. In addition, CA can simultaneously capture the position information in the horizontal and vertical directions and achieve the full fusion of spatio-temporal features, which helps the model to better detect and recognize. On this basis, the loss function and activation function are improved and replaced with SIoU and Mish respectively.

[0125] The continuous behavior micro-Doppler spectrograms of 12 common behaviors including solitary elderly people collected are used as the input of the object detection model. After action segmentation, annotation, coordinate transformation and label file generation, the samples are divided into a training set, a validation set and a test set; first, the model is trained through the training set to enable it to extract the features of the corresponding behaviors; then the validation set is used to test the learning effect of the model, and the model with the best effect is saved; by using the best model for the test set, the test results are obtained, and the classification results and start and end times of each behavior included in each continuous behavior spectrogram are output. The recognition results are as Figure 7 shown.

Claims

1. A radar continuous action detection and behavior recognition method for intelligent elderly care applications, characterized in that Including the following steps: Step 1: Use radar to collect continuous human behavior data and perform data preprocessing; Step 2: Selectively image the micro-Doppler features using a dynamic feature visualization method; Step 3: Use a behavior detection model improved based on a visual target detection model to detect the obtained micro-Doppler map containing continuous action information, so as to obtain the type and start and end times of each action, and identify the human behavior in the intelligent elderly care application scenario according to the correlation between actions; The specific content of Step 2 is as follows: First, take out the elements in the obtained micro-Doppler matrix column by column, and sort the taken-out elements in descending order of their magnitudes to obtain a column vector r'; Secondly, select a mapping factor α, 0 < α < 100, retain the first α% of the elements in r', and set the remaining elements to zero; Then, restore according to the positions of the elements in r' in the original micro-Doppler matrix to obtain a processed micro-Doppler matrix; Finally, perform a linear mapping of the colors of the processed micro-Doppler matrix to obtain an imaged micro-Doppler map; The construction of the micro-Doppler matrix is specifically to obtain a continuous behavior micro-Doppler matrix through analog-to-digital conversion, matrix reorganization, static elimination, and range-Doppler map (RDM) construction of continuous behavior. The detailed content is described as follows: The analog form obtained after the original signal collected by the radar passes through mixing and a low-pass filter is: First, convert the above analog signal into a digital signal through analog-to-digital conversion, and then perform matrix reorganization; Reorganize the digital signal into the form of a three-dimensional data matrix according to the transceiver antenna order; First, separate the data corresponding to 4 receiving antennas in the previously obtained two-dimensional matrix; And arrange the matrix row by row based on the original sampling points until the last chirp is completed; At this time, assume that the nth sampling point of the mth chirp is expressed as: Where, λ represents the signal wavelength, R represents the distance of the human body relative to the radar, and T f is the interval of ADC sampling in the fast time dimension, and T s is the sampling interval in the slow time dimension, and f b =(B / T c )τ; In addition, N r is the number of sampling points, and M is the total number of chirps transmitted during the behavior acquisition process; Since multiple receiving antennas are used, there is a relative distance between each receiving antenna, so each antenna will generate an additional phase shift; Here, taking two receiving antennas as an example, assume the distance between the receiving antennas is d i , and the target human body is located at the position of θ angle in the front left of the radar. At this time, the phase change can be mathematically deduced as the following equation: Therefore, the nth sampling point of the mth chirp on the ith receiving antenna is expressed as: n ∈ [1, N r , m ∈ [1, M], i ∈ [1, N a ​ Finally, splice 12 data matrices in the order of virtual receiving antennas to obtain a three-dimensional data matrix; This three-dimensional data matrix contains the complete micro-motion information of the human target and will be used as the basis for subsequent construction of the micro-Doppler spectrogram; When the radar system collects human behavior data, the echo signal carries interference generated by objects in the environment. First, process the previously obtained three-dimensional matrix separately according to the antenna order. Here, take the ith receiving antenna as an example, and calculate the row average of the corresponding data matrix row by row in turn, as shown in the formula: Among them, represents the value of the nth sampling point of the mth chirp on the ith receiving antenna; subsequently, each column of the original data matrix is subtracted from the average value matrix. Here, the mth column is taken as an example: n ∈ [1, N r , m ∈ [1, M], i ∈ [1, N a ​ Finally, the target data matrix after static cancellation is obtained as Y'[n, m, i]; where n ∈ [1, N r , m ∈ [1, M], i ∈ [1, N a ; It is necessary to perform Fourier transforms on the row and column dimensions of the target data matrix after static elimination respectively; First, perform an N-point distance FFT on the distance dimension of the three-dimensional array after adding a Blackman window to obtain the distance information of the target: r ​ k ∈ [1, N r , m ∈ [1, M], i ∈ [1, N a ​ Subsequently, a Hamming window is added in the velocity dimension to perform an N d -point Doppler FFT to obtain the velocity information of the target: k ∈ [1, N r , l ∈ [1, M], i ∈ [1, N a ​ Finally, move the zero-frequency component to the center of the spectrum and take the average of the antenna dimension to obtain the corresponding RDM, which is specifically expressed as: k ∈ [1, N r , l ∈ [1, M] Project the above-obtained RDM onto the velocity dimension and sum up in the column direction in the order of frames to obtain a micro-Doppler feature matrix containing target motion information. Here, take the qth frame as an example, and its calculation process is described as: In the formula, represents the k-th sampling point of the (1 + (q - 1)p)-th chirp on the i-th receiving antenna; in addition, p ∈ [1, N d , q ∈ [1, M / N d .

2. The radar continuous action detection and behavior recognition method for intelligent elderly care applications according to claim 1, characterized in that The specific steps of Step 1 are as follows: First, perform analog-to-digital conversion on the obtained original signal to obtain a digital signal; second, perform matrix reorganization on the digital signal and arrange it in the order of virtual receiving antennas; then, perform static cancellation on the reorganized three-dimensional matrix to eliminate the influence of background noise; finally, obtain the micro-Doppler matrix corresponding to the continuous behavior through range-FFT and Doppler-FFT.

3. The radar continuous action detection and behavior recognition method for intelligent elderly care applications according to claim 2, characterized in that The collection of human continuous behavior data specifically uses a radar board to collect the echo signals of human continuous behavior, and collects multiple continuous behavior sequences containing various actions, where each behavior contains one or more atomic actions; the collected continuous behavior sequences include the following sequences: (1) Jump - Walk - Drink water - Sit down - Stand up - Fall; (2) Sit down - Stand up - Drink water - Walk - Jump - Fall; (3) Walk - Sit down - Stand up - Drink water - Jump - Fall; (4) Stretch - Drink water - Walk - Sit down - Stand up - Fall; (5) Sit down - Stand up - Stretch - Walk - Drink water - Fall; (6) Drink water - Walk - Stretch - Sit down - Stand up - Fall; (7) Tai Chi - Jog - Stretch - Cough - Fall - Wave for help; (8) Cough - Jog - Pick up something - Stretch - Fall - Wave for help; (9) Pick up something - Jog - Jog - Stretch - Fall - Wave for help; (10) Stretch - Pick up something - Jog - Cough - Sit down - Stand up.

4. The radar continuous action detection and behavior recognition method for intelligent elderly care applications according to claim 1, characterized in that, The specific steps of Step 3 are as follows: Through a dedicated calibration tool, calibrate a single behavior in the micro-Doppler spectrogram to obtain the coordinates of the upper left corner and the lower right corner in the calibration box corresponding to each behavior, and then calculate the center point coordinates and width and height values from the obtained upper left corner and lower right corner coordinates; by introducing a target detection model and adding a coordinate attention mechanism CA on this basis, extract and fuse the temporal and spatial features of the continuous behavior spectrogram; Improve the detection performance by improving the loss function and activation function to obtain a continuous behavior detection and recognition model; on this basis, divide the continuous behavior spectrogram dataset into a training set, a validation set, and a test set; first input the training set and the validation set into the improved target detection model; extract the features of the training set through the model, and the CA attention mechanism performs the extraction and fusion of spatio-temporal domain features; obtain the model with the best effect through the validation set; finally, test the obtained model through the test set, so as to realize the segmentation and detection of continuous actions and identify human behaviors according to the correlation between actions.

Images