A method for identifying micro-movements of human joints using depth camera-assisted radar

Abstract

The present invention discloses a method for using a depth camera to assist radar in identifying micro-motions of human joints. The method comprises: simultaneously receiving data collected by radar and a depth camera on the motion states of several human targets, obtaining radar data and depth camera data of the human target micro-motions, respectively; processing the data according to the radar format to obtain one-dimensional time series radar data; performing time-frequency analysis on the processed radar data to obtain micro-Doppler signatures corresponding to the human targets; low-pass filtering the depth camera data, converting the time-varying distance information of each joint of a human target after the filtering process into time-varying velocity information of each joint; and matching the micro-Doppler signature of a human target with the time-varying velocity information of each joint. The present invention solves the problem of difficulty in distinguishing the micro-Doppler frequency components corresponding to different joints of a human target in the radar micro-Doppler signature.

Description

Technical Field

[0001] The present invention relates to the field of application of human micro-Doppler characteristics and depth camera data, and in particular to a method for using a depth camera to assist radar in identifying micro-movements of human joints. Background Art

[0002] When a radar target experiences radial motion relative to the radar, the frequency of the radar echo will produce a Doppler effect. In many cases, in addition to translational motion, any structural component of an object will also experience some oscillatory motion—called micromotion. Examples include the rotating blades of a helicopter. The flapping of a bird's wings in flight and the swinging of a pedestrian's arms and legs are all considered micromotions. Micromotions can cause Doppler frequency modulation of the transmitted radar signal around the carrier frequency. This Doppler frequency modulation is called the micro-Doppler frequency. Different targets have different micromotion characteristics, and therefore produce different micro-Doppler frequencies. The unique micro-Doppler frequencies generated by different radar targets are called micro-Doppler signatures, which can be used for target identification and classification. Human targets exhibit typical micromotions during motion. Due to the flexibility of human joints, both the micromotions of human targets and the micro-Doppler signatures they generate are highly complex and unique. Human micro-motions primarily consist of the swinging of the arms and legs. Therefore, the micro-Doppler signature of a human target is a combination of the micro-Doppler frequency components of the arms and legs. Furthermore, an arm also includes different components, such as the elbow, wrist, and hand, which all contribute differently to the micro-Doppler frequency. Therefore, the human body's micro-Doppler signature can be used to identify and classify the target's motion state.

[0003] In recent years, human micro-Doppler features combined with classification and recognition algorithms have been used to classify the motion state of human targets and recognize gait, and much progress has been made. However, for the micro-motion of each joint of a human target, a single-channel radar can only obtain the micro-Doppler features of one (or more) human targets as a whole. In addition, without prior knowledge or when the motion state is complex, it is difficult to distinguish the contribution of each joint of the human body to the micro-Doppler features using only the radar micro-Doppler features. Therefore, the micro-Doppler features obtained by a single-channel radar alone cannot quantitatively analyze the micro-Doppler frequencies contributed by each joint of the human target. See reference [1] (J. Li and P. Stoica, MIMO Radar Signal Processing, John Wiley & Sons, 2008.) A solution is proposed to increase the number of channels of the radar system and use multi-input multi-output technology, but this will seriously increase the complexity of the radar system. Considering that a radar system with high system complexity is not desirable in civilian equipment, it is not recommended to use multi-input multi-output technology to separate the micro-Doppler frequencies of each joint or limb. Another solution is to use an algorithm to estimate the micro-Doppler frequencies of different limbs. See reference [2] (R. G. Raj, V. C. Chen, and R. Lipps, “Analysis of radar human gait signatures,” IET Image Process, Vol. 4, No. 3, 2010.). However, the algorithm proposed above has only been verified for simple motion states in simulated micro-Doppler signatures. Currently, there is little work on separating the different components of micro-Doppler signatures of different limbs or joints of the human body, but this work is the basis for quantitative analysis of the motion state of each limb or joint of the human body, and the results of this work have the potential to improve the ability to identify and classify human motion state or gait. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose a method for using a depth camera to assist radar in identifying micro-movements of human joints.

[0005] To achieve the above objectives, the present invention proposes a method for identifying human joint micro-movements using a depth camera-assisted radar. The method is based on radar and a depth camera and includes:

[0006] Step 1) simultaneously receiving data collected by radar and depth camera on the motion status of several human targets, and obtaining radar data and depth camera data of micro-motion of human targets respectively;

[0007] Step 2) Processing is performed according to the radar format to obtain radar data in a one-dimensional time series;

[0008] Step 3) performing time-frequency analysis on the processed radar data to obtain micro-Doppler characteristics corresponding to human targets;

[0009] Step 4) low-pass filtering the depth camera data, and then converting the distance information of each joint of a human target over time after filtering into the velocity information of each joint over time;

[0010] Step 5) Match the micro-Doppler characteristics of a human target with the velocity information of each joint that changes over time.

[0011] As an improvement to the above method, the motion state of the human target includes: walking, running or other motion states; the depth camera data is a two-dimensional distance information matrix of each human target obtained from the data collected by the depth camera.

[0012] As an improvement to the above method, step 2) specifically includes:

[0013] Step 2-1) When the radar is a non-modulated continuous wave radar, the radar data is mixed and sampled by an ADC to obtain a one-dimensional time series of radar echoes;

[0014] Step 2-2) Otherwise, pulse compression processing is performed on the radar data to obtain two-dimensional range-slow time domain data, and then the range gate where the human target is located is intercepted and added to obtain the one-dimensional time series radar data of the human target.

[0015] As an improvement to the above method, step 2-1) specifically includes:

[0016] When the radar is a non-modulated continuous wave radar, the received radar echo signal s r (t) After mixing and ADC sampling, the radar echo is obtained as a one-dimensional time series s rdn (t i )for:

[0017]

[0018] Among them, t i represents the i-th moment, N represents the N radar scattering centers in the area illuminated by the radar, k n is the amplitude attenuation coefficient of the nth scattering center in the area illuminated by the radar, A represents the amplitude of the non-modulated continuous wave radar, is the initial phase caused by the initial position of the nth scattering center, j represents the imaginary part, is the Doppler frequency generated by the nth scattering center, satisfying the following formula:

[0019]

[0020] v n represents the radial velocity of the nth scattering center, and λ represents the wavelength of the radar.

[0021] As an improvement to the above method, step 2-2) specifically includes:

[0022] Pulse compression is performed on the radar data to obtain M rows and N r The range-slow time domain two-dimensional data of the column, where M is the number of pulses emitted by the radar, N is the r The number of fast time sampling points for each pulse;

[0023] Based on the fact that the motion trajectory of the human target only exists within a limited range gate, we get M rows and N s Column distance-slow time domain two-dimensional data, where N s ≤N r ;

[0024] N s The column range gate data is added according to the corresponding row to obtain M rows of one-dimensional time series radar data s p (t i ), where i = 1, 2, 3,…, M.

[0025] As an improvement to the above method, step 3) specifically includes:

[0026] Taking into account both time resolution and frequency resolution, a suitable window length is selected to analyze the one-dimensional time series of radar echoes. rdn (t i ) or one-dimensional time series radar data s p (t i ) Use short-time Fourier transform, Gabor transform or WVD transform to perform time-frequency analysis to obtain the micro-Doppler characteristics of human targets, which are two-dimensional data in the slow time-Doppler frequency domain.

[0027] As an improvement to the above method, step 4) specifically includes:

[0028] The depth camera data is low-pass filtered, and the distance information of each joint after filtering is converted into the speed information of each joint over time. Each joint point of the human target is equivalent to a radar scattering center, and the nth joint point of the human target at the i-th frame t is obtained. i Instantaneous speed value at the moment for:

[0029]

[0030] Among them, FrameRate is the frame rate of the depth camera, is the nth joint point at the i-th frame t i Distance information at the moment, is the n-1th joint point at the i-th frame t i Distance information at the moment.

[0031] As an improvement to the above method, step 5) specifically includes:

[0032] Convert the micro-Doppler characteristics of a human target into the slow time-velocity domain;

[0033] Sequentially extract the instantaneous velocity value of each joint of the human target;

[0034] The time units and speed units of the instantaneous velocity data and the micro-Doppler characteristics are unified, and then the instantaneous velocity data of each joint of the human target are matched with the micro-Doppler characteristics, thereby realizing the identification of the micro-Doppler components of each joint in the human radar micro-Doppler characteristics.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] 1. The method proposed in this paper is applicable to human micro-Doppler signatures obtained under all radar systems. Experimental data demonstrates that by using depth camera data that can be arbitrarily extracted from human joint data and matching it with radar micro-Doppler signatures, the micro-Doppler frequency components of each joint can be identified in the micro-Doppler signatures.

[0037] 2. The method of the present invention uses depth camera data to assist in identifying micro-motion information of different joints of human targets in radar data. This not only solves the problem of difficulty in distinguishing the micro-Doppler frequency components corresponding to different joints of human targets in radar micro-Doppler characteristics, but also has the advantages of low cost, contactless operation, and easy data acquisition of depth cameras, thereby achieving the purpose of obtaining a sufficient data set to train radar data to automatically identify micro-motion information of various joints of human targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the method for identifying micro-movements of human joints using a depth camera-assisted radar according to the present invention;

[0039] Figure 2 This is a schematic diagram of the radar and depth camera data collection experimental scene;

[0040] Figure 3 It is the micro-Doppler characteristic of the human target in the state of swinging arms in place;

[0041] Figure 4 This is a schematic diagram of N joint points used by the depth camera sensor to identify human targets;

[0042] Figure 5 It is a distance-time curve of the human target in the state of swinging arms in place after low-pass filtering collected by the depth camera;

[0043] Figure 6 It is the matching result of the velocity-time curve and micro-Doppler characteristics of the torso and legs obtained using the depth camera data when the human target is in the state of swinging arms in place;

[0044] Figure 7 This is the matching result of the velocity-time curve and micro-Doppler characteristics of the left arm obtained using the depth camera data when the human target is in the state of swinging arms in place;

[0045] Figure 8 This is the matching result of the velocity-time curve and micro-Doppler characteristics of the right arm obtained using depth camera data when the human target is in the state of swinging arms in place;

[0046] Figure 9 It is the micro-Doppler characteristic of the human target in the walking state;

[0047] Figure 10 It is a distance-time curve of the human target in the walking state after low-pass filtering collected by the depth camera;

[0048] Figure 11 It is the matching result of the velocity-time curve and micro-Doppler characteristics of the torso obtained by using the depth camera data when the human target is walking;

[0049] Figure 12 It is the matching result of the velocity-time curve and micro-Doppler characteristics of the left leg obtained using the depth camera data when the human target is walking;

[0050] Figure 13 It is the matching result of the velocity-time curve and micro-Doppler characteristics of the right leg obtained using the depth camera data when the human target is walking;

[0051] Figure 14 This is the matching result of the velocity-time curve and micro-Doppler characteristics of the left arm obtained using depth camera data when the human target is walking;

[0052] Figure 15 It is the matching result of the velocity-time curve and micro-Doppler characteristics of the right arm obtained using the depth camera data when the human target is in a walking state. DETAILED DESCRIPTION

[0053] Depth camera sensors are low-cost human motion tracking devices that can identify the human skeleton, track the three-dimensional coordinates of each joint in real time, and record human motion data. Consequently, depth cameras have become widely used for gesture and gait recognition. Currently, depth camera-based human motion tracking simulations are being developed to generate training data for classification algorithms. Furthermore, depth camera sensors are used to acquire data synchronously with radar, providing an additional data source. The depth camera data is considered the ground truth for human motion.

[0054] Existing work has verified that depth camera sensors have high application potential in areas such as human motion state recognition and classification, and gait recognition.

[0055] The present invention aims to identify the micro-Doppler frequency components contributed by individual limbs or joints based on the human body's micro-Doppler signature. A method is proposed to utilize depth camera data to assist in identifying the micro-Doppler components of individual joints within the radar micro-Doppler signature of the human body. To achieve automatic identification of individual joints within the radar micro-Doppler signature, depth camera data provides an important foundation for assisting in learning and identifying micro-Doppler information for joints. Furthermore, quantitatively analyzing the motion information of individual limbs or joints based on the human micro-Doppler signature helps improve the accuracy of existing algorithms for human motion state classification and gait recognition based on human micro-Doppler signatures. This method combines radar micro-Doppler signatures with human motion data collected by a depth camera. By placing radar and depth camera sensors together and simultaneously collecting motion data of a human target, the two sensors are processed separately. The resulting radar micro-Doppler signature is then aligned with the depth camera's velocity curve to identify the micro-Doppler components of individual joints within the radar micro-Doppler signature of the human body. Experimental data validates the effectiveness of the proposed method.

[0056] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0057] Example 1

[0058] like Figure 1 As shown, the present invention proposes a method for identifying micro-movements of human joints using a depth camera-assisted radar, the method comprising:

[0059] Step 1) The radar and depth camera simultaneously collect data on the human target, and obtain radar data and depth camera data of the human micro-motion target respectively; specifically including:

[0060] During the data collection process, the radar and depth camera are placed in the same position and data collection is performed simultaneously. The human target is within the line of sight of the radar and depth camera, and faces (or turns its back) to the radar and depth camera while walking, running, or performing other motion states to complete the radar and depth camera data collection and storage.

[0061] When the radar is a non-modulated continuous wave system, the transmitted signal is a non-modulated continuous wave signal s(t):

[0062] s(t)=Aexp(jw0t) (1)

[0063] Where A represents the amplitude of the non-modulated continuous wave signal, w0=2πf0 is the frequency of the non-modulated continuous wave signal, and t represents time. After the transmitted signal is irradiated on a moving target, it reaches the radar receiving antenna after scattering, and the echo signal s r (t):

[0064]

[0065] Where k is the attenuation coefficient of the amplitude, R0 is the distance between the target and the radar at the initial moment, τ is the time delay from the transmission to the reception of the radar signal, and v represents the radial velocity of the radar target (when moving close to the radar, the velocity is positive, otherwise it is negative). r (t) can be obtained after mixing s rd (t), which is the data collected by the radar:

[0066] s rd (t) = kAexp[j2π(f d t-φ0)] (3)

[0067] where f d The Doppler frequency generated for the moving target is:

[0068]

[0069] If there are N targets in the area illuminated by the radar, the signals received by the radar are N different s rd (t) The sum of the signals, after ADC sampling, the one-dimensional time series of the radar echo is s rdn (t i ):

[0070]

[0071] N represents the N radar scattering centers existing in the area illuminated by the radar. Taking the human target as an example, when the size of the human target is much larger than the incident wavelength, the torso and limbs of each human target in the radar's line of sight can be regarded as different radar scattering centers.

[0072] When a depth camera is used to capture data about a human subject, the stored data is the distance information of N joints in each frame, representing a two-dimensional data matrix. To study the micro-Doppler frequency of human joints, we only need to extract the one-dimensional distance information sequence of the corresponding joints from the two-dimensional depth camera data, which changes over the number of frames (i.e., time).

[0073] When using a depth camera to collect data on multiple human targets simultaneously, it can identify and mark different human targets, and store a two-dimensional distance information matrix for each human target.

[0074] Step 2) If the radar is a non-modulated continuous wave radar, skip step 3) and proceed directly to step 4); otherwise, proceed to step 3);

[0075] Step 3) performing pulse compression processing on the radar echo signal to obtain data in the two-dimensional range-slow time domain, intercepting the distance unit where the human target is located, and performing addition processing according to the corresponding time series to obtain a one-dimensional time series; specifically comprising:

[0076] For pulse radar data, the original radar echo data needs to be pulse compressed to obtain M rows and N columns of range-slow time domain two-dimensional data. Generally, in the process of radar acquiring data, the motion trajectory of human targets only exists within a limited range gate. Therefore, in order to reduce the amount of data and avoid the influence of environmental clutter, the range gate data where the human target exists can be extracted. That is, the amount of distance dimension data of the above-mentioned range-full time two-dimensional data can be reduced to obtain M rows and N columns of range-slow time domain two-dimensional data. s Column distance-slow time domain two-dimensional data, where N s ≤N r . N of the above two-dimensional data s The column range gate data is added according to the corresponding row, and the one-dimensional time series s of M rows is obtained. p (t i ), where i = 1, 2, 3,…, M.

[0077] Step 4) performing time-frequency analysis on the radar one-dimensional time series to obtain the micro-Doppler characteristics of the human body; specifically, the following steps are performed:

[0078] For the one-dimensional time series s of radar data obtained in step 1) rdn (t i ) or the radar data one-dimensional time series s obtained in step 3) p (t i ), short-time Fourier transform is used to realize the time-frequency analysis of the one-dimensional time series of radar data. The short-time Fourier transform of the time series x(t) can be expressed as:

[0079] STFT(t,w)=∫x(τ)g(τ-t)e -jwt dτ (6)

[0080] Where g(t) is a window function with a very narrow time width. It slides along the time axis and completes the time-frequency analysis of the one-dimensional time series of radar data according to formula (6). The smaller the width of the window function, the lower the frequency resolution and the higher the time resolution. Conversely, the larger the width of the window function, the higher the frequency resolution and the smaller the time resolution. The short-time Fourier transform degenerates into the Fourier transform.

[0081] Therefore, when selecting the time width of the window function g(t), both time resolution and frequency resolution should be considered. After selecting an appropriate window length, the one-dimensional time series s of the radar data obtained in step 1) is rdn (t i ) or the radar data one-dimensional time series s obtained in step 3) p (t i ) is subjected to short-time Fourier transform to obtain the micro-Doppler characteristics of the human target, which is two-dimensional data in the slow time-Doppler frequency domain.

[0082] Step 5) First, perform low-pass filtering on the depth camera data, and then convert the filtered distance information of each joint over time into the velocity information of each joint over time. Specifically, it includes:

[0083] The depth camera sensor detects human targets and collects data as real-time position information of N joints, that is, the distance information of N joints of the human target in the depth camera's sight from the depth camera as time changes. The superscript t i Indicates the time point of the i-th frame, and the subscript n is the n-th joint point. In order to reduce the influence of high-frequency noise, a low-pass filter is used to smooth the raw data of the depth camera, and then the instantaneous speed is calculated according to formula (7):

[0084]

[0085] In order to facilitate the joint processing of depth camera data and radar data, Indicates the nth joint point at the time t in the i-th frame i The instantaneous speed value at time t i -t i-1 , represents the time difference between the i-th frame and the adjacent i-1-th frame. For the depth camera sensor, the time difference between adjacent frames is the inverse of the frame rate, that is, formula (7) can be rewritten as:

[0086]

[0087] Where FrameRate is the frame rate. The instantaneous velocities of the N joints of the human target can be calculated according to formula (8).

[0088] Step 6) Matching the micro-Doppler characteristics obtained in step 4) with the velocity information of each joint over time obtained in step 5). Specifically including:

[0089] The Doppler frequency dimension of the slow time-Doppler frequency domain two-dimensional data obtained in step 4) is converted into a velocity dimension according to formula (4), that is, the micro-Doppler characteristics of the human target are converted into the slow time-velocity domain for display; the instantaneous velocity data of the N joint points of the human target obtained in step 5) are sequentially extracted, and the time unit and velocity unit are unified with the time unit and velocity unit of the micro-Doppler characteristics, and then the instantaneous velocity data of each joint is matched with the micro-Doppler characteristics, so as to achieve the goal of resolving the micro-Doppler components of each joint of the human body in the human radar micro-Doppler characteristics.

[0090] Figure 2 Schematic diagram of the experimental scenario for human target detection. The radar and depth camera sensors are placed in the same location to simultaneously collect motion data of the human target, which is within the line of sight of the radar and depth camera sensors. Figure 3 This is the micro-Doppler signature detected by radar when a human target is swinging its arms in place. It can be seen that the scattered energy from the torso, which is at zero Doppler frequency, is the strongest, while the scattered energy from the arms is weaker than that from the torso. Figure 4 A diagram showing the N joints identified by the depth camera sensor. Since the depth camera data is used to analyze radar micro-Doppler signatures, it's not necessary to analyze all joints sequentially. Here, we extract the depth camera data for only 11 typical and important joints, highlighted in yellow. Each arm is represented by two joints, each leg by three, and the torso by one. Figure 5 This is a low-pass filtered distance-time domain graph of 11 joint points captured by the depth camera. It shows that the human target is approximately 3 meters from the depth camera sensor (or radar), swinging its arms in place. Only the arm distance curve oscillates, while the torso and legs remain essentially stationary.

[0091] Figure 6This is the matching result of the speed-time curve of the torso and legs obtained from the depth camera data and the micro-Doppler characteristic spectrum obtained from the radar when the human target is swinging its arms in place. The solid curve in the figure is obtained from the depth camera data, the curve marked "torso" represents the speed-time curve of the torso, the curves marked "knee joint", "ankle joint" and "foot" represent the speed curve of the legs, and the spectrum in the figure is obtained from the radar data. According to the matching results, since the experiment shows that the human target is swinging its arms in place, the torso and legs are basically stationary, and the micro-Doppler frequency is basically at zero Doppler frequency. Figure 6 It can be seen that the method for resolving various components of the micro-Doppler feature proposed in the present invention can distinguish the contribution of the trunk and leg movements in the micro-Doppler feature, so that the micro-Doppler frequency components of the trunk and legs can be clearly distinguished from the micro-Doppler feature.

[0092] Figure 7 This figure shows the matching result of the velocity-time curve of the left arm obtained from the depth camera data and the micro-Doppler characteristic spectrum obtained from the radar while the human target is swinging its arms in place. The solid curve in the figure is obtained from the depth camera data, the curve labeled "left elbow joint" represents the velocity-time curve of the left elbow joint (solid line), and the curve labeled "left wrist joint" represents the velocity curve of the left wrist joint (dashed line). The spectrum in the figure is obtained from the radar data. The matching results show that the Doppler frequency generated by elbow joint movement is always smaller than the Doppler frequency generated by wrist joint movement. Based on the depth camera data curve, the micro-Doppler frequency components contributed by the elbow and wrist joints can be distinguished from the micro-Doppler characteristics. Figure 8 This is the matching result of the speed-time curve of the right arm obtained from the depth camera data and the micro-Doppler characteristic spectrum obtained from the radar when the human target is in the state of swinging arms in situ. Similarly, the solid curve in the map is obtained from the depth camera data, the curve marked "right elbow joint" represents the speed-time curve of the elbow joint of the right arm (solid line), and the curve marked "right wrist joint" represents the speed curve of the wrist joint of the right arm (dashed line). The spectrum in the figure is obtained from the radar data. It can be seen from the matching results that the Doppler frequency generated by the elbow joint movement is always smaller than the Doppler frequency generated by the wrist joint movement. According to the depth camera data curve, not only can the micro-Doppler frequency components contributed by the elbow joint and wrist joint be distinguished from the micro-Doppler features, but also the micro-Doppler frequencies of the left and right arms in the micro-Doppler features can be distinguished. If there is only the radar micro-Doppler feature, it is difficult to distinguish the micro-Doppler frequency components of the left and right arms. From Figure 7 and Figure 8It can be seen that the method for resolving various components of the micro-Doppler feature proposed in the present invention can resolve the contribution of the arm movement in the micro-Doppler feature, so that the micro-Doppler frequency component of the arm can be clearly identified from the micro-Doppler feature.

[0093] Figure 9 is the micro-Doppler characteristic of a human target in a walking state detected by the radar. Figure 2 As shown in the figure, a human target faces the radar and depth camera sensors, first walking forward, then walking backward without turning. The micro-Doppler characteristic spectrum obtained by the radar shows that the micro-Doppler frequency contributions of the limbs and torso are difficult to distinguish due to the complex motion state and the non-stationary motion of the limbs and torso. Figure 10 This is a low-pass filtered distance-time plot of 11 joint points captured by the depth camera. It can be seen that during the radar and depth camera data acquisition process, the human target walked from a distance of approximately 3.5 meters to a distance of approximately 1 meter, then moved away to a distance of approximately 3 meters. Throughout the walking process, both the legs and arms exhibited a distinct oscillatory motion.

[0094] Figure 11 The matching result of the speed-time curve of the torso obtained from the depth camera data and the micro-Doppler characteristic spectrum obtained from the radar when the human target is walking. The solid curve in the figure is obtained from the depth camera data, the curve marked "torso" represents the speed-time curve of the torso (solid line), and the spectrum in the figure is obtained from the radar data. Figure 11 It can be seen that the method for identifying each component of the micro-Doppler feature proposed in the present invention can distinguish the micro-Doppler frequency component of the torso in the micro-Doppler feature.

[0095] Figure 12 This figure shows the matching result of the velocity-time curve of the left leg obtained from depth camera data and the micro-Doppler signature spectrum obtained from radar while the subject is walking. The curve is obtained from the depth camera data. The dotted line labeled "left knee joint," the solid line labeled "left ankle joint," and the dash-dotted line labeled "left foot" represent the velocity curves of the left knee, ankle, and foot, respectively. The spectrum is obtained from radar data. The matching results allow the micro-Doppler frequency components at different joints of the left leg to be distinguished from the micro-Doppler signature. Figure 13This figure shows the matching result of the velocity-time curve of the right leg obtained from depth camera data and the micro-Doppler signature spectrum obtained from radar while the subject is walking. The curve is obtained from the depth camera data. The dotted line labeled "right knee joint," the solid line labeled "right ankle joint," and the dash-dotted line labeled "right foot" represent the velocity curves of the right knee, ankle, and foot, respectively. The spectrum is obtained from radar data. The matching results allow the micro-Doppler frequency components at different joints of the right leg to be distinguished from the micro-Doppler signature. At the same time, it can be seen that among the three joints of the leg, the foot has the largest micro-Doppler frequency, and the knee joint has the smallest micro-Doppler frequency. When the human target walks towards the radar and depth camera sensor, the micro-Doppler frequency of the leg basically remains above zero Doppler frequency, and when the left foot reaches the maximum Doppler frequency, the right foot is at the minimum Doppler frequency, that is, zero Doppler frequency; when away from the radar and depth camera sensor, the micro-Doppler frequency of the leg basically remains below zero Doppler frequency, and when the left foot reaches the maximum absolute value Doppler frequency, the right foot is at zero Doppler frequency. Figure 12 and Figure 13 It can be seen that the method of distinguishing each component of the micro-Doppler feature proposed in the present invention can distinguish the contribution of leg movement in the micro-Doppler feature, so that the micro-Doppler frequency components of different joints of the leg can be clearly distinguished from the micro-Doppler feature.

[0096] Figure 14 This figure shows the matching result of the velocity-time curve of the left arm obtained from depth camera data and the micro-Doppler characteristic spectrum obtained from radar while the human target is walking. The curve is obtained from the depth camera data. The curve labeled "left elbow joint" represents the velocity-time curve of the left elbow joint (solid line), and the curve labeled "left wrist joint" represents the velocity curve of the left wrist joint (dotted line). The spectrum in the figure is obtained from radar data. The matching results show that the Doppler frequency generated by elbow joint movement is always smaller than the Doppler frequency generated by wrist joint movement. Based on the depth camera data curve, the micro-Doppler frequency components contributed by the elbow and wrist joints of the left arm can be distinguished from the micro-Doppler characteristics. Figure 15 This is the matching result of the speed-time curve of the right arm obtained from the depth camera data and the micro-Doppler characteristic spectrum obtained from the radar when the human target is walking. Similarly, the curve is obtained from the depth camera data. The curve marked "right elbow joint" represents the speed-time curve of the right elbow joint (solid line), and the curve marked "right wrist joint" represents the speed curve of the right wrist joint (dotted line). The spectrum in the figure is obtained from the radar data. Figure 14 and Figure 15It can be seen that the method for identifying various components of the micro-Doppler feature proposed in the present invention can distinguish the micro-Doppler frequency components of different joints of the right arm from the micro-Doppler feature.

[0097] The method proposed in this invention for distinguishing the different micro-Doppler frequency components at each joint in the human micro-Doppler signature employs short-time Fourier transforms for time-frequency analysis of radar data, but is also applicable to micro-Doppler signatures obtained using other time-frequency analyses, such as Gabor transforms and WVD (Wigner-Ville Distribution). Furthermore, while the motion states of the human target analyzed in this invention are swinging arms in place and walking, the method is equally applicable to other human motion states, such as running, falling, and waving. Furthermore, while the analysis in this invention utilizes only one human target, it is also applicable to situations where multiple human targets are simultaneously in different or the same motion states, such as two people walking toward each other or three people in the motion states of walking, running, and waving, respectively. The method proposed in this invention has important practical significance for extracting and quantitatively analyzing the motion states of individual human limbs or joints by identifying the micro-Doppler frequencies of these limbs or joints.

[0098] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and are intended to be encompassed by the claims of the present invention.

Claims

1. A method for identifying micro-movements of human joints using a depth camera-assisted radar, implemented based on radar and a depth camera, comprising: Step 1) simultaneously receiving data collected by radar and depth camera on the motion status of several human targets, and obtaining radar data and depth camera data of micro-motion of human targets respectively; Step 2) Processing is performed according to the radar format to obtain radar data in a one-dimensional time series; Step 3) performing time-frequency analysis on the processed radar data to obtain micro-Doppler characteristics corresponding to human targets; Step 4) low-pass filtering the depth camera data, and then converting the distance information of each joint of a human target over time after filtering into the velocity information of each joint over time; Step 5) matching the micro-Doppler characteristics of a human target with the velocity information of each joint over time; The step 2) specifically includes: Step 2-1) When the radar is a non-modulated continuous wave radar, the radar data is mixed and sampled by an ADC to obtain a one-dimensional time series of radar echoes; Step 2-2) Otherwise, pulse compression processing is performed on the radar data to obtain data in the two-dimensional range-slow time domain, and then the range gate where the human target is located is intercepted and summed to obtain the radar data of the one-dimensional time series of the human target; The step 2-2) specifically includes: Pulse compression is performed on the radar data to obtain M rows and N r The range-slow time domain two-dimensional data of the column, where M is the number of pulses emitted by the radar, N is the r The number of fast time sampling points for each pulse; Based on the fact that the motion trajectory of the human target only exists within a limited range gate, we get M rows and N s Column distance-slow time domain two-dimensional data, where N s ≤N r ; N s The column range gate data is added according to the corresponding row to obtain M rows of one-dimensional time series radar data s p (t i ), where i = 1, 2, 3,…, M.

2. The method for identifying human joint micro-movements using a depth camera-assisted radar according to claim 1, characterized in that: The motion state of the human target includes: walking, running or other motion states; the depth camera data is a two-dimensional distance information matrix of each human target obtained from the data collected by the depth camera.

3. The method for identifying human joint micro-movements using a depth camera-assisted radar according to claim 1, characterized in that: The step 2-1) specifically includes: When the radar is a non-modulated continuous wave radar, the received radar echo signal s r (t) After mixing and ADC sampling, the radar echo is obtained as a one-dimensional time series s rdn (t i )for: Among them, t i represents the i-th moment, N represents the N radar scattering centers in the area illuminated by the radar, k n is the amplitude attenuation coefficient of the nth scattering center in the area illuminated by the radar, A represents the amplitude of the non-modulated continuous wave radar, is the initial phase caused by the initial position of the nth scattering center, j represents the imaginary part, is the Doppler frequency generated by the nth scattering center, satisfying the following formula: v n represents the radial velocity of the nth scattering center, and λ represents the wavelength of the radar.

4. The method for identifying human joint micro-movements using a depth camera-assisted radar according to claim 3, wherein: The step 3) specifically includes: Taking into account both time resolution and frequency resolution, a suitable window length is selected to analyze the one-dimensional time series of radar echoes. rdn (t i ) or one-dimensional time series radar data s p (t i ) Use short-time Fourier transform, Gabor transform or WVD transform to perform time-frequency analysis to obtain the micro-Doppler characteristics of human targets, which are two-dimensional data in the slow time-Doppler frequency domain.

5. The method for identifying human joint micro-movements using a depth camera-assisted radar according to claim 1, wherein: The step 4) specifically includes: The depth camera data is low-pass filtered, and the distance information of each joint after filtering is converted into the speed information of each joint over time. Each joint point of the human target is equivalent to a radar scattering center, and the nth joint point of the human target at the i-th frame t is obtained. i Instantaneous speed value at the moment for: Among them, FrameRate is the frame rate of the depth camera, is the nth joint point at the i-th frame t i Distance information at the moment, is the n-1th joint point at the i-th frame t i Distance information at the moment.

6. The method for identifying human joint micro-movements using a depth camera-assisted radar according to claim 5, characterized in that: The step 5) specifically includes: Convert the micro-Doppler characteristics of a human target into the slow time-velocity domain; Sequentially extract the instantaneous velocity value of each joint of the human target; The time units and speed units of the instantaneous velocity data and the micro-Doppler characteristics are unified, and then the instantaneous velocity data of each joint of the human target are matched with the micro-Doppler characteristics, thereby realizing the identification of the micro-Doppler components of each joint in the human radar micro-Doppler characteristics.

Images