A hand key point tracking method based on FMCW radar

Abstract

A method for tracking hand key points based on FMCW radar addresses the issues of lighting constraints and privacy leaks in current methods using optical cameras for tracking hand key points. This method utilizes a cross-modal supervision approach, using images and data captured by cameras and radar as neural network inputs during training. The radar data is synchronized with the video stream, and hand key point information is extracted from the video stream for preprocessing, which is then used as monitoring for radio frequency signals processed by the neural network. After training, the system only needs to use radio frequency signals as input to output hand key point tracking. This method improves the accuracy of gesture recognition while protecting personal privacy and regardless of lighting conditions. It is robust, stable, real-time, and efficient.

Description

Technical Field

[0001] The present invention belongs to the intersection of wireless perception and computer vision, relates to the fields of millimeter wave radar and neural network technology, and specifically relates to a hand key point tracking method based on FMCW radar. Background Art

[0002] In recent years, with the advancement of technology, human-computer interaction has gradually developed to the point where it is closely intertwined with daily life. The pursuit of more efficient and simplified information exchange has become the core of human-computer interaction research. The emergence of new technologies such as facial recognition, posture recognition, lip reading, speech recognition, and gesture recognition has gradually transformed the traditional computer-centric interaction model, allowing users to truly become the core component of the human-computer interaction model. These new technologies are more in line with the communication methods and habits between humans and machines, significantly improving the user experience without excessive learning costs.

[0003] Using optical cameras to collect information about hand gestures is a relatively mature gesture recognition method. While high-resolution cameras enable visual gesture recognition to achieve recognition rates exceeding 90%, the lighting conditions of the optical camera's environment significantly impact its recognition rate. In environments with excessively bright or dim lighting, the accuracy of gesture description is significantly reduced. While current technology can address this issue by adding night vision cameras, this also increases technical costs, significantly limiting its scope of application. Furthermore, this method poses privacy concerns. Due to the inherent characteristics of optical images, images, videos, and other information may be leaked. In an era of extreme privacy sensitivity, such information leaks can have a significant impact on product and technology development. Furthermore, this method consumes a lot of energy and requires a lot of computing resources, preventing its widespread application in systems with relatively simple environments. Summary of the Invention

[0004] To address the above technical issues, the present invention proposes a hand keypoint tracking method that utilizes FMCW radar for gesture recognition. This method has the advantage that the data stream for FMCW radar-based gesture recognition is the radar signal, not the optical image signal itself. Even if the signal is leaked, it is difficult for an attacker to directly see any useful information, thus providing a certain degree of security for the system. Furthermore, dynamic gesture recognition based on FMCW radar can be integrated into a high-speed processing chip with low energy consumption and a compact size, making it possible to embed it into portable devices.

[0005] A hand key point tracking method based on FMCW radar includes the following steps:

[0006] Step 1: Initialize the FMCW radar system and configure the parameters for hand information sampling, including the transmit and receive antenna pairs, number of sampling points, and sampling time. At the same time, use the camera to capture the complete hand motion trajectory.

[0007] Step 2: Preprocess the acquired image information and RF signal accordingly. For image information, save the GBR image stored by the camera as an RGB image. For RF signal, perform clutter suppression first, and perform Fourier transform (FFT) in the distance-velocity dimension to form a range-Doppler heat map (RDI). Process the signal in the horizontal and vertical directions relative to the ground to form a horizontal heat map (H). l and vertical heatmap H v ;

[0008] Step 3: Process the hand information in the saved image information, capture the hand information in the image, and automatically mark each key point of the hand based on the obtained information to obtain a hand key point confidence map from the video;

[0009] Step 4, we will get H l and H v The RF encoding network is used to encode the information and input it into the CNN. The feature map is input into different convolutional layers of the CNN to extract the features. The encoded heat map information is then decoded by the RF decoding network to obtain the key point confidence map from the RF.

[0010] Step 5: Using cross-modal learning and supervised learning methods, the image information and RF information of different modalities are interacted. The network that obtains the video keypoint confidence map is called the teacher network, and the network that obtains the RF keypoint confidence map is called the student network. A cross-supervised learning teacher-student network is constructed. The reliability of the keypoint confidence map obtained by RF signal processing is tested through this network. Through cross-supervised learning, the position of the hand keypoints obtained in the RF signal is identified and tracked.

[0011] Step 6: After the training is completed, the system can track the key points of the hand using only the radio frequency signal without the assistance of video images.

[0012] Furthermore, in step 1, the original signal of the dynamic gesture is collected by the FMCW radar. Let the period of each frequency-modulated continuous pulse chirp be t, S be the frequency growth slope, τ be the delay of the signal from the radar to the gesture and then back, and f be the radar carrier frequency; the radar's transmitted signal S1 is expressed as:

[0013] S1=sin(2πft+πSt·t)

[0014] The received signal S2 is expressed as:

[0015] S2=sin[2πf(t-τ)+πS(t-τ) 2 ]

[0016] After passing through the mixer and low-pass filter, the output intermediate frequency signal S is:

[0017]

[0018] Perform one-dimensional Fourier transform on the above formula to obtain the frequency of the intermediate frequency signal f IF , assuming the distance from the gesture target to the radar is d and the speed of light is c, the formula is:

[0019]

[0020] The above method is used to repeatedly perform the same processing on multiple linear frequency modulation pulses chirp, and then the processed signals are spliced into a frame of data to obtain the RF signal from the radar.

[0021] Furthermore, in step 2, the hand image captured by the camera is converted into a corresponding RGB image of size 200×200 for storage; for the radio frequency signal, a feature extraction method based on the frequency domain is used to transform the complex time domain signal into the frequency domain through the Fourier transform method in both the horizontal and vertical directions, observe the status of each frequency component of the signal, and extract the features in the frequency domain; the Fourier transform of the continuous signal is Through FFT processing, a spectrum with different separated peaks is generated, and each peak indicates the existence of an object at a specific distance; further, the phase of each valid data at the same distance is taken to perform FFT again to distinguish multiple targets with different speeds at the same distance; then after performing phase FFT, the phase difference ω1 and ω2 of each target are obtained, and then the targets with different speeds are obtained. The hand feature map obtained at this time is the range-Doppler map RDI; save one horizontal and one vertical antenna array, the horizontal heat map is the projection of the signal reflection on the plane parallel to the ground, and the vertical heat map is the projection of the signal reflection on the plane perpendicular to the ground.

[0022] Furthermore, in step 4, the RF coding network uses 10 layers of 9×5×5 spatiotemporal convolution, with a step size of 1×2×2 for each layer. Batch normalization is used after the input is completed; the ReLU activation function f(x)=max(0,x) is used after each layer, and after encoding is completed, it is input into the CNN.

[0023] Furthermore, in step 4, the radio frequency decoding network decodes the encoded heat map information. The decoding network has 5 layers, except for the last layer with a step size of Except for , the stride lengths of other layers are In addition, the ReLu function is used after each layer, and for the last layer, the sigmoid function is used as the output layer.

[0024] Furthermore, in step 5, the image information and RF signal are input into the teacher network and the student network respectively. The student network receives the key point confidence map marked by the teacher network and compares it with its own predicted key point confidence map. These key point confidence maps from the teacher network provide cross-modal supervision for the student network, allowing the student network to learn from them and successfully predict the key point confidence map.

[0025] Furthermore, in step 5, the goal of training the student network is to minimize the difference between its prediction and the teacher network prediction, and the loss is defined as the sum of the binary cross entropy losses of each pixel in the confidence map:

[0026]

[0027] in and is the confidence of each pixel on the confidence map c; the student network accepts the key point confidence map marked by the teacher network and compares it with the key point confidence map predicted by itself. These key point confidence maps from the teacher network provide cross-modal supervision for the student network, allowing the student network to learn from them and successfully predict the key point confidence map.

[0028] Furthermore, in step 6, after the training is completed, the tracking of the key points of the hand only requires placing the hand in front of the radar to obtain the position coordinates of the key points of the hand, without the need to use video image auxiliary marking, that is, the key points of the hand can be tracked only through the radio frequency signal.

[0029] Beneficial effects of the present invention:

[0030] (1) The present invention uses FMCW radar to identify and track key points of the hand. The radar's electromagnetic waves are not affected by factors such as light, smoke, and visual distance, so the requirements for the environment are relatively low. Even if the environmental conditions change, the reliability and accuracy of motion perception can be high.

[0031] (2) The present invention uses FMCW radar to identify and track key points of the hand. Its data stream is the radar signal, not the optical image itself. Even if the signal is leaked, it is difficult for the attacker to directly see any useful information, which provides a certain degree of security for the system.

[0032] (3) The present invention utilizes FMCW radar to perform hand key point recognition and tracking. The dynamic hand key point tracking of the FMCW radar can be integrated into a high-speed processing chip with low energy consumption and small size, and has high portability and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the hand key point tracking method described in an embodiment of the present invention.

[0034] Figure 2 Schematic diagram of a cross-supervisory teacher-student network for key point tracking in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram showing the completion of image information calibration in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0037] This paper proposes a hand keypoint tracking method that uses FMCW radar for gesture recognition. This method has the advantage that the data stream for FMCW radar-based gesture recognition is the radar signal, not the optical image signal itself. Even if the signal is leaked, it is difficult for an attacker to directly see any useful information, which provides a certain degree of security for the system. Furthermore, dynamic gesture recognition based on FMCW radar can be integrated into a high-speed processing chip with low energy consumption and a small size, making it possible to embed it in portable devices.

[0038] like Figure 1 As shown, the main steps of this method are as follows:

[0039] Step 1: Initialize the FMCW radar system and synchronously acquire the hand image captured by the camera and the RF signal collected by the radar:

[0040] The original signal of the dynamic gesture is collected by the FMCW radar. Let the period of each frequency-modulated continuous pulse chirp be t, S be the frequency growth slope, τ be the delay from the radar to the gesture and then back, and f be the radar's carrier frequency. The radar's transmitted signal S1 is expressed as:

[0041] S1=sin(2πft+πSt·t)

[0042] The received signal S2 is expressed as:

[0043] S2=sin[2πf(t-τ)+πS(t-τ) 2 ]

[0044] After passing through the mixer and low-pass filter, the output intermediate frequency signal S is:

[0045]

[0046] Perform one-dimensional Fourier transform on the above formula to obtain the frequency of the intermediate frequency signal f IF , let the distance from the gesture target to the radar be d, and the speed of light be c, the formula is:

[0047]

[0048] By repeatedly performing the same processing on multiple linear frequency modulation pulses (chirps) using the above method and then splicing the processed signals into a frame of data, the RF signal from the radar can be obtained.

[0049] Step 2: Preprocess the collected image information and RF signal

[0050] For the hand image captured by the camera, since the image stored in the camera is a GBR image, it needs to be converted into an RGB image through an algorithm to facilitate the subsequent marking of the key points of the hand in the image; for the radio frequency signal, a feature extraction method based on the frequency domain is used. Since the radio frequency signal is a complex number, each pixel in this mapping has a real component and an imaginary component. Therefore, the complex time domain signal can be transformed into the frequency domain through the Fourier transform method in both the horizontal and vertical directions, and the status of each frequency component of the signal can be clearly observed, and the features can be extracted in the frequency domain. The Fourier transform of the continuous signal is

[0051]

[0052] FFT processing generates a spectrum with distinct, discrete peaks. Each peak indicates the presence of an object at a specific distance. Further, the phase of each valid data point at the same distance is subjected to another FFT, distinguishing multiple targets of varying speeds at the same distance. After performing the phase FFT, the phase differences ω1 and ω2 for each target are obtained, and targets of varying speeds are identified. This yields a hand feature map, the range-Doppler map (RDI). A horizontal heat map is the projection of the signal reflection onto a plane parallel to the ground, while a vertical heat map is the projection of the signal reflection onto a plane perpendicular to the ground.

[0053] Step 3: Build a teacher network using image information

[0054] like Figure 2As shown in the figure, after the image acquisition is completed, it is input into the teacher network to obtain the position information of the key points of the hand. The teacher network is mainly built based on Google's MediaPipe Hands model. After importing the MediaPipe Hands model, the relevant parameters are set, the input data type is set to continuous static images, the confidence threshold of the model is set to 0.5, and the tracking threshold of the model is set to 0.5. After the parameters are set, the image data containing continuous hand movements is input to the model. The model can process the hand information from the image, capture the hand information in the image, and automatically mark each key point of the hand based on the obtained image information. At the same time, the pixel position of each key point relative to the image can be obtained. The processed image is shown as follows Figure 3 As shown in the figure, it contains the position information of 21 key points of the hand and marks them. After processing, a confidence map of the hand key points can be obtained from the video. The teacher network built based on the MediaPipeHands model can locate the 21 key horizontal and vertical coordinates of the hand, using these coordinates as labels to supervise the learning of the RF network (student network) and test the prediction effect of the RF network.

[0055] Step 4: Build a student network using RF information

[0056] like Figure 2 As shown, for the processing of RF signals, a horizontal heat map H will be obtained. l and vertical heatmap H v The encoding is performed through the RF coding network. Each coding network takes 100 frames (3.3 seconds) of RF heat map as input. The RF coding network uses 10 layers of 9×5×5 spatiotemporal convolution, and the step size of each layer is 1×2×2. Batch normalization is used after the input is completed. In order to solve the problem that after the input data, the neural network multiplies the first-order derivative of the activation function when performing directional error propagation, and the gradient decays each time it is passed. When the number of network layers is large, the gradient G will continue to decay until it disappears. After each layer, the ReLU activation function f(x)=max(0,x) is used. After the encoding is completed, it is input into CNN. There are different convolutional layers in CNN. The feature map is input into different convolutional layers to realize feature extraction. The encoded heat map information is then decoded through the RF decoding network. The decoding network has 5 layers, except for the last layer with a step size of Except for , the stride lengths of other layers are Furthermore, ReLu functions are used after each layer, and the final layer uses a sigmoid function as the output layer. During the training of the student network, a complex-valued RF heatmap is represented by two real-valued channels storing the real and imaginary parts. The entire network is implemented using PyTorch. RF information undergoes an encoding and decoding process to generate keypoint confidence maps derived from the RF.

[0057] Step 5: Construct a teacher-student network using cross-supervised learning.

[0058] Using synchronized images and RF signals as a bridge, we track the key points of the hand. The image information and RF signals are input into the teacher network and the student network respectively. The goal of the student network is to minimize the difference between its prediction and the teacher network's prediction. In this paper, the loss is defined as the sum of the binary cross entropy losses of each pixel in the confidence map:

[0059]

[0060] in and is the confidence of each pixel on the confidence map c. The student network receives the keypoint confidence maps labeled by the teacher network and compares them with its own predicted keypoint confidence maps. These keypoint confidence maps from the teacher network provide cross-modal supervision for the student network, allowing the student network to learn from them and successfully predict keypoint confidence maps.

[0061] Step 6: The trained system uses RF signals to track key points of the hand.

[0062] After the system training is completed, the tracking of the key points of the hand only requires placing the hand in front of the radar to obtain the position coordinates of the key points of the hand, without the need for auxiliary marking using video images, that is, the tracking of the key points of the hand can be achieved only through radio frequency signals.

[0063] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed in the present invention should be included in the protection scope recorded in the claims.

Claims

1. A hand keypoint tracking method based on FMCW radar, characterized by: The method comprises the following steps: Step 1: Initialize the FMCW radar system and configure the parameters for hand information sampling, including the transmit and receive antenna pairs, number of sampling points, and sampling time. At the same time, use the camera to capture the complete hand motion trajectory. Step 2: Preprocess the acquired image information and RF signal accordingly. For image information, save the GBR image stored by the camera as an RGB image. For RF signal, perform clutter suppression first, and perform Fourier transform (FFT) in the distance-velocity dimension to form a range-Doppler heat map (RDI). Process the signal in the horizontal and vertical directions relative to the ground to form a horizontal heat map (H). l and vertical heatmap H v ; Step 3: Process the hand information in the saved image information, capture the hand information in the image, and automatically mark each key point of the hand based on the obtained information to obtain a hand key point confidence map from the video; Step 4, we will get H l and H v The RF encoding network is used to encode the information and input it into the CNN. The feature map is input into different convolutional layers of the CNN to extract the features. The encoded heat map information is then decoded by the RF decoding network to obtain the key point confidence map from the RF. Step 5: Using cross-modal learning and supervised learning methods, the image information and RF information of different modalities are interacted. The network that obtains the video keypoint confidence map is called the teacher network, and the network that obtains the RF keypoint confidence map is called the student network. A cross-supervised learning teacher-student network is constructed. The reliability of the keypoint confidence map obtained by RF signal processing is tested through this network. Through cross-supervised learning, the position of the hand keypoints obtained in the RF signal is identified and tracked. Step 6: After the training is completed, the system can track the key points of the hand using only the radio frequency signal without the assistance of video images.

2. The method for tracking key points of a hand based on FMCW radar according to claim 1, wherein: In step 1, the original signal of the dynamic gesture is collected by the FMCW radar. Let the period of each frequency-modulated continuous pulse chirp be t, S be the frequency growth slope, τ be the delay from the radar to the gesture and then back, and f be the radar carrier frequency. The radar's transmitted signal S1 is expressed as: S1=sin(2πft+πSt·t) The received signal S2 is expressed as: S2=sin[2πf(t-τ)+πS(t-τ) 2 ] After passing through the mixer and low-pass filter, the output intermediate frequency signal S is: Perform one-dimensional Fourier transform on the above formula to obtain the frequency of the intermediate frequency signal f IF , assuming the distance from the gesture target to the radar is d and the speed of light is c, the formula is: The above method is used to repeatedly perform the same processing on multiple linear frequency modulation pulses chirp, and then the processed signals are spliced into a frame of data to obtain the RF signal from the radar.

3. The method for tracking key points of a hand based on FMCW radar according to claim 1, wherein: In step 2, the hand image captured by the camera is converted into a corresponding 200×200 RGB image for storage. A frequency-domain feature extraction method is used for the RF signal. The complex time-domain signal is transformed into the frequency domain using the Fourier transform method in both the horizontal and vertical directions. The frequency components of the signal are observed, and features are extracted in the frequency domain. The Fourier transform of a continuous signal is Through FFT processing, a spectrum with different separated peaks is generated, and each peak indicates the existence of an object at a specific distance; further, the phase of each valid data at the same distance is taken to perform FFT again to distinguish multiple targets with different speeds at the same distance; then after performing phase FFT, the phase difference ω1 and ω2 of each target are obtained, and then the targets with different speeds are obtained. The hand feature map obtained at this time is the range-Doppler map RDI; save one horizontal and one vertical antenna array, the horizontal heat map is the projection of the signal reflection on the plane parallel to the ground, and the vertical heat map is the projection of the signal reflection on the plane perpendicular to the ground.

4. The method for tracking key points of a hand based on FMCW radar according to claim 1, wherein: In step 4, the radio frequency coding network uses 10 layers of 9×5×5 spatiotemporal convolutions, with a stride of 1×2×2 per layer, and batch normalization is used after the input is completed; After each layer, the ReLU activation function f(x)=max(0,x) is used, and after encoding, it is input into CNN.

5. The method for tracking key points of a hand based on FMCW radar according to claim 1, wherein: In step 4, the RF decoding network decodes the encoded heat map information. The decoding network has 5 layers, except for the last layer with a step size of Except for , the stride lengths of other layers are In addition, the ReLu function is used after each layer, and for the last layer, the sigmoid function is used as the output layer.

6. The method for tracking key points of a hand based on FMCW radar according to claim 1, characterized in that: In step 5, the image information and RF signal are input into the teacher network and the student network respectively. The student network receives the key point confidence map marked by the teacher network and compares it with its own predicted key point confidence map. These key point confidence maps from the teacher network provide cross-modal supervision for the student network, allowing the student network to learn from them and successfully predict the key point confidence map.

7. The method for tracking key points of a hand based on FMCW radar according to claim 6, characterized in that: In step 5, the goal of training the student network is to minimize the difference between its prediction and the teacher network prediction, and the loss is defined as the sum of the binary cross entropy losses of each pixel in the confidence map: in and is the confidence of each pixel on the confidence map c; the student network accepts the key point confidence map marked by the teacher network and compares it with the key point confidence map predicted by itself. These key point confidence maps from the teacher network provide cross-modal supervision for the student network, allowing the student network to learn from them and successfully predict the key point confidence map.

8. The method for tracking key points of a hand based on FMCW radar according to claim 1, characterized in that: In step 6, after training is completed, tracking of the hand key points only requires placing the hand in front of the radar to obtain the position coordinates of the hand key points, without the need for auxiliary marking using video images, that is, tracking of the hand key points can be achieved only through radio frequency signals.

Images