Finger fine motion identification method and system based on data fusion

Abstract

The invention belongs to the technical field of information processing and human-computer interaction, and particularly relates to a finger fine action recognition method and system based on data fusion. The method comprises the following steps: acquiring an echo signal and a video stream of the same finger action through a millimeter wave radar and an optical camera; by processing echo signals and video streams, detection and positioning of all fingers are achieved, and a one-dimensional time sequence capable of reflecting micrometric displacement is constructed; processing the time sequence, and extracting a micro-motion feature component associated with the fine motion of the finger; and finally, performing data fusion on the finger positioning information obtained by the video and the micro-motion feature obtained by the radar, accurately identifying the moving finger based on the fusion result, and calculating the accurate displacement and speed of the moving finger, thereby realizing high-precision finger fine motion identification. According to the method, the advantages of high spatial resolution of optical videos and high speed resolution of radars are combined, the limitation of a single sensor is effectively overcome, and the accuracy and robustness of finger recognition are improved.

Description

Technical Field

[0001] This invention belongs to the field of information processing and human-computer interaction technology, specifically relating to a method and system for fine finger movement recognition based on data fusion. Background Technology

[0002] With the development of immersive interaction and intelligent control technologies, high-precision finger motion recognition has become a core requirement for achieving natural human-computer interaction. Currently, mainstream finger motion recognition technologies are mainly divided into two categories, but each has obvious limitations and cannot meet the application needs in complex scenarios.

[0003] Optical video-based methods capture hand images using a camera and rely on computer vision and deep learning algorithms (such as CNN, YOLO, Mask R-CNN, etc.) to detect finger joints and movement trajectories. The advantage of this method is its high spatial resolution, allowing for intuitive differentiation of different finger movements; however, it has significant drawbacks: first, it is sensitive to lighting conditions, with recognition accuracy dropping sharply in dim, bright, or complex lighting environments; second, it is susceptible to occlusion, with tracking interruptions occurring when fingers are crossed or obstructed by objects; third, obtaining 3D motion parameters requires binocular or multi-view cameras, increasing system complexity and cost; and fourth, there is a risk of privacy breaches, as video capture may involve the security of users' personal information.

[0004] Millimeter-wave radar-based methods transmit continuously frequency modulated waveforms (FMCW) in the 77-81 GHz band (commonly used in automotive / industrial applications) to analyze the micro-Doppler effect and phase changes of hand-reflected echoes, enabling the detection of minute movements. Its core advantages are strong anti-interference capabilities, unaffected by lighting or obstructions, and the ability to accurately measure radial micro-displacement (with sub-millimeter accuracy) and velocity using phase information. However, its drawback lies in low spatial resolution, making it difficult to distinguish specific fingers corresponding to dense scattering points on the hand, easily generating "ghosting points," and failing to achieve stable tracking and identification of specific fingers.

[0005] In existing technologies, some solutions attempt to combine two technologies, but most remain at the level of "simple stitching," lacking precise spatiotemporal correlation mechanisms and deep feature fusion strategies. This fails to effectively address the core contradictions of radar's inability to distinguish fingers and video's poor anti-interference capabilities. Therefore, how to construct a fine finger movement recognition solution that balances high spatial resolution, high speed accuracy, and strong environmental robustness has become a critical issue that urgently needs to be addressed in the current technological field. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a method and system for fine finger movement recognition that integrates millimeter-wave radar and optical video information. Through deep collaboration between millimeter-wave radar and optical video, it achieves the dual objectives of "accurately distinguishing finger identities and precisely quantifying motion parameters".

[0007] The technical solution adopted in this invention is:

[0008] A method for recognizing fine finger movements based on data fusion, comprising:

[0009] S1. Synchronous Data Acquisition: Simultaneously acquire millimeter-wave radar echo signals and optical video data of the target finger movements using a high-precision clock synchronization signal; specifically including:

[0010] S11. Using a millimeter-wave radar module, a continuous frequency modulated waveform (FMCW) signal is transmitted at a preset pulse repetition frequency, and the echo signal reflected by the target finger is received in real time. The echo signal contains the target's distance and speed information.

[0011] S12. Using an optical camera module, at a preset frame rate High-resolution video sequence frames containing the target finger movements are captured to obtain optical video data;

[0012] S13. The application synchronization module sends out a unified and precise clock signal to ensure that the acquisition timing of the original analog-to-digital conversion (ADC) data of the millimeter-wave radar echo signal is strictly aligned with the capture timing of each frame of the optical video data, achieving microsecond-level time synchronization, thereby providing an accurate time reference for subsequent data fusion.

[0013] S2. Video Data Processing: Utilizing computer vision algorithms to process the optical video data to detect, identify, and accurately locate the real-time dynamic spatial position of at least one finger; specifically including:

[0014] S21. Data Preprocessing: Denoise the input video frame by frame, perform gamma correction, color normalization, and crop the region of interest (ROI) to unify the resolution. ;

[0015] S22, Finger Detection and Key Point Extraction: This involves processing the pre-processed video frames... Input a pre-trained deep network (such as an improved version of YOLO / Mask R-CNN / FPN-Transformer) to output the 2D pixel coordinates, bounding box, and keypoint set for each finger. and confidence level ;

[0016] S23, Pixel-to-Camera Coordinate Transformation (using depth or depth estimation algorithm):

[0017] ,

[0018] in For the camera intrinsic parameter matrix, The depth of the corresponding pixel (provided by the depth estimation module if it is a monocular camera);

[0019] S3. Radar Echo Processing: This involves signal processing and analysis of millimeter-wave radar echo signals to extract micro-motion characteristics representing finger movements; specifically including:

[0020] S31. Original Sampling and Fast Time FFT (Range-FFT):

[0021] Fast time sampling for each Chirp Perform a discrete Fourier transform to obtain the distance spectrum:

[0022] ,

[0023] in, It is an index along the frequency dimension, from which the distance-mid-frequency plot is obtained;

[0024] S32. Range Cell Detection: Detect peaks and determine target range cell indices using the CFAR method on the range spectrum. The CFAR threshold T is expressed as:

[0025] ,

[0026] in Neighborhood noise estimation Settings and false alarm rate Related;

[0027] S33. Construction of slow-time complex sequences:

[0028] In the following On continuous chirp (slow time sampling), at the distance cell Extract the corresponding complex frequency domain value. This constitutes a complex time series:

[0029] ;

[0030] S34. Phase extraction and phase-displacement relationship:

[0031] For each complex number Phase extraction yields a micro-motion time series signal:

[0032] ,

[0033] Based on the phase change, the radial displacement of the finger is directly calculated. The relationship between the phase change and the radial displacement is as follows:

[0034] ,

[0035] in, For radar wavelength, Let be the change in distance between the target and the sensor, and thus the continuous form of the instantaneous radial velocity is:

[0036] ;

[0037] S35. Phase Difference Discrete Velocity Estimation and Phase Unwrapping:

[0038] The phase values ​​of the complex echo signals at adjacent slow-time sampling points are extracted to obtain the phase sequence. By differentiating the phases of adjacent sampling points and combining this with an unwrap operation, the discrete radial velocity estimate of the target in the slow time dimension is obtained. The calculation method is as follows:

[0039] ,

[0040] in For phase unwinding operation, The time interval between adjacent Chirp segments;

[0041] S36. Preprocessing and Filtering:

[0042] For slow-time phase sequences, detrending, windowing (e.g., Hanning window) and bandpass filtering are first performed to remove low-frequency overall motion and high-frequency noise; short-time FFT, spectral estimation or autoregressive methods can be used to improve the accuracy of rate estimation under noise; the spatial location information from the video end and the micro-motion features extracted by the radar are spatiotemporally registered, mapped and fused to accurately identify finger categories and calculate their fine motion parameters;

[0043] S4. Fusion and recognition steps:

[0044] S41. Timing Alignment: Establishing radar slow-time samples using synchronization timestamps. With video frame time mapping relationship Time interpolation or resampling is performed on both sequences to ensure matching sampling rates;

[0045] S42. Spatial Mapping and Overlap Calculation:

[0046] The transformation matrix from the visual coordinate system to the radar coordinate system is obtained through calibration. The finger position in the video is projected onto the radar polar coordinate (range, angle) domain; the coverage overlap (intersection over union) is calculated. To measure the video detection area With radar detection area Overlap in the distance-angle domain:

[0047] ;

[0048] S43. Time Correlation Measurement: Calculating the Time Correlation Coefficient (Pearson) to measure video velocity sequences With radar velocity sequence Similarity within the aligned time interval:

[0049] ;

[0050] in, The standard deviation represents the sequence of finger movement speeds in video data, used to characterize the dispersion of finger movement amplitude in the video signal; The standard deviation of the radar micro-motion velocity sequence is used to characterize the fluctuation range of the finger micro-motion magnitude in the radar signal. Both are obtained by statistical calculation of the velocity sequence within their respective time windows; if and If they are from the same finger, then it is determined that they are from the same finger. , The set threshold;

[0051] S44. Multimodal Feature Construction:

[0052] Constructing fused feature vectors :

[0053] ,

[0054] in,

[0055] ;

[0056] in, This is a feature vector of finger movements constructed based on video data. This is a temporal coordinate sequence of key finger points, used to describe the finger trajectory. The macroscopic motion speed of the video is calculated based on the changes in the position of key finger points. This is a pose descriptor extracted based on finger pose, used to represent the relative pose information of the finger during movement;

[0057] ;

[0058] in, This is a feature vector of finger micro-movements constructed based on radar signals. The phase change rate, calculated from radar phase changes, is used to characterize the speed of finger micro-movements. The energy of the intrinsic mode function (IMF) obtained after empirical mode decomposition of radar echoes is used to characterize the energy distribution of micro-motions. The finger micro-movement velocity is obtained based on Doppler frequency shift estimation;

[0059] S45. Classification and Recognition: Integrating Features Input a classifier (such as LSTM, Temporal Convolutional Network, or Transformer-based classifier) ​​to determine the action category (such as "tapping", "sliding", "grabbing", etc.), and output the action label and confidence score. ;

[0060] S46. Output format:

[0061] For each identified finger ID, the system simultaneously outputs: finger ID, action category, and instantaneous speed. Cumulative displacement Confidence level ;

[0062] S47. Precise Motion Parameter Extraction:

[0063] Based on the two-dimensional / three-dimensional macroscopic motion information of the finger obtained from the video and the phase micro-motion characteristics of the millimeter-wave radar, precise motion parameters that can characterize the instantaneous motion state of the target finger are obtained through time alignment, spatial mapping, and data fusion, including at least:

[0064] (1) Instantaneous velocity;

[0065] (2) Short-time cumulative displacement;

[0066] (3) The three-dimensional velocity vector after fusing the video motion component and the radar micro-motion component;

[0067] (4) High-precision radial displacement obtained by phase unwrapping.

[0068] The above parameters are calculated by combining the macroscopic finger movement speed obtained from video analysis and the radial minute velocity component derived from radar micro-motion phase information, and then using a fusion algorithm to calculate the instantaneous three-dimensional velocity vector of the finger; by integrating the real-time velocity or directly deriving the cumulative displacement of the finger in a short period of time from the radar phase change, the precise displacement of the finger in space is obtained.

[0069] Furthermore, the specific implementation process of step S22 is as follows:

[0070] S221. Data Preprocessing: The optical video frames are preprocessed by image denoising, enhancement, and grayscale conversion. The image size is unified and normalized to ensure data consistency during subsequent deep neural network processing. This process uses bilateral filtering and histogram equalization techniques to significantly improve image quality.

[0071] S222, Feature Extraction: Extracting features from the preprocessed video frames. The input is a pre-trained deep neural network to extract finger features and effectively distinguish the background; the deep neural network automatically learns and extracts spatial features from video frames through multi-scale convolution operations.

[0072] S223, Region Proposal and Classification: The deep network generates potential finger region proposals based on the extracted features, and a classifier determines whether these proposals are fingers; for each detected finger, a confidence score is assigned to each finger. This ensures that high-confidence finger areas are prioritized for identification;

[0073] S224. Precise localization: For the region identified as a finger, the network further outputs its precise bounding box coordinates, or obtains the contour information of each finger through instance segmentation to precisely locate the spatial position of the finger.

[0074] S225. Motion State Judgment: Based on the changes in the positioning coordinates of the same finger or the key points of the skeleton in consecutive frames, calculate the displacement and velocity vectors of the finger on the image plane, and combine them with a preset motion threshold. To determine whether the finger is stationary or moving, further determine the direction of its movement.

[0075] A fine finger movement recognition system based on data fusion includes:

[0076] A millimeter-wave radar module that collects millimeter-wave radar echo signals of the target's finger movements;

[0077] An optical camera module that captures optical video data of the target finger movements;

[0078] A clock synchronization module for synchronously triggering the millimeter-wave radar module and the optical camera module;

[0079] A data processing module is configured to perform a data fusion-based method for recognizing fine finger movements.

[0080] The beneficial effects of this invention are as follows:

[0081] 1) Strong robustness: The radar's resistance to light and occlusion compensates for the environmental sensitivity of the video. The high spatial resolution of the video solves the problem of the radar "not being able to distinguish fingers". The recognition accuracy is still ≥90% in dim and occluded scenes.

[0082] 2) High precision: Radar phase information enables sub-millimeter displacement and 0.1mm / s velocity measurement, while video provides precise finger identification association. The fusion of the two achieves dual precise output of "identity + parameters".

[0083] 3) Good real-time performance: It adopts a lightweight deep learning model (YOLOv8-tiny) and efficient signal processing algorithms (FFT, CFAR), with an end-to-end latency of ≤50ms, which meets the real-time requirements of interactive scenarios;

[0084] 4) Controllable cost: No need for multi-view cameras or high-end radar, using industrial-grade general-purpose hardware, the system cost is 30% lower than similar high-precision solutions. Attached Figure Description

[0085] Figure 1 This is a hardware structure block diagram of a fine finger movement recognition system provided in an embodiment of the present invention.

[0086] Figure 2 This is an overall flowchart of a method for recognizing fine finger movements provided in an embodiment of the present invention.

[0087] Figure 3 This is a schematic diagram of the feature fusion process in an embodiment of the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this application clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0089] Example:

[0090] This example uses the following four main hardware modules, such as Figure 1 As shown, the details are as follows:

[0091] Millimeter-wave radar module: adopts automotive / industrial grade radar (such as TI IWR6843), operates at a frequency of 77-81GHz, has a chirp bandwidth of 1-4GHz, supports 10-50kHz PRF, and outputs raw ADC data;

[0092] Optical camera module: Uses a USB HD camera or depth camera (such as Intel RealSense), with a frame rate of 30-60fps, a resolution of 1280×720 to 1920×1080, and supports RGB or RGBD data output;

[0093] Clock synchronization module: Uses a high-precision clock card (such as NI PXI-6602) with a synchronization accuracy of ±50ns, and outputs a unified trigger signal to the radar and camera to ensure timing alignment;

[0094] Data processing module: Employs an embedded processor (such as NVIDIA Jetson AGX) or PC to run preprocessing, feature extraction, and fusion recognition algorithms, achieving real-time data processing and result output (latency ≤50ms).

[0095] In this example, the fine motor finger recognition method includes four key steps, each building upon the previous one to form a complete multimodal fusion recognition chain, such as... Figure 2 As shown:

[0096] 1. Synchronous data acquisition:

[0097] Radar startup: Set the number of sampling points per chirp to 512, the sampling rate to 150MHz, and the precision to 16-bit. Continuously transmit FMCW signals, receive finger reflection echoes, and store the raw ADC data.

[0098] Camera startup: Captures hand video at 30fps, with each frame corresponding to a synchronization timestamp;

[0099] Timing verification: The radar chirp trigger signal and the camera frame trigger signal were observed with an oscilloscope to confirm that the time deviation was ≤ ±1μs.

[0100] A high-precision clock synchronization module is used to achieve microsecond-level timing alignment between millimeter-wave radar echo signals and optical video data, providing a time reference for subsequent fusion. Specifically, this includes:

[0101] Millimeter-wave radar module: Transmits FMCW signals at a pulse repetition frequency (PRF) of 10-50kHz, receives echo signals reflected from fingers in real time, and obtains raw analog signals containing distance and speed information;

[0102] Optical camera module: Captures hand video at 30-60fps frame rate and 1280×720 to 1920×1080 resolution, and outputs image sequences including the spatial position of the fingers;

[0103] Clock synchronization module (e.g., NI PXI-6602): Emits a unified clock signal with a period of 10-20ms and an accuracy of ±50ns to strictly align the radar ADC sampling timing with the camera frame capture timing, ensuring that the timestamp deviation between the two is ≤±1μs (i.e., , The acquisition time for the nth radar chirp is... (The time for capturing the nth frame of video).

[0104] 2. Video data processing steps:

[0105] Using improved computer vision algorithms, high-precision finger positioning and preliminary motion state determination are achieved, specifically including:

[0106] Data preprocessing: Denoising (bilateral filtering, 3×3 convolution kernel), gamma correction (γ=1.2-1.5), color normalization (RGB channels normalized to [0,1]), and region of interest (ROI) cropping (only the hand area is retained) are performed on the video frames in sequence to unify the image resolution to 640×640 pixels;

[0107] Finger detection and keypoint extraction: Preprocessed frames are input into an improved YOLOv8 model (training dataset: MSCOCO Hand+RHD, 150,000 samples, 150 iterations), outputting the two-dimensional pixel coordinates of each finger. ), bounding box ( Key skeleton points (≥5 points per finger, such as fingertips and knuckles) and confidence level (threshold set to 0.5 to filter low confidence results);

[0108] Coordinate transformation (optional): If using a depth camera or monocular depth estimation algorithm, transform the coordinates through the camera intrinsic matrix. ( Focal length Convert pixel coordinates to 3D camera coordinates (primary point coordinates). The formula is: , ( (pixel depth)

[0109] Motion state determination: Calculate the displacement vector of the same finger in consecutive frames (e.g., frame k and frame (k+1)). ,speed ( This is combined with a speed threshold (e.g., 1 pixel / frame) to determine whether the finger is stationary or moving, and the direction angle is also used. To determine the direction of motion, specifically, calculate the displacement of the index fingertip in adjacent frames (33.3ms interval). Pixels, speed Pixels per second (approximately 0.2 m / s) is considered motion, with the direction angle... (Horizontally to the upper right).

[0110] 3. Radar echo processing:

[0111] The raw radar signal is processed in layers to extract the micro-motion features of the finger (phase, radial displacement, velocity), specifically including:

[0112] Raw Sampling and Fast-Time FFT: The raw data acquired by the radar ADC (sampling rate 100-200MHz, 16-bit precision, 256-1024 points per chirp) is subjected to an FFT along the fast-time dimension (within a single chirp) (the number of points is the same as the number of sampling points, with a Hanning window to suppress spectral leakage). Specifically, a 512-point FFT is performed on each chirp of ADC data after applying a Hanning window, yielding the range-IF signal with a range resolution of [missing information]. cm; obtain distance-intermediate frequency signal ( For distance cell index, For Chirp serial number, (For Hanning window);

[0113] Distance cell detection: The constant false alarm rate (CFAR) algorithm (CA-CFAR, 16-32 reference cells, 4-8 protection cells) is employed. Detect the peak value of the range spectrum to determine the index of the range cell where the hand target is located. (corresponding distance) m), the CFAR threshold formula is ( A threshold factor related to the false alarm rate. (This is an estimate of the neighborhood noise).

[0114] Construction of slow-time complex sequences: from 64 consecutive chirps (N=32-128, slow time length) (distance unit) Extracting complex frequency domain signals , forming a complex time series ;

[0115] Phase Extraction and Unwrapping: Calculating Phase And untangle, and obtain radial displacement Instantaneous speed (This is the instantaneous speed of the simulated striking action);

[0116] Micro-motion parameter calculation: based on radar phase-range relationship ( For radar wavelength, At the speed of light, (for the carrier frequency), deriving the radial micro-displacement. Instantaneous radial velocity ( (Chirp interval);

[0117] Filtering optimization: Detrend and bandpass filtering (1-10Hz, retaining the frequency band of finger micro-movement) are performed on the phase sequence to remove low-frequency interference such as arm swing and high-frequency noise.

[0118] 4. Fusion Recognition Steps

[0119] like Figure 3 As shown, by combining spatiotemporal correlation and multimodal feature fusion, accurate matching and action recognition of "finger identity - micro-motion features" are achieved, specifically including:

[0120] Timing alignment: Based on the synchronization timestamp, interpolation or resampling is performed on the radar slow time series (sampling rate = PRF) and the video frame series (sampling rate = frame rate) to ensure that the sampling rates of the two are consistent (e.g., uniformly 30Hz).

[0121] Spatial mapping and overlap calculation: The transformation matrix from the visual coordinate system to the radar coordinate system is obtained through a calibration board (20mm×20mm checkerboard). The video finger bounding box is then mapped to the radar range-angle domain to obtain the video coverage area. ; Calculate its relationship with the radar micro-motion area The Intersection over Union (IOU) ratio is given by the formula: In this example, IOU = 0.65;

[0122] Temporal correlation verification: Calculate video velocity sequence With radar velocity sequence Pearson correlation coefficient ,like and To determine that both originated from the same finger, the Pearson correlation coefficient in this case... Determine if a match is found;

[0123] Multimodal feature construction: Constructing fused feature vectors ,in For video features (finger shape features, , ), Radar characteristics ( Phase change rate Vibration frequency In this example ;

[0124] Action recognition: Input TCN classifier, output "index finger, action: tapping, confidence: 0.94, instantaneous velocity: 3.65mm / s, cumulative displacement: 11.69mm";

[0125] Action classification and parameter output: Input a classifier (such as LSTM, Temporal Convolutional Network (TCN), or Transformer-based classifier) ​​to identify action categories ("tapping", "sliding", "grabbing", "bending", etc., with a classification accuracy of ≥95%). Simultaneously output the finger ID, instantaneous velocity (accuracy ≤±0.1mm / s), cumulative displacement (accuracy ≤±0.05mm), and confidence score. In this example, the input is a TCN classifier, and the output is "index finger, action: tapping, confidence score: 0.94, instantaneous velocity: 3.65mm / s, cumulative displacement: 11.69mm".

Claims

1. A finger fine motion recognition method based on data fusion, characterized in that, Comprise: S1, synchronous data acquisition: through high-precision clock synchronization signal, synchronous acquisition of target finger action millimeter wave radar echo signal and optical video data; Specifically comprising: S11, using millimeter wave radar module, emitting continuous frequency modulation waveform (FMCW) signal with preset pulse repetition frequency, and receiving echo signal reflected by target finger in real time, the echo signal contains the distance, speed original information of target; S12, using an optical camera module to capture high-definition video sequence frames containing target finger actions at a preset frame rate and resolution, to obtain optical video data; S13, application synchronization module sends uniform and accurate clock signal, ensures that the original analog-digital conversion (ADC) data acquisition timing of millimeter wave radar echo signal is strictly aligned with the timing of each frame image capture of optical video data, realizes microsecond level time synchronization, thereby providing accurate time reference for subsequent data fusion; S2, video data processing: using computer vision algorithm to process the optical video data to detect, identify and accurately locate the real-time dynamic spatial position of at least one finger; Specifically comprising: S21, data preprocessing: denoising, gamma correction, color normalization and cropping the region of interest (ROI) for each frame of the input video, and uniform resolution ; S22, finger detection and key point extraction: input the pre-processed video frame into the pre-trained deep network to output the two-dimensional pixel coordinates of each finger, the bounding box and the key point set and the confidence ; S23, pixel to camera coordinate transformation: ， wherein is a camera intrinsic matrix, is a corresponding pixel point depth; S3, radar echo processing: signal processing and analysis on millimeter wave radar echo signal to extract micro-motion features representing finger movement; Specifically comprising: S31, original sampling and fast time FFT: Fast time sampling of each chirp Discrete Fourier transform to get range profile ， wherein is an index in the frequency dimension, thereby obtaining a range-mid-frequency plot; S32, Distance Cell Detection: CFAR method is used to detect the peak on the range profile and determine the target range cell index CFAR threshold T is expressed as: ， wherein neighborhood noise estimate, set in relation to a false alarm rate related; S33, slow time complex sequence construction: In a subsequent step On a continuous slow-time sampling, the corresponding complex frequency domain values are extracted at the distance unit , constituting a complex time series: ； S34, phase extraction and phase-displacement relationship: for each complex number extracting the phase to obtain the microseismic time series signal: ， Based on phase change, directly calculate the radial displacement of finger, the relationship between phase change and radial displacement is: ， wherein, is the radar wavelength, is the change in the distance between the target and the sensor, whereby the continuous form of the instantaneous radial velocity is: ； S35, phase difference discrete velocity estimation and phase unwrapping: The phase values of the complex echo signals at adjacent slow-time sampling points are extracted to obtain a phase sequence By differentiating the phases of adjacent sampling points and combining an unwrapping operation, discrete radial velocity estimation values of the target in the slow-time dimension are obtained, and the calculation method is as follows: ， wherein is a phase unwrapping operation, is a neighboring Chirp time interval; S36, pretreatment and filtering: The slow time phase sequence is first de-trended, windowed and band-pass filtered to remove low-frequency overall motion and high-frequency noise; The spatial position information at the video end and the micro-motion features extracted by radar are time and space registered, mapped and fused to accurately identify the finger category and calculate its fine motion parameters; S4, fusion recognition step: S41, timing alignment: utilize synchronized timestamps to establish radar slow-time samples mapping relationship with video frame time time interpolation or resampling to ensure sampling rate matching;​ S42, spatial mapping and overlap calculation: The transformation matrix from the vision coordinate system to the radar coordinate system is obtained by calibration The finger position in the video is projected to the radar polar coordinate domain; the coverage overlap degree is calculated to measure the video detection area and the radar detection area the coincidence degree in the range-angle domain: ； S43, Time correlation measure: compute time correlation coefficient to measure video speed sequence with radar speed sequence similarity within aligned time interval: ； wherein, represents the standard deviation of the sequence of finger motion velocity in the video data, used to represent the dispersion degree of the amplitude of finger motion in the video signal; represents the standard deviation of the sequence of radar micro-motion velocity, used to represent the fluctuation range of the order of magnitude of finger micro-motion in the radar signal, both of which are obtained by statistical calculation on the sequence of velocity in the respective time window; if and , then it is determined that both come from the same finger, wherein , is a set threshold value; S44, multi-modal feature construction: Constructing a fusion feature vector : ， Wherein, ； wherein, is a finger motion video feature vector constructed based on the video data, is a time series of coordinates of the finger key points, used to describe the finger trajectory, is a video macro-motion speed calculated according to the change of the finger key point positions, is a pose descriptor extracted based on the finger pose, used to represent the relative pose information of the finger in the motion process; ； wherein, is a finger micro-motion feature vector constructed based on radar signals, is a phase change rate calculated from the radar phase change, used to represent the finger micro-motion speed, is an intrinsic mode function (IMF) energy of the radar echo after empirical mode decomposition, used to represent the micro-motion energy distribution, is a finger micro-motion speed estimated based on the Doppler shift; S45, classification and recognition: the fusion features are classified and recognized The input classifier determines the action category, and outputs the action label and confidence ; S46, output format: For the recognized finger ID, the system simultaneously outputs: finger ID, motion class, instantaneous speed , cumulative displacement , confidence ; S47, accurate motion parameter extraction: Based on the finger two-dimensional / three-dimensional macro-motion information obtained at the video end and the millimeter wave radar phase micro-motion features, through time alignment, spatial mapping and data fusion, the accurate motion parameters capable of representing the instantaneous motion state of target finger are obtained, at least including: (1) instantaneous velocity; (2) short-time cumulative displacement; (3) three-dimensional velocity vector fused by video motion component and radar micro-motion component; (4) high-precision radial displacement obtained by phase unwrapping. 2.The fine finger motion recognition method based on data fusion according to claim 1, characterized in that, The specific implementation process of step S22 is: S221, data preprocessing: image denoising, enhancement, grayscale preprocessing operation is performed on the optical video frame, the image size is unified and normalized, and the data consistency during subsequent deep neural network processing is ensured; This process uses bilateral filtering and histogram equalization technology to significantly improve the image quality; S222, feature extraction: the pre-processed video frames A pre-trained deep neural network is inputted to extract finger features and effectively distinguish the background; the deep neural network automatically learns and extracts spatial features in the video frames through multi-scale convolution operation; S223, Region proposal and classification: the deep network generates potential finger region proposals according to the extracted features, and judges whether these proposals are fingers through a classifier; according to each detected finger, a confidence score is assigned to each finger , ensuring that high-confidence finger regions are identified first; S224, accurate positioning: for the region confirmed as finger, the network further outputs its accurate bounding box coordinates, or further obtains the contour information of each finger through instance segmentation, and accurately positions the spatial position of finger; S225, motion state judgment: according to the change of the same finger position coordinates or skeleton key points in the continuous frames, the displacement and speed vector of the finger on the image plane are calculated, and the pre-set motion threshold is combined to judge whether the finger is in a stationary state or motion occurs, and further judge the motion direction. , judge whether the finger is in a stationary state or motion occurs, and further judge the motion direction.

3. A fine finger motion recognition system based on data fusion, characterized by, Comprise: a millimeter wave radar module, which collects millimeter wave radar echo signals of the target finger action; an optical camera module, which collects optical video data of the target finger action; a clock synchronization module for synchronously triggering the millimeter wave radar module and the optical camera module; a data processing module configured to perform the method of any one of claims 1 or 2.

Images