Gesture recognition control method and device based on virtual reality device and medium

Abstract

The invention discloses a gesture recognition control method and device based on virtual reality equipment and a medium, and relates to the technical field of man-machine interaction, and the method comprises the steps: carrying out the dynamic weight fusion of a three-dimensional position of a hand joint point, a motion track feature muscle activation mode and a force intensity feature, and generating a gesture feature vector; inputting a residual neural network to carry out gesture classification and gesture force estimation value calculation, and recognizing gesture information; performing matching analysis on the gesture information and a historical tactile effect library to obtain a gesture feedback mode, and integrating the gesture feedback mode with physical attributes of an object to generate a tactile feedback factor; according to the method, the gesture feedback mode is coupled with the physical attribute of the current interaction object, the tactile feedback factor is generated, dynamic fusion of the tactile parameter, the rigidity coefficient, the damping coefficient and the roughness is achieved, the feedback strength is matched with the material characteristic, and the interaction reality sense is improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a gesture recognition control method, device and medium based on virtual reality devices. Background Technology

[0002] In the field of virtual reality interaction, gesture recognition, as a core technology for achieving natural human-computer interaction, has widely adopted tracking methods based on visual sensors or inertial measurement units. These methods capture the outline or trajectory of the hand to determine actions and generate corresponding control commands based on pre-defined mapping rules. Relying on mature image processing and pattern recognition theories, these methods can achieve basic gesture classification and spatial positioning, supporting object manipulation and interface navigation in virtual environments. This demonstrates the deep integration of multimodal perception and intelligent decision-making in human-computer collaboration.

[0003] However, existing technologies still have room for improvement in terms of refined analysis of movement intentions and physical consistency of haptic feedback. On the one hand, relying solely on external movement trajectories is insufficient to fully capture the temporal characteristics of muscle activation and the continuous changes in force intensity, resulting in inaccurate estimation of force intentions. On the other hand, the generation of haptic feedback parameters is mostly based on empirical rules and lacks a dynamic coupling mechanism with user physiological signals and the physical properties of interactive objects, affecting the realism and immersive experience of the feedback. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a gesture recognition control method based on virtual reality devices, which solves the problems of inaccurate estimation of force application intention due to reliance on external motion trajectories and inaccurate dynamic coupling between tactile feedback parameters and physiological signals and physical properties.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a gesture recognition and control method based on a virtual reality device, comprising, Electrical signals and micro-electromagnetic wave data of the forearm muscle group were collected and preprocessed to generate aligned multimodal data; Based on spatiotemporally aligned multimodal data, the three-dimensional coordinate set of hand joints is calculated and motion trajectory features are extracted. At the same time, muscle activation patterns and force intensity features are obtained through electromyography signal time-frequency analysis. The three-dimensional position of the hand joints, the motion trajectory features, the muscle activation patterns, and the force intensity features are dynamically weighted and fused to generate a gesture feature vector. This vector is then input into a residual neural network for gesture classification and gesture force estimation to identify gesture information. The gesture information is matched and analyzed with a historical haptic effect database to obtain gesture feedback patterns, and then integrated with the physical properties of objects to generate haptic feedback factors. Based on the tactile feedback factor, the end-to-end delay from gesture recognition to tactile feedback is calculated, and the intensity mapping curve of tactile feedback is generated by dynamic fitting of electromyography signals to obtain the gesture recognition control report.

[0007] As a preferred embodiment of the gesture recognition and control method based on virtual reality device described in this invention, the electrical signals of the forearm muscle group include surface electromyography signals, muscle activation timing characteristics, force intensity indicators, and muscle coordination patterns. The micro-electromagnetic wave data includes the range-Doppler matrix formed by the continuous wave reflection signal, the target radial velocity and range resolution, and the micro-Doppler characteristics; The preprocessing includes interference elimination, intermediate frequency filtering, dynamic range compression, and spatial coordinate transformation.

[0008] In a preferred embodiment of the gesture recognition and control method based on virtual reality devices described in this invention, the specific steps for generating aligned multimodal data are as follows: The preprocessed electrical signals and micro-electromagnetic wave data of the forearm muscle group are time-aligned using a high-precision clock to generate a unified time reference coordinate. Based on a unified time reference coordinate, a coordinate transformation matrix is ​​established, and multimodal data fusion is performed to generate spatiotemporally aligned multimodal data.

[0009] As a preferred embodiment of the gesture recognition and control method based on virtual reality devices described in this invention, the specific steps for calculating the three-dimensional coordinate set of hand joints and extracting motion trajectory features based on spatiotemporally aligned multimodal data are as follows. The multimodal data is equally segmented according to a unified threshold standard to generate multimodal data frames, and parameter detection is performed to extract valid reflection points and obtain a set of reflection points. Based on the set of reflection points, a neighborhood radius is defined, and a set of three-dimensional coordinates of hand joints is generated using the multi-scale DBSCAN clustering algorithm. Based on spatiotemporally aligned multimodal data, spatiotemporal feature joint extraction and sequence dependency learning are performed to construct a motion prediction model; Based on the three-dimensional coordinate set of hand joints, a KD tree data structure is established and input into the motion prediction model to generate joint position prediction values. Based on the predicted joint positions, a time-continuous sequence of hand joint motion trajectories is generated using the Hungarian algorithm. The instantaneous velocity and motion direction angle of each joint are calculated to generate motion trajectory features.

[0010] As a preferred embodiment of the gesture recognition control method based on virtual reality devices described in this invention, the specific steps for obtaining muscle activation patterns and force intensity characteristics through electromyography signal time-frequency analysis are as follows: Multimodal data is segmented using a sliding window to extract muscle subband energy features, and time-domain and frequency-domain analyses are performed to generate electromyographic feature vectors. Muscle co-element decomposition is performed on the electromyographic feature vectors to generate a co-activation weight matrix, and the temporal and spatial distribution entropy values ​​of co-activation are calculated to obtain the muscle activation pattern. Based on muscle activation patterns, an electromyography-force mapping model is established. Multimodal data is input into the electromyography-force mapping model, and a sliding window weighting operation is performed to obtain force intensity features.

[0011] As a preferred embodiment of the gesture recognition and control method based on virtual reality devices described in this invention, the following steps are taken: The three-dimensional position of hand joints, motion trajectory features, muscle activation patterns, and force intensity features are dynamically weighted and fused to generate a gesture feature vector, which is then input into a residual neural network for gesture classification and gesture force estimation to recognize gesture information. The three-dimensional position of the hand joints, the motion trajectory features, muscle activation patterns, and force intensity features are subjected to feature projection operations to generate motion feature projection, electromyographic feature projection, and force feature projection. These are then stitched and fused according to feature dimensions to generate a gesture feature vector. The gesture feature vector is input into the residual neural network, convolution and normalization operations are performed to obtain residual block groups, and the data are compared and analyzed with historical gesture data to obtain gesture classification. Define the total loss function, calculate the deviation between the force regression and the true force value, and obtain the estimated value of the gesture force. Gesture classification and gesture force estimation are fused and analyzed to identify gesture information.

[0012] As a preferred embodiment of the gesture recognition and control method based on virtual reality devices described in this invention, the steps of matching and analyzing gesture information with a historical tactile effect database to obtain gesture feedback patterns, and integrating them with the physical properties of objects to generate tactile feedback factors are as follows. The gesture information is matched and analyzed with the historical haptic effect database through a dynamic matching algorithm. Candidate records belonging to the same gesture type and force range are selected, and the pattern similarity is calculated to obtain the gesture feedback pattern. The gesture feedback mode is coupled with the physical properties of the current interactive object to generate haptic feedback factors.

[0013] As a preferred embodiment of the gesture recognition control method based on virtual reality devices described in this invention, the steps include: calculating the end-to-end delay from gesture recognition to tactile feedback based on the tactile feedback factor, dynamically fitting electromyographic signals to generate a tactile feedback intensity mapping curve, and obtaining a gesture recognition control report. Record the start time of the gesture feature vector input to the residual neural network and the end time of the generation of tactile feedback factors, and calculate the end-to-end delay from gesture recognition to tactile feedback; Based on the end-to-end delay from gesture recognition to tactile feedback, dynamic energy calculation is performed to obtain a time-seriesd real-time electromyography intensity sequence, and a dynamic curve fitting operation is performed through a nonlinear regression function to generate a tactile feedback intensity mapping curve. The intensity mapping curve of haptic feedback, gesture information, and haptic feedback factors are standardized and integrated to generate a gesture recognition control report.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the gesture recognition control method based on a virtual reality device as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the gesture recognition control method based on a virtual reality device as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: by defining the neighborhood radius based on the set of reflection points and generating a set of three-dimensional coordinates of hand joints through the multi-scale DBSCAN clustering algorithm, the density clustering and spatial structure analysis of effective reflection points in micro-electromagnetic wave data are realized, and the spatial position of the hand is accurately identified; by coupling the gesture feedback mode with the physical properties of the current interactive object, a tactile feedback factor is generated, realizing the dynamic fusion of tactile parameters with stiffness coefficient, damping coefficient and roughness, so that the feedback intensity matches the material properties, and the realism of the interaction is improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a gesture recognition control method based on virtual reality devices.

[0019] Figure 2 A flowchart for generating a gesture recognition control report.

[0020] Figure 3 A flowchart for generating gesture feature vectors.

[0021] Figure 4 A flowchart for obtaining the tactile feedback intensity mapping curve. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a gesture recognition control method based on a virtual reality device, including the following steps: S1. Collect electrical signals and micro-electromagnetic wave data of the forearm muscle group and perform preprocessing to generate aligned multimodal data; The electrical signals of the forearm muscle group include surface electromyography signals, muscle activation timing characteristics, force intensity indicators, and muscle synergy patterns. Furthermore, the electrical signals of the forearm muscle groups are non-invasively acquired using multi-channel surface electromyography (sEMG) sensing. Specifically, high-sensitivity silver / silver chloride electrode pairs are symmetrically arranged on the skin surface of the target forearm muscle groups (such as the flexor carpi radialis, flexor carpi ulnaris, and brachioradialis) according to their anatomical positions. The weak electrical signals generated during muscle contraction are picked up using a bipolar lead method. After being processed by a low-noise preamplifier, bandpass filter, and power frequency notch filter, the weak electrical signals are digitized by a high-resolution analog-to-digital converter and simultaneously recorded as raw surface electromyography signals. Based on this, the muscle activation timing features are extracted through time-domain analysis, and the force intensity index (expressed as % MVC) is calculated by combining the calibrated maximum voluntary contraction test. The multi-channel signals are then decomposed using algorithms such as non-negative matrix factorization to identify muscle synergy patterns.

[0026] Micro-electromagnetic wave data includes the range-Doppler matrix formed by the continuous wave reflection signal, the target radial velocity and range resolution, and micro-Doppler characteristics; Furthermore, micro-electromagnetic wave data is acquired non-contactly using millimeter-wave radar sensors deployed near the operating area. These sensors emit continuous wave (CW) or frequency-modulated continuous wave (FMCW) electromagnetic signals (typically operating in the 24 GHz or 60 GHz band) and receive echo signals reflected from the hand and forearm muscles. A mixer coherently processes the transmitted and echo signals to extract phase and amplitude change information. A Fast Fourier Transform (FFT) is then used to perform two-dimensional processing on multiple frames of echo data, generating a range-Doppler matrix containing range and Doppler frequency information. Based on the range-Doppler matrix, a peak detection algorithm is used to calculate the radial velocity of each part of the hand relative to the sensor and the precise range resolution. Further, a Short-Time Fourier Transform (STFT) or wavelet transform is used to analyze the high-frequency modulation components in the signal caused by finger joint micro-movements and muscle tremors, extracting micro-Doppler features reflecting fine motion dynamics, thereby achieving highly sensitive and occlusion-resistant dynamic perception of hand gestures.

[0027] Preprocessing includes interference removal, intermediate frequency filtering, dynamic range compression, and spatial coordinate transformation; Furthermore, the preprocessing process performs multi-level optimization on the electrical signals and micro-electromagnetic wave data of the forearm muscle group acquired by the micro-electromagnetic wave radar: First, interference elimination is performed by removing environmental fixed reflections and noise through background clutter suppression (such as high-pass filtering or stationary target subtraction) to improve the signal-to-noise ratio; then, intermediate frequency filtering is performed, and the mixed intermediate frequency signal is filtered out by a band-pass filter (typically in the range of 100 Hz–10 kHz) to remove DC drift and high-frequency noise, retaining effective motion information; next, dynamic range compression is performed, and the dynamic range of strongly reflective targets is compressed by logarithmic transformation of the signal amplitude to enhance the visibility of weakly reflective areas (such as fingertips and tendons); finally, spatial coordinate transformation is performed, and the distance-angle data in the radar polar coordinate system is converted into a three-dimensional rectangular coordinate system consistent with the gesture motion space by combining the Tsai-Lenz calibration algorithm and the ICP point cloud registration method to ensure accurate alignment of subsequent multi-source data in the spatiotemporal dimensions.

[0028] The preprocessed electrical signals and micro-electromagnetic wave data of the forearm muscle group are time-aligned using a high-precision clock to generate a unified time reference coordinate. Furthermore, time alignment was performed on the preprocessed electrical signals and micro-electromagnetic wave data of the forearm muscle group. First, a high-precision hardware clock was used to stamp the electrical signals and micro-electromagnetic wave data of the forearm muscle group with absolute timestamps to ensure accurate recording of sampling times. The sampling time of the surface electromyography (EMG) signal of the forearm muscle group was used as the reference, and the frame start time of the range-Doppler matrix was used as the reference for the micro-electromagnetic wave data. The electrical signals and micro-electromagnetic wave data of the forearm muscle group were resampled to a unified time series using an interpolation algorithm. After resampling, the surface EMG signal, muscle activation timing characteristics, force intensity index, muscle coordination mode and range-Doppler matrix, target radial velocity, distance resolution, and micro-Doppler features at each time point were aligned according to the timestamps to ensure that the multimodal information at the same time point corresponds to the same physiological and motor state. Finally, a unified time reference coordinate was generated as the basis for subsequent multimodal data fusion.

[0029] It should be noted that a high-precision clock refers to a timing source with extremely low phase noise and high frequency stability, such as a clock signal provided by a temperature-controlled crystal oscillator or an atomic clock, whose time error is controlled within the microsecond level, and is used to provide a synchronized time reference for multi-sensor data acquisition.

[0030] Based on a unified time reference coordinate, a coordinate transformation matrix is ​​established by combining the ICP point cloud registration method with the Tsai-Lenz calibration algorithm, and multimodal data fusion is performed to generate spatiotemporally aligned multimodal data.

[0031] Furthermore, based on a unified time reference coordinate, the electrical signals of the forearm muscle group and the micro-electromagnetic wave data are registered in the spatial dimension. First, the three-dimensional spatial positions of key hand points (such as the force transmission key points of the thumb-index finger pinching action and the posture reference points of wrist rotation movement) are extracted using the distance-Doppler matrix in the micro-electromagnetic wave data to form radar point cloud data. The ICP point cloud registration method is used to perform iterative nearest-point matching on the point cloud data and calculate the optimal rigid body transformation parameters. During the registration process, the Tsai-Lenz calibration algorithm is combined to accurately calibrate the extrinsic parameters (rotation matrix and translation vector) between sensors, for example, by using known hand standards. Action sequences (such as extension-fist clenching) are used to construct a calibration reference. By minimizing the Euclidean distance error between point clouds, the coordinate transformation matrix from the micro-electromagnetic wave data coordinate system to the electrical signal derivation coordinate system of the forearm muscle group is solved. The coordinate transformation matrix is ​​used to spatially map the target radial velocity, distance resolution, and micro-Doppler features in the micro-electromagnetic wave data, so that they are in the same spatial reference as the surface electromyography signal, muscle activation timing features, force intensity index, and muscle coordination mode. Finally, under a unified time reference coordinate and a unified spatial coordinate system, the electrical signals of the forearm muscle group and the micro-electromagnetic wave data are fused in a multimodal manner to generate spatiotemporally aligned multimodal data.

[0032] S2. Based on the spatiotemporally aligned multimodal data, calculate the three-dimensional coordinate set of hand joints and extract motion trajectory features. At the same time, obtain muscle activation patterns and force intensity features through electromyography signal time-frequency analysis. The multimodal data is equally segmented according to a unified threshold standard to generate multimodal data frames, and parameter detection is performed to extract valid reflection points and obtain a set of reflection points. Furthermore, the spatiotemporally aligned multimodal data is equally segmented along the temporal dimension according to a unified threshold standard. Simultaneous slicing is performed on surface electromyography (EMG) signals, muscle activation temporal features, force intensity indicators, muscle coordination patterns, range-Doppler matrices, target radial velocity, range resolution, and micro-Doppler features using a fixed time window length (e.g., 50 milliseconds) and a step size of 25 milliseconds. This ensures strict alignment of each modal data segment in time, generating a series of time-segment aligned multimodal data frames. Subsequently, a two-parameter CFAR detection method is applied to each range-Doppler matrix, with guard and training units set up. Combined with a hierarchical decision-making mechanism, background noise is suppressed while weakly reflective targets are preserved, detecting effective reflection points of the hand in space. The position coordinates, radial velocity, and echo intensity information of all detected effective reflection points are collected to form a reflection point set.

[0033] It should be noted that the unified threshold standard refers to the synchronous segmentation of surface electromyography signals, muscle activation timing features, force intensity indicators, muscle coordination patterns, range-Doppler matrix, target radial velocity, range resolution, and micro-Doppler features in multimodal data processing to ensure that each modal signal has the same judgment benchmark during time segmentation. This is achieved by using consistent time window length and step size parameters (e.g., 50 ms window, 25 ms step size) to segment the signals simultaneously.

[0034] Based on the set of reflection points, a neighborhood radius is defined, and a set of three-dimensional coordinates of hand joints is generated using the multi-scale DBSCAN clustering algorithm. Furthermore, based on the spatial coordinates of each valid reflection point in the reflection point set, a neighborhood radius is defined to measure the local density between points, adapting to the physical dimensions of the hand joints. A multi-scale DBSCAN clustering algorithm is employed, setting a minimum number of points (e.g., 5), to perform density clustering on the reflection point set, outputting the density confidence scores of the hand joints. By traversing multiple neighborhood radius scales and combining prior knowledge of hand anatomy, a weighted average of the spatial coordinates is calculated for the density confidence scores of each hand joint, with the weights determined by the echo intensity, generating the corresponding three-dimensional spatial position of the joint. Finally, all cluster centers are arranged according to the hand topology, forming a three-dimensional coordinate set of the hand joints.

[0035] It should be noted that prior knowledge of hand anatomy refers to the standardized skeletal-muscle topological relationship of the hand based on human biomechanics research, including the spatial distribution pattern of medically defined joints (such as the fixed distance ratio between phalanges), joint degree of freedom constraints (such as the metacarpophalangeal joints can only flex and extend / abduct), and the mechanical transmission characteristics of muscle-skeletal attachment points.

[0036] It should be noted that the hand topology refers to the hierarchical and adjacency-based three-dimensional spatial structure formed by the connection of bones and joints between various joints of the hand, including the connection sequence and motion constraints of key points such as the palm root, palmodigital, proximal interphalangeal joints, and distal phalangeal joints.

[0037] It should be noted that the neighborhood radius is defined by combining hand anatomy and the spatial distribution characteristics of reflection points. For example, based on the average distance between finger joints and radar ranging resolution, 15 mm is selected as the initial value, and the optimal range is determined through multi-scale clustering verification. For example, with a fine scan in a step size of 0.5 mm, combined with radar point cloud density distribution and clustering stability index, the radius range that makes the joint recognition accuracy exceed 98% is finally selected as the optimal range to ensure that each neighborhood contains one and only one reflection point of a hand joint.

[0038] Based on spatiotemporally aligned multimodal data, a motion prediction model is constructed by performing joint extraction of spatiotemporal features and learning of sequence dependencies through a spatiotemporal graph convolutional neural network. Furthermore, based on spatiotemporally aligned multimodal data, a dynamic graph structure sequence is constructed, with the three-dimensional coordinate set of hand joints as nodes and the hand topology as edges. Each node feature includes surface electromyography (EMG) signals, muscle activation temporal features, force intensity indicators, muscle coordination patterns, target radial velocity, and micro-Doppler features at the corresponding time. The time series length is, for example, 64 frames (50 milliseconds per frame). A spatiotemporal graph convolutional neural network is used to process the graph sequence. In the spatial dimension, the feature information of adjacent joints is aggregated through graph convolution operations, and in the temporal dimension, the temporal dynamics of joint movement are captured through one-dimensional convolution or gated temporal convolution. The spatiotemporal graph convolutional neural network contains multiple spatiotemporal convolutional blocks, each of which outputs enhanced node feature representations. Finally, the three-dimensional coordinate set of hand joints at the next time step is regressed through a fully connected layer. The model is trained by minimizing the mean squared error loss function between the predicted coordinates and the actual three-dimensional coordinate set of hand joints to learn the nonlinear mapping relationship and long program sequence dependency between the multimodal data and hand movements, and finally, a motion prediction model is constructed.

[0039] It should be noted that the training process of the motion prediction model uses spatiotemporally aligned multimodal data as input. First, the data is divided into a multimodal data frame sequence according to a uniform threshold standard. Each sequence contains 64 consecutive frames (50 milliseconds per frame) of surface electromyography signals, muscle activation temporal features, force intensity indicators, muscle coordination patterns, distance-Doppler matrix, target radial velocity, distance resolution, micro-Doppler features, and the corresponding three-dimensional coordinate set of hand joints. Training samples are constructed using the three-dimensional coordinate set of hand joints, with the multimodal data of the Xth frame preceding the current frame as input and the three-dimensional coordinate set of hand joints of the (X+1)th frame as input. The goal is to combine the features of the hand joints into a predictive target. After initializing the parameters of the spatiotemporal graph convolutional neural network, the three-dimensional coordinate set of the hand joints is input into the spatiotemporal graph convolutional neural network for forward propagation. Through graph convolution operations, features of the hand topology are aggregated in the spatial dimension, and dynamic change patterns are extracted in the temporal dimension through gated temporal convolution. Finally, the predicted value of the three-dimensional coordinate set of the hand joints at the next moment is output. The mean square error between the predicted value and the actual three-dimensional coordinate set of the hand joints is calculated as the loss function. An adaptive optimization algorithm (e.g., Adam, with a learning rate set to 0.001) is used for backpropagation to update the network weights, completing the training process.

[0040] It should be noted that the training process of the spatial graph convolutional neural network is based on spatiotemporally aligned multimodal data. First, the data is segmented into a multimodal data frame sequence according to a unified threshold standard (e.g., a 50-millisecond time window and a 25-millisecond stride), constructing an input sequence containing 64 consecutive frames. Each frame includes surface electromyography (EMG) signals, muscle activation temporal features, force intensity indicators, muscle coordination patterns, distance-Doppler matrix, target radial velocity, distance resolution, and micro-Doppler features, and is time-synchronized with the corresponding three-dimensional coordinate set of hand joints. The three-dimensional coordinate set of hand joints is used as nodes in the graph structure, based on the hand topology... The system constructs edge connections to form a dynamic spatiotemporal graph sequence. Feature extraction is performed through multi-layer spatiotemporal graph convolutional blocks, where spatial convolution operations aggregate multimodal data frame sequences of adjacent joints at each time step, and temporal convolution operations use a gated temporal convolutional network to capture the temporal evolution of joint motion. The final layer outputs the predicted value of the three-dimensional coordinate set of hand joints at the next time step. The mean square error between the predicted value and the actual three-dimensional coordinate set of hand joints is used as the loss function, and the Adam optimizer (learning rate, e.g., 0.001, β1=0.9, β2=0.999) is used to update the parameters, completing the training process.

[0041] Based on the three-dimensional coordinate set of hand joints, a KD tree data structure is established using the binary tree indexing method, and the joint position prediction value is generated by inputting it into the motion prediction model. Furthermore, based on the three-dimensional coordinate set of hand joints, the three-dimensional spatial coordinates of each joint are used as data points. A KD-tree data structure is constructed using a binary tree indexing method. Spatial partitioning dimensions are selected (e.g., x, y, z coordinates are divided in turn), and the median is recursively selected as the partition point at each level to form a binary tree structure to support efficient spatial neighborhood queries. The current multimodal data frame sequence (including surface electromyography signals, muscle activation temporal features, force intensity indicators, muscle coordination patterns, target radial velocity, micro-Doppler features, etc.) is jointly organized with the spatial structure information represented by the KD-tree into a spatiotemporal graph input format, which is then input into the trained motion prediction model. The motion prediction model extracts spatial correlation and temporal dynamic features through a spatiotemporal graph convolutional neural network and outputs the predicted joint position values ​​of the three-dimensional coordinate set of hand joints at the next moment. Based on the predicted values ​​of joint positions, a time-continuous sequence of hand joint motion trajectories is generated using the Hungarian algorithm. The instantaneous velocity and motion direction angle of each joint are obtained based on the Doppler frequency shift formula to generate motion trajectory features. Furthermore, based on the predicted joint position values, the three-dimensional coordinate sets of hand joints in adjacent time frames are considered as two matching sets. Using Euclidean distance as the matching cost, a cost matrix is ​​constructed. The minimum cost bipartite graph matching is solved using the Hungarian algorithm to achieve a unique correspondence between joints across frames, eliminating label jumps caused by clustering or noise. Matching is performed sequentially between consecutive frames to form an ordered coordinate sequence of each joint on the time axis, generating a time-continuous hand joint motion trajectory sequence. Based on the hand joint motion trajectory sequence, the displacement vector is calculated for the position difference of each joint at adjacent time points (e.g., a time interval of 50 milliseconds). The instantaneous velocity magnitude and direction of each joint are inversely deduced using the Doppler frequency shift formula. The angle between the velocity vector and the reference coordinate system is further calculated to obtain the motion direction angle. Finally, the instantaneous velocity and motion direction angle are organized according to the joint number and time order to generate motion trajectory features.

[0042] Multimodal data is segmented using a sliding window to extract muscle subband energy features, and time-domain and frequency-domain analyses are performed to generate electromyographic feature vectors. Furthermore, a sliding window segmentation is performed on the surface electromyography (EMG) signals in the spatiotemporally aligned multimodal data. A unified threshold standard is used to set the time window length to ensure synchronization with the multimodal data. Within each window, the EMG signals are decomposed into multiple sub-bands using a bandpass filter bank, and the muscle sub-band energy of each sub-band is obtained. Simultaneously, time-domain features such as absolute value integral and variance are calculated for the time-domain signals, and power spectral density is obtained for the frequency-domain signals through Fast Fourier Transform, and frequency-domain features such as average frequency and median frequency are extracted. The muscle sub-band energy features of each sub-band, as well as the features obtained from time-domain and frequency-domain analysis, are concatenated in a predefined order to form a fixed-dimensional feature vector, which ultimately generates the EMG feature vector.

[0043] The muscle co-activation weight matrix is ​​generated by performing muscle co-activation element decomposition on the electromyographic feature vector using nonnegative matrix factorization, and the temporal and spatial distribution entropy values ​​of co-activation are calculated to obtain the muscle activation mode. The specific expression for calculating the entropy value of a spatial distribution is as follows: ; in, Indicates the first Spatial distribution entropy of individual muscle coordinators; Indicates the feature dimension; Indicates the first The first collaborative element in the th... Normalized weighting coefficients for each muscle channel (typically ranging from 0 to 1).

[0044] Furthermore, the electromyographic feature vector sequence is decomposed using nonnegative matrix factorization. The electromyographic feature vectors are arranged in chronological order to construct a co-activation weight matrix. Each column represents the weight distribution of a muscle co-element, and each row corresponds to the contribution coefficient of a forearm muscle group. Based on the co-activation weight matrix, the loading coefficients (0-1 normalized) of each muscle channel on the co-element are calculated. Principal component analysis (PCA) is used to reduce the dimensionality to three-dimensional space for clustering to obtain the spatial distribution pattern. The spatial distribution entropy value of each co-element is further calculated to reflect the spatial concentration or dispersion of muscle activation. The higher the entropy value, the wider the activation distribution. Combining the temporal activation curve in the co-activation time sequence matrix with the spatial distribution entropy value, the co-working law of muscles in different movement stages is comprehensively characterized, and the muscle activation pattern is finally obtained.

[0045] Based on muscle activation patterns, an electromyography-force mapping model was established using multivariate linear regression. Multimodal data were then input into the electromyography-force mapping model, and force intensity features were obtained through sliding window weighted fusion.

[0046] Furthermore, based on the co-activation weight matrix and co-activation time sequence matrix in the muscle activation mode, the temporal activation intensity of each co-activation element is extracted as an input variable, and combined with the synchronously acquired force intensity index as the target output to construct a sample set. The sample set is fitted by multivariate linear regression, the regression coefficient matrix is ​​solved, and an electromyography-force mapping model is established to realize the quantitative conversion from muscle co-activation state to force output. The muscle activation mode in the subsequently acquired multimodal data is input into the electromyography-force mapping model to obtain the force estimate for each time frame. The sliding window weighted fusion method is used to weight the force estimate within the continuous window using the Hanning window function to suppress instantaneous fluctuations, and finally obtain smooth and accurate force intensity features.

[0047] It should be noted that the training process of the electromyography-force mapping model uses muscle activation patterns and synchronous force intensity indicators as training data. First, the co-activation time sequence matrix is ​​extracted from the electromyography feature vector sequence using non-negative matrix factorization. Each row is taken as the time activation intensity sequence of muscle co-element, and each sequence corresponds to the activation level of a time frame. The activation intensity is taken as the input variable, and the synchronously measured force intensity indicator is taken as the output variable. Input-output sample pairs are constructed and linearly combined with the corresponding regression coefficients. The regression coefficient matrix that minimizes the sum of squared errors between the predicted value and the actual force intensity indicator is solved by the least squares method to complete the training process.

[0048] S3. Dynamically weight and fuse the three-dimensional position of the hand joints, motion trajectory features, muscle activation patterns, and force intensity features to generate a gesture feature vector, and input it into the residual neural network for gesture classification and gesture force estimation to identify gesture information. The three-dimensional position of the hand joints, the motion trajectory features, muscle activation patterns, and force intensity features are subjected to feature projection operations to generate motion feature projection, electromyographic feature projection, and force feature projection. These are then stitched and fused according to feature dimensions to generate a gesture feature vector. Furthermore, feature projection operations are performed on the three-dimensional position, motion trajectory features, muscle activation patterns, and force intensity features of the hand joints. First, a multi-scale feature extraction method is used to perform spatiotemporal pyramid layering on the three-dimensional position sequence of the hand joints, extracting displacement, acceleration, and joint angle change rate at different time windows and spatial scales. These are then mapped to a fixed-dimensional semantic space through linear transformation to generate motion feature projections. Fourier transforms are performed on the instantaneous velocity and motion direction angle sequences in the motion trajectory features, and the first 10 low-frequency coefficients are extracted. Combined with mean and standard deviation statistics, these are projected through a fully connected layer to generate motion feature projections. The co-activation temporal matrix in the muscle activation patterns is sliced ​​by time frame and mapped to a fixed-length vector through a dimensionality reduction network (e.g., a linear layer with ReLU activation) to generate electromyographic feature projections. The time series values ​​of the force intensity features are normalized and projected into a low-dimensional vector through a single-layer neural network to generate force feature projections. Finally, the motion feature projections, electromyographic feature projections, and force feature projections are sequentially concatenated and fused along the feature dimensions to form a high-dimensional joint feature vector, ultimately generating a gesture feature vector.

[0049] It should be noted that feature dimension refers to the number of independent feature elements contained in the feature vector.

[0050] The gesture feature vector is input into the residual neural network, convolution and normalization operations are performed to obtain residual block groups, and the data are compared and analyzed with historical gesture data to obtain gesture classification. Furthermore, the gesture feature vector is input into a residual neural network, which consists of multiple residual blocks. Each residual block contains two layers of one-dimensional convolutional operations, batch normalization, and a ReLU activation function. After the second convolutional layer, identity mapping or projection mapping is used to achieve cross-layer connections. After the gesture feature vector is processed by the initial convolutional layer and batch normalization, it passes through the residual block group step by step. The output of the residual block is normalized to maintain the stable feature distribution. After extracting high-order abstract features in the deep network, the feature map is compressed into a fixed-length vector through global average pooling. A multi-scale feature matching algorithm is used to calculate the similarity between the gesture feature vector and the category templates in the stored historical gesture data. Euclidean distance or cosine similarity is used for comparative analysis to match the nearest neighbor historical category, and finally, the gesture classification is obtained.

[0051] It should be noted that the training process of the residual neural network uses gesture feature vectors as input, and the corresponding gesture classification labels and true force values ​​as supervision signals. First, the gesture feature vectors are fed into the network, and after initial convolutional layers and batch normalization, they are sequentially passed through a group of residual blocks composed of multiple residual blocks. Each residual block contains two one-dimensional convolutional layers, batch normalization, and a ReLU activation function, and is connected through identity or projection shortcuts. The features output from the deeper layers of the network are global average pooled and then connected to two fully connected branches: one for gesture classification, outputting the softmax probability distribution of 20 gesture classes; the other for force regression, outputting a single continuous value. The total loss function is defined as the weighted sum of the classification loss and the regression loss. The classification loss uses the cross-entropy loss function, and the regression loss uses the mean squared error loss function. An optimizer is used to minimize the total loss function, and all network parameters are updated through backpropagation to complete the training process.

[0052] Define the total loss function, calculate the deviation between the force regression and the true force value, and obtain the estimated value of the gesture force. Furthermore, a force regression branch is added to the residual neural network. This branch is derived from the deep feature vector and outputs a single continuous value as the force prediction result through a fully connected layer. The total loss function is defined as a weighted sum of the classification loss and the regression loss, where the classification loss uses the cross-entropy loss function and the regression loss uses the mean squared error loss function, for example, with a weight ratio of 1:0.5. The synchronously labeled true force value is obtained, and the mean squared error between the force regression output and the true force value is calculated and used as a regression loss term to participate in the optimization of the total loss function. During training, the network parameters are updated through backpropagation so that the force regression output gradually approaches the true force value. During the inference phase, the force regression result output by the residual neural network is directly used as the gesture force estimate.

[0053] The gesture classification and gesture strength estimation are fused and analyzed using a dynamic confidence weighting method to identify gesture information.

[0054] Furthermore, a dynamic confidence weighting method is used to fuse and analyze the gesture classification results output by the residual neural network and their corresponding classification confidence and gesture force estimation values. The confidence of the gesture classification is used as the reliability weight of the gesture classification result. For example, when the confidence is below 0.7, it is considered a low confidence state. At the same time, the force estimation confidence is calculated based on the stability of the change of the gesture force estimation value. Dynamic weighting coefficients are generated using the two types of confidence to weight and fuse the gesture classification result of the current frame with the recognition result of the previous frame to suppress classification jitter. For the force estimation value, the final output is smoothed according to its confidence. When the confidence is high, the original estimate is retained, and when the confidence is low, it is compensated by the historical trend prediction value. Finally, the weighted gesture classification result and the calibrated gesture force estimation value are combined to generate complete gesture information containing action type and force level.

[0055] S4. Match and analyze the gesture information with the historical tactile effect library to obtain the gesture feedback pattern, and integrate it with the physical properties of the object to generate tactile feedback factors. The gesture information is matched and analyzed with the historical haptic effect database through a dynamic matching algorithm. Candidate records belonging to the same gesture type and force range are selected, and the pattern similarity is calculated to obtain the gesture feedback pattern. Furthermore, a dynamic matching algorithm compares the gesture classification results in the gesture information with the gesture type codes in the historical haptic effect database, filtering out records with consistent codes. Based on the MVC percentage range of the gesture force estimation value in the gesture information, candidate records belonging to the same force range are further filtered to form a candidate haptic feedback pattern set. A multi-scale similarity fusion algorithm is used to calculate the comprehensive similarity score between the current gesture and historical patterns. The trajectory shape is similar using the Frèchet distance metric curve, the force curve is compared using the Jensen-Shannon divergence distribution characteristics, and the phase difference is analyzed using cross-correlation. Finally, the optimal haptic feedback pattern whose similarity matches the physical properties of the current object is selected. The Dynamic Time Warping (DTW) algorithm is used to calculate the distance between motion trajectory features, combined with the Pearson correlation coefficient of the force curve, and weighted fusion is performed to obtain the comprehensive pattern similarity, for example, with weights of 0.6 and 0.4. The top three candidate records with the highest comprehensive pattern similarity are selected as the optimal matching results, and the corresponding haptic parameter sets are obtained to finally generate the gesture feedback pattern.

[0056] It should be noted that the construction process of the haptic effect library involves collecting a large amount of multimodal interaction data from users when performing standard gestures. Each type of gesture is labeled with a corresponding gesture type code, and the MVC percentage value of the applied force is recorded simultaneously, dividing it into 5 force ranges. In virtual or real interaction scenarios, for objects with different materials and physical properties, the set of haptic feedback parameters perceived by users is recorded, including frequency, amplitude, mode, and duration. Each record is stored in a hierarchical data structure, including gesture type code, force range division, haptic parameter set, and usage frequency statistics. Through long-term accumulation and user feedback, the ratings and call frequencies of each mode are continuously updated to form a historical haptic effect library with self-evolving capabilities.

[0057] By combining Hooke's Law and the damping effect, the gesture feedback mode is coupled with the physical properties of the current interactive object to generate tactile feedback factors.

[0058] Furthermore, by combining Hooke's Law and the damping effect, the tactile parameters in the gesture feedback mode are coupled with the physical properties of the current interactive object to obtain the physical properties of the current interactive object, including stiffness coefficient, damping coefficient, and roughness. The base frequency in the gesture feedback mode is adjusted according to the formula based on the stiffness coefficient to generate the adjusted frequency. This adjusted frequency is then corrected by combining the gesture force estimate and physical properties. Finally, the adjusted frequency, corrected amplitude value, stiffness coefficient, damping coefficient, and roughness are combined into a structured output to generate the tactile feedback factor.

[0059] It should be noted that physical attributes refer to the quantifiable mechanical characteristics of the current interactive object, including stiffness coefficient, damping coefficient, and roughness, which are used to characterize the deformation resistance, energy dissipation characteristics, and surface texture features of the object when subjected to force.

[0060] S5. Based on the tactile feedback factor, calculate the end-to-end delay from gesture recognition to tactile feedback, and perform dynamic fitting of electromyographic signals to generate the intensity mapping curve of tactile feedback, and obtain the gesture recognition control report.

[0061] Record the start time of the gesture feature vector input to the residual neural network and the end time of the generation of tactile feedback factors, and calculate the end-to-end delay from gesture recognition to tactile feedback using the timestamp difference method; Furthermore, the starting moment of the gesture feature vector input to the residual neural network is recorded and timestamped using a high-precision clock (e.g., time resolution up to 100 nanoseconds); the ending moment of the completion of tactile feedback factor generation, i.e. the moment when the Hooke's law and damping effect combined method completes parameter coupling and outputs tactile feedback factor, is also timestamped; based on a unified time reference coordinate, the end-to-end delay from the gesture feature vector entering the residual neural network to the generation of tactile feedback factor, and from gesture recognition to tactile feedback is calculated using the timestamp difference method.

[0062] Based on the end-to-end delay from gesture recognition to tactile feedback, dynamic energy calculation is performed to obtain a time-seriesd real-time electromyography intensity sequence, and a dynamic curve fitting operation is performed through a nonlinear regression function to generate a tactile feedback intensity mapping curve. Furthermore, based on the end-to-end delay from gesture recognition to tactile feedback, the surface electromyography (SEMG) signal is time-aligned to ensure that subsequent analysis is based on the true response timing. The corrected SEMG signal is then segmented into sliding windows using a unified threshold standard (e.g., a 50-millisecond time window and a 25-millisecond step size). The envelope energy of the signal within each window is calculated to generate a time-seriesd real-time EEMG intensity sequence. The corrected amplitude value from the tactile feedback factor at the corresponding moment of each real-time EEMG intensity sequence segment is extracted as the target output. Using the real-time EEMG intensity sequence as input and the corrected amplitude value as the desired output, a dynamic curve fitting operation is performed to construct a nonlinear regression function. By minimizing the error between the predicted amplitude and the actual corrected amplitude value, the regression function parameters are optimized, ultimately outputting a time-varying adjustment curve, generating a tactile feedback intensity mapping curve.

[0063] The intensity mapping curve of haptic feedback, gesture information, and haptic feedback factors are standardized and integrated to generate a gesture recognition control report.

[0064] Further standardization and integration of the haptic feedback intensity mapping curve, gesture information, and haptic feedback factors are performed. First, the haptic feedback intensity mapping curve is time-aligned and normalized, mapping its amplitude value to the 0.0–1.0 range, and timestamps of each data point are labeled with a unified time reference coordinate. The gesture classification results and gesture force estimates in the gesture information are encoded as structured fields. The adjusted frequency, corrected amplitude value, pattern type, duration, and spatial distribution parameters in the haptic feedback factors are formatted, with the frequency unit being Hz, the amplitude being a normalized value, and the pattern type being identified by a string. The above three parts of data are aligned by timestamp and merged into a single record entry containing a timestamp, gesture classification, gesture force estimate, various parameters of the haptic feedback factors, and sampling points of the haptic feedback intensity mapping curve. Finally, the data is organized according to a binary protocol format to generate a gesture recognition control report.

[0065] This embodiment also provides a computer device applicable to the gesture recognition control method based on a virtual reality device, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the gesture recognition control method based on a virtual reality device as proposed in the above embodiment.

[0066] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the gesture recognition control method based on a virtual reality device as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0067] In summary, this invention achieves density clustering and spatial structure analysis of effective reflection points in micro-electromagnetic wave data by defining a neighborhood radius based on the set of reflection points and generating a three-dimensional coordinate set of hand joints using a multi-scale DBSCAN clustering algorithm, thus accurately identifying the spatial position of the hand. Furthermore, by coupling the gesture feedback mode with the physical properties of the current interactive object, a tactile feedback factor is generated, achieving dynamic fusion of tactile parameters with stiffness coefficient, damping coefficient, and roughness. This ensures that the feedback intensity matches the material properties, enhancing the realism of the interaction.

[0068] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A gesture recognition control method based on a virtual reality device, characterized in that: include, Electrical signals and micro-electromagnetic wave data of the forearm muscle group were collected and preprocessed to generate aligned multimodal data; Based on spatiotemporally aligned multimodal data, the three-dimensional coordinate set of hand joints is calculated and motion trajectory features are extracted. At the same time, muscle activation patterns and force intensity features are obtained through electromyography signal time-frequency analysis. The three-dimensional position of the hand joints, the motion trajectory features, the muscle activation patterns, and the force intensity features are dynamically weighted and fused to generate a gesture feature vector. This vector is then input into a residual neural network for gesture classification and gesture force estimation to identify gesture information. The gesture information is matched and analyzed with a historical haptic effect database to obtain gesture feedback patterns, and then integrated with the physical properties of objects to generate haptic feedback factors. Based on the tactile feedback factor, the end-to-end delay from gesture recognition to tactile feedback is calculated, and the intensity mapping curve of tactile feedback is generated by dynamic fitting of electromyography signals to obtain the gesture recognition control report.

2. The gesture recognition control method based on a virtual reality device as described in claim 1, characterized in that: The electrical signals of the forearm muscle group include surface electromyography signals, muscle activation timing characteristics, force intensity indicators, and muscle synergy patterns. The micro-electromagnetic wave data includes the range-Doppler matrix formed by the continuous wave reflection signal, the target radial velocity and range resolution, and the micro-Doppler characteristics; The preprocessing includes interference elimination, intermediate frequency filtering, dynamic range compression, and spatial coordinate transformation.

3. The gesture recognition control method based on a virtual reality device as described in claim 2, characterized in that: The specific steps for generating aligned multimodal data are as follows. The preprocessed electrical signals and micro-electromagnetic wave data of the forearm muscle group are time-aligned using a high-precision clock to generate a unified time reference coordinate. Based on a unified time reference coordinate, a coordinate transformation matrix is ​​established, and multimodal data fusion is performed to generate spatiotemporally aligned multimodal data.

4. The gesture recognition control method based on a virtual reality device as described in claim 3, characterized in that: The specific steps for calculating the three-dimensional coordinate set of hand joints and extracting motion trajectory features based on spatiotemporally aligned multimodal data are as follows. The multimodal data is equally segmented according to a unified threshold standard to generate multimodal data frames, and parameter detection is performed to extract valid reflection points and obtain a set of reflection points. Based on the set of reflection points, a neighborhood radius is defined, and a set of three-dimensional coordinates of hand joints is generated using the multi-scale DBSCAN clustering algorithm. Based on spatiotemporally aligned multimodal data, spatiotemporal feature joint extraction and sequence dependency learning are performed to construct a motion prediction model; Based on the three-dimensional coordinate set of hand joints, a KD tree data structure is established and input into the motion prediction model to generate joint position prediction values. Based on the predicted joint positions, a time-continuous sequence of hand joint motion trajectories is generated using the Hungarian algorithm. The instantaneous velocity and motion direction angle of each joint are calculated to generate motion trajectory features.

5. The gesture recognition control method based on a virtual reality device as described in claim 4, characterized in that: The specific steps for obtaining muscle activation patterns and force intensity characteristics using time-frequency analysis of electromyography signals are as follows. Multimodal data is segmented using a sliding window to extract muscle subband energy features, and time-domain and frequency-domain analyses are performed to generate electromyographic feature vectors. Muscle co-element decomposition is performed on the electromyographic feature vectors to generate a co-activation weight matrix, and the temporal and spatial distribution entropy values ​​of co-activation are calculated to obtain the muscle activation pattern. Based on muscle activation patterns, an electromyography-force mapping model is established. Multimodal data is input into the electromyography-force mapping model, and a sliding window weighting operation is performed to obtain force intensity features.

6. The gesture recognition control method based on a virtual reality device as described in claim 5, characterized in that: The process involves dynamically weighting and fusing the three-dimensional position of hand joints, motion trajectory features, muscle activation patterns, and force intensity features to generate a gesture feature vector. This vector is then input into a residual neural network for gesture classification and force estimation to identify gesture information. The specific steps are as follows: The three-dimensional position of the hand joints, the motion trajectory features, the muscle activation patterns, and the force intensity features are subjected to feature projection operations to generate motion feature projection, electromyographic feature projection, and force feature projection. These are then stitched and fused according to the feature dimensions to generate a gesture feature vector. The gesture feature vector is input into the residual neural network, convolution and normalization operations are performed to obtain residual block groups, and the data are compared and analyzed with historical gesture data to obtain gesture classification. Define the total loss function, calculate the deviation between the force regression and the true force value, and obtain the estimated value of the gesture force. Gesture classification and gesture force estimation are fused and analyzed to identify gesture information.

7. The gesture recognition control method based on a virtual reality device as described in claim 6, characterized in that: The process of matching and analyzing gesture information with a historical haptic effect database to obtain gesture feedback patterns, and integrating this with the physical properties of objects to generate haptic feedback factors, is detailed below. The gesture information is matched and analyzed with the historical haptic effect database through a dynamic matching algorithm. Candidate records belonging to the same gesture type and force range are selected, and the pattern similarity is calculated to obtain the gesture feedback pattern. The gesture feedback mode is coupled with the physical properties of the current interactive object to generate haptic feedback factors.

8. The gesture recognition control method based on a virtual reality device as described in claim 7, characterized in that: The specific steps include calculating the end-to-end delay from gesture recognition to tactile feedback based on the tactile feedback factor, dynamically fitting electromyographic signals to generate a tactile feedback intensity mapping curve, and obtaining a gesture recognition control report. Record the start time of the gesture feature vector input to the residual neural network and the end time of the generation of tactile feedback factors, and calculate the end-to-end delay from gesture recognition to tactile feedback; Based on the end-to-end delay from gesture recognition to tactile feedback, dynamic energy calculation is performed to obtain a time-seriesd real-time electromyography intensity sequence, and a dynamic curve fitting operation is performed through a nonlinear regression function to generate a tactile feedback intensity mapping curve. The intensity mapping curve of haptic feedback, gesture information, and haptic feedback factors are standardized and integrated to generate a gesture recognition control report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the data fusion method for monitoring risks and hidden dangers in transmission line channels as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the data fusion method for monitoring risks and hidden dangers in transmission line channels as described in any one of claims 1 to 8.

Images