Dynamic gesture recognition and device control method and system

Abstract

The application relates to a dynamic gesture recognition and device control method and system based on a data glove and a space-time graph convolution. The method comprises the following steps: collecting spatial coordinates, joint angles and motion trajectory time sequence data of each joint node of a hand through a data glove; constructing a hand topology structure and extracting joint included angle, finger bending degree and finger opening and closing degree features; calculating velocity vectors, acceleration vectors and trajectory curvature features based on continuous inter-frame joint position changes, and constructing a kinematics feature set; constructing a space-time graph convolution network, fusing position and velocity features in a space graph convolution layer, dynamically adjusting convolution kernel weights in a time convolution layer by using acceleration features, and introducing an attention mechanism based on joint motion intensity; training a network model to output a gesture recognition result through supervised learning; associating the recognition result with predefined semantic labels to construct a structured gesture semantic knowledge base; and mapping gesture semantics to device control instructions to realize real-time gesture control of a target device.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a dynamic gesture recognition method and system based on data gloves and spatiotemporal graph convolution. Background Technology

[0002] Gesture recognition, as an important means of natural human-computer interaction, has broad application prospects in fields such as command of intelligent unmanned systems and control of intelligent equipment. Existing gesture recognition technologies are mainly divided into two categories: vision-based gesture recognition and inertial sensor-based gesture recognition.

[0003] Vision-based gesture recognition technology acquires gesture images through a camera, extracts hand joint nodes using computer vision algorithms, and classifies gestures. For example, CN121281136A discloses a skeleton gesture recognition method based on a differential attention map convolutional network, which models the spatiotemporal topological relationship between skeleton nodes in the gesture image to be recognized for gesture prediction. CN120260137B discloses a gesture prediction method based on multi-view gesture images. However, pure vision methods have the following inherent defects: (1) they are easily affected by environmental factors such as changes in lighting, complex backgrounds, and hand occlusion, resulting in poor recognition stability; (2) they depend on image quality, and their recognition performance drops significantly in low-light or strong-light environments; (3) there is an inherent depth blur problem in recovering three-dimensional hand poses from two-dimensional images.

[0004] Inertial sensor-based gesture recognition technology directly collects hand joint motion parameters through wearable devices such as data gloves. For example, the multimodal smart data glove developed by Harbin Institute of Technology uses a flexible sensor array combined with a six-axis inertial measurement unit for gesture recognition, while the Chinese patent application CN116893739A filed by Wuhan University of Light Industry uses a data glove to acquire hand motion data and combines it with a spatiotemporal graph convolution model for action recognition.

[0005] However, existing sensor-based gesture recognition technologies still have the following shortcomings: (1) They do not make full use of the temporal evolution features of dynamic gestures. Most methods only use joint position coordinates and ignore kinematic features such as velocity and acceleration; (2) The network structure design is relatively simple and lacks a dynamic attention mechanism for key joints during gesture execution; (3) They have insufficient ability to parse continuous gesture sequences and are difficult to support application scenarios where multiple instructions are issued continuously.

[0006] Furthermore, most existing gesture recognition technologies stop at "gesture category labels" and fail to form a complete "human → gesture → device" interaction loop. There is a semantic gap between the operator's gesture intent and the target device's motion response, lacking an end-to-end interaction link from perception to execution, which limits the application value of gesture interaction technology in real human-machine collaboration scenarios. Summary of the Invention

[0007] The present invention aims to provide a dynamic gesture recognition and device control method and system based on data gloves and spatiotemporal graph convolution to solve the above-mentioned technical problems.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: According to one aspect of the present invention, a dynamic gesture recognition and device control method based on data gloves and spatiotemporal graph convolution is provided, comprising the following steps: S1. The operator wears data gloves to perform preset gesture actions and collects spatial coordinates, joint angles and motion trajectory time data of 21 joint nodes of the hand through the data gloves; S2. Construct a hand topology structure with 21 joint nodes including wrist joint, finger joint and fingertip. Extract joint angle, finger bending degree and finger opening degree features based on the spatial constraint relationship between hand joints to form a structured feature sequence. Reduce the dimensionality of the original sensing data to a one-dimensional array containing 27 features. S3. Based on the joint position changes between consecutive frames, calculate the joint velocity vector, acceleration vector and trajectory curvature features, and construct a set of kinematic features; S4. Construct a spatiotemporal graph convolutional network, fuse position and velocity features in the spatial graph convolutional layer, dynamically adjust the convolutional kernel weights using acceleration features in the temporal convolutional layer, and introduce an attention mechanism based on joint motion intensity. S5. Train the network model through supervised learning and output the dynamic gesture recognition results; S6. Associate the recognition results with predefined semantic labels to build a structured gesture semantic knowledge base, realizing the mapping from perceptual features to high-level semantics; S7. Map the recognized gesture semantics into device control commands, generate control signals and send them to the target device to achieve real-time control of device movement through gestures.

[0009] According to another aspect of the present invention, a dynamic gesture recognition and device control system based on data gloves and spatiotemporal graph convolution is also provided, comprising: The data acquisition module includes a data glove equipped with a high-precision inertial measurement unit, used to acquire spatial coordinates, joint angles, and motion trajectory time-series data of 21 joint nodes of the hand; The hand topology modeling module is used to construct a hand topology structure containing 21 joint nodes, extract joint angle, finger flexion and finger opening features, and reduce the original sensor data to a one-dimensional array containing 27 features. A kinematic feature extraction module is used to calculate joint velocity vectors, acceleration vectors, and trajectory curvature features based on the joint position changes between consecutive frames, and to construct a kinematic feature set. The spatiotemporal graph convolutional network module is used to construct a spatiotemporal graph structure of the hand. It integrates position and motion features in the spatial graph convolutional layer, dynamically adjusts the weights of the convolutional kernels using acceleration features in the temporal convolutional layer, and dynamically allocates the weights of each joint in feature aggregation through a motion intensity attention mechanism. The network training module is used to train a network model through supervised learning and output gesture recognition results. A gesture semantic knowledge base construction module is used to associate recognition results with predefined semantic labels to construct a structured gesture semantic knowledge base; The device control module is used to map the recognized gesture semantics into device control commands, generate control signals, and send them to the target device.

[0010] Through the above steps, the operator's hand gestures are recognized, understood and converted into control commands for the target device in real time. The target device then executes the corresponding motion response according to the commands, forming a complete human-machine interaction closed loop from the operator (person) to the target device (machine).

[0011] The beneficial effects of adopting the above technical solution in this invention are as follows: 1. Environmental adaptability: The data gloves directly collect hand joint motion parameters, which do not rely on visual sensors at all. They are not affected by environmental factors such as changes in lighting, complex backgrounds, or hand occlusion, and have stable recognition capabilities in all weather and complex environments. 2. Kinematic feature mining capability: Based on traditional joint position coordinates, velocity vectors, acceleration vectors and trajectory curvature features are explicitly introduced to construct a complete set of kinematic features, thereby improving the ability to capture the spatiotemporal evolution patterns of dynamic gestures. 3. Dynamic gesture recognition capability: A spatiotemporal graph convolutional network is proposed, which improves the ability to recognize complex dynamic gestures through hierarchical feature fusion, acceleration-based dynamic convolutional kernel adjustment, and motion intensity-based attention mechanism. 4. Continuous gesture parsing capability: Supports automatic segmentation and semantic parsing of continuous gesture sequences, and can parse multiple gesture combinations into continuous command sequences to meet the application requirements of continuous issuance of multiple commands; 5. Natural human-computer interaction capability: This invention uses the operator's gestures as the interaction entry point and the target device's motion response as the interaction exit point. The operator does not need to learn a complex interactive interface or carry additional communication equipment; they can complete the command and control of the device simply through natural gestures. Attached Figure Description

[0012] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 A flowchart of a dynamic gesture recognition and device control method based on data gloves and spatiotemporal graph convolution according to an embodiment of the present invention; Figure 2 A schematic diagram of the topological structure of the 21 joint nodes of the hand; Figure 3 A schematic diagram illustrating the principle of dimensionality reduction for finger curvature data; Figure 4 This is a diagram of the spatiotemporal graph convolutional network structure. Figure 5 This is a diagram illustrating the effect of continuous gesture semantic recognition and device control. Figure 6 This is a schematic diagram of a dynamic gesture recognition and device control system based on data gloves and spatiotemporal graph convolution according to an embodiment of the present invention. Detailed Implementation

[0014] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0015] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0016] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0017] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to mean “and / or” unless otherwise expressly stated herein.

[0018] Now combined Figures 1 to 5 The preferred embodiments of the present invention will be described herein. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0019] like Figure 1 As shown, one embodiment of the present invention provides a dynamic gesture recognition and device control method based on data gloves and spatiotemporal graph convolution, including the following steps: S1, Data Glove Acquisition A high-precision inertial measurement unit (IMU) data glove was used as the data acquisition device. This data glove embeds multiple IMUs, enabling real-time acquisition of the angle changes and motion states of each joint in the hand. The operator wears the data glove to perform preset hand gestures, and the glove acquires the spatial coordinates and joint angle data of 21 joint nodes in the hand at a frequency of 30 frames per second. It should be understood that the sampling frequency is not limited to 30 frames per second; any frequency sufficient to meet the temporal modeling requirements of dynamic hand gestures is acceptable.

[0020] S2, Hand Topology Modeling like Figure 2 As shown, the hand topology includes a total of 21 joint nodes, including the wrist joint, finger joints, and fingertips. The wrist joint is joint node 0, the thumb includes joint nodes 1-4 (including one non-visual motion joint 1), the index finger includes joint nodes 5-8, the middle finger includes joint nodes 9-12, the ring finger includes joint nodes 13-16, and the little finger includes joint nodes 17-20.

[0021] The coordinates and associated information of each node are defined by a spatial linked list, forming a complete hand skeleton topology.

[0022] For the original high-dimensional sensor data, this invention proposes a data dimensionality reduction method based on the topology of the hand to extract features such as joint angle, finger curvature and finger opening and closing.

[0023] Taking the index finger as an example, joint nodes 5, 6, and 7 naturally form space △5,6,7, while joint nodes 6, 7, and 8 form space △6,7,8. The ratios of angle ∠5,6,7 and the ratios from wrist joint node 0 to its two endpoints (joint nodes 5 and 7) are also relevant. l 0,5 : l 0,7 Define the space △5,6,7, angles ∠6,7,8, and the ratios from wrist joint node 0 to its two endpoints (joint nodes 6 and 8). l 0,6 : l 0,8Define the spatial coordinates △6, 7, 8, and the spatial positions of the index finger using angles ∠0, 5, 6. This method is extended to the middle, ring, and little fingers, achieving a complete representation and dimensionality reduction of the bending degree of all four fingers.

[0024] like Figure 3 As shown, the thumb has a non-visual motion joint 1, and the detection of this joint has a large error. Therefore, this invention employs a special dimensionality reduction method. The ratios of ∠2, 3, 4 and motion joint 17 to its two endpoints (joint nodes 2 and 4) are used. l 17,2 : l 17,4 Define the space △2,3,4, and use ∠17,2,3 to define the spatial position of the thumb.

[0025] Taking the index and middle fingers as examples, ∠5, 6, 9, and 10 represent the degree of opening between the two fingers. Because... l 5,6 and l 9,10 These angles are typically not in the same plane; ∠5, 6, 9, 10 represent spatial angles. This invention projects these spatial angles onto planes 0, 5, 6 to obtain planar angles, thus eliminating data errors caused by viewing angle and coordinate reconstruction. This method can be extended to other finger combinations.

[0026] Through the dimensionality reduction process described above, the original gesture sensing data is transformed into a one-dimensional array containing 27 features, significantly reducing the computational load of subsequent recognition algorithms. Among these, 23 features characterize finger curvature, as shown in Table 1, including 9 length ratios and 13 angles; and 4 features characterize finger opening / closing, as shown in Table 2.

[0027] Table 1. Characteristics representing finger curvature

[0028] Table 2. Characteristics representing finger openness / closeness

[0029] S3, Kinematic Feature Extraction Based on joint position vectors from three consecutive frames , and Calculate the kinematic characteristics. The velocity vector is... and The acceleration vector is The trajectory curvature is .in, Represents the cross product of vectors. This represents the Euclidean norm.

[0030] The aforementioned kinematic features characterize the dynamics of the gesture execution process from different dimensions: the velocity vector reflects the speed and direction of joint movement, the acceleration vector reflects the degree of abrupt change in movement, and the trajectory curvature reflects the degree of bending of the movement path. Together, these three features constitute a complete set of kinematic features, providing temporal information for subsequent recognition.

[0031] The structural features and kinematic features obtained from the dimensionality reduction are integrated to form a complete feature representation for each gesture sample. This feature representation includes both spatial structural information of the hand (joint angles, finger flexion, and finger opening) and temporal dynamic information (velocity, acceleration, and curvature), providing multimodal data input for subsequent spatiotemporal graph convolutional networks.

[0032] S4. Construct a spatiotemporal graph convolutional network like Figure 4 As shown, the network as a whole adopts an enhanced spatiotemporal graph convolutional architecture, which includes spatial graph convolutional layers, temporal convolutional layers, and an attention mechanism module.

[0033] In the spatial graph convolutional layer, the joint position coordinates and motion velocity vectors are concatenated as features and used as the initial input of this layer. This operation enhances the ability to represent the motion state of joints in spatial topological relationships, enabling the model to better distinguish between static poses and dynamic gesture trajectories.

[0034] In the temporal convolutional layer, the weights of the temporal convolutional kernel are dynamically adjusted using joint acceleration features. .in, The original temporal convolution kernel, The temporal convolution kernel size, and These represent the number of input and output feature channels, respectively. The adjusted convolutional kernel weights, This is the joint acceleration feature vector; For learnable projection matrix; For bias terms; The sigmoid activation function is used. Through the above modulation operation, the convolutional kernel weights are dynamically adjusted according to the real-time changes in joint acceleration, which enhances the network's feature response to sudden motion changes and suppresses the response to uniform motion segments, thereby improving the model's sensitivity to sudden motion changes such as rapid starts and stops and direction changes.

[0035] In a specific network configuration embodiment, let the number of input feature channels of the current temporal convolutional layer be... Number of output feature channels temporal convolution kernel size Then the original temporal convolution kernel The dimension is Joint acceleration eigenvectors The dimension is consistent with the number of input feature channels. This acceleration feature is obtained by calculating the second-order difference of the joint position vectors over three consecutive frames. Learnable projection matrix. Projecting the acceleration features from 27 dimensions to the 64-dimensional output channel space, the bias term... The modulation process is as follows: First, the linear transform is calculated. , in The modulation coefficients are then generated using the Sigmoid function. ,in Finally, the modulation coefficients are applied channel by channel to each output channel of the original convolutional kernel, that is, for the , The time position, the The input channel and the first One output channel, Same output channel In all time locations and input channel Share the same modulation coefficient In a forward propagation, suppose the acceleration feature vector of a certain frame... (27 dimensions in total, values ​​vary from frame to frame), projected by the matrix and bias After the transformation, the linear transformation result of a certain output channel is: The corresponding modulation coefficient is This means that the convolution kernel weights of the output channel are scaled to approximately 68% of their original value. If the acceleration characteristic amplitude is large at the moment of abrupt change in motion, the linear transformation result of a certain output channel reaches... The corresponding modulation coefficient is With approximately 90% of the convolutional kernel weights retained, the network's response to the temporal characteristics at that moment is enhanced. If the acceleration characteristics are close to zero during the uniform motion segment, the linear transformation result of a certain output channel... The modulation coefficient is The convolutional kernel weights were suppressed to approximately 27%, resulting in a weaker temporal response of the network to this uniform velocity segment. The above parameters... , and convolution kernel weights All optimizations are performed through end-to-end backpropagation over the network. It should be noted that in this embodiment... , , The specific values ​​are merely illustrative and do not constitute a limitation of the present invention. In practical applications, the number of channels and the size of the convolutional kernels in each layer of the network can be adaptively adjusted according to the number of gesture categories, sample size, and computing power constraints of the deployment platform. The core technology of the present invention lies in the mechanism itself that uses acceleration features to dynamically modulate the temporal convolutional kernel. Compared with standard temporal convolution, this dynamic adjustment mechanism only adds a projection matrix to the network structure. (In this embodiment, it is) (parameters) and bias terms (In this embodiment, there are 64 parameters), which adds a total of 1792 parameters, accounting for less than 0.5% of the total number of network parameters. Moreover, the inference stage only adds one matrix multiplication and one Sigmoid activation operation, which has a negligible impact on real-time performance. It can significantly improve the model's sensitivity and recognition accuracy to motion change features in dynamic gestures without increasing computational overhead.

[0036] Define joints During the period The average motion intensity within is Attention weights are generated using the Softmax function. .in, The total number of joints. For joints At any moment The velocity vector. This attention weight is used to dynamically adjust the contribution of each joint node to the global gesture features during feature aggregation. Joints with high motion intensity receive higher attention weights, thereby highlighting the role of key motion joints.

[0037] S5, Network Training The network is trained end-to-end using a supervised learning paradigm, outputting dynamic gesture recognition results. The gesture database contains 50 standard dynamic gestures, with 15 samples per gesture class. Each sample contains a complete sequence of joint nodes and its corresponding unique encoded label. The loss function is cross-entropy loss with weight decay. .in, The total number of samples, For real labels, To predict probabilities for the network, For network parameters, This is the hyperparameter for regularization intensity.

[0038] During training, all samples were divided into training, validation, and test sets in a 7:1.5:1.5 ratio. The training set was used to update network parameters, the validation set for hyperparameter tuning and overfitting monitoring, and the test set for evaluating the final recognition performance. The optimizer used the Adam algorithm with an initial learning rate of 0.001 and an exponential decay rate estimated by the first moment. The exponential decay rate estimated by the second moment Numerical stability constant The batch size is set to 32, meaning 32 samples are randomly sampled per batch to calculate gradients and update network parameters. The total number of training epochs is set to 200. The initial 80 epochs maintain a learning rate of 0.001, epochs 81-140 decay to 0.0001, and epochs 141-200 further decay to 0.00001. After each training epoch, the loss and recognition accuracy are calculated on the validation set. If the validation set loss does not decrease for 15 consecutive epochs, early stopping is triggered to terminate training. The network parameters corresponding to the epoch with the smallest validation set loss are used as the final model. During training, random noise (0.02 times the standard deviation of the corresponding joint coordinates) is applied to the joint coordinate data before each batch of data is input into the network, and the time axis of the gesture samples is randomly scaled (scaling factor uniformly sampled between 0.9 and 1.1) to enhance the model's robustness to sensor measurement errors and execution speed differences. Weight decay coefficient. A value of 0.0005 is used to impose an L2 norm constraint on all learnable parameters to prevent overfitting. Under the above training strategy, the network's overall recognition accuracy on the test set is 78.75%. During training, the validation set loss curve tends to stabilize after the 40th round, without significant overfitting, indicating that the weight decay and early stopping mechanisms effectively suppress the risk of overfitting.

[0039] S6. Construct a gesture semantic knowledge base The recognition results are associated with predefined semantic labels to construct a structured gesture semantic knowledge base, realizing the mapping from perceptual features to high-level semantics. The structured gesture semantic knowledge base is organized in a hierarchical directory structure, with gesture category as the basic organizational unit. It stores gesture number, gesture category name, semantic description, sample data index, and typical dynamic feature expressions, and supports bidirectional mapping and expansion between gesture category and semantic description.

[0040] In one specific implementation, the structured gesture semantic knowledge base is organized using a combination of index files and a file directory hierarchy. The root directory of the knowledge base stores time-series data files corresponding to various control gestures, with core mapping relationships recorded in the index files. The index files use gesture numbers as keys to create independent knowledge entries for each type of gesture. Each knowledge entry includes at least the following fields: gesture number (a unique identifier for each type of gesture); gesture category name (a semantic label identifying this type of gesture); semantic description (using natural language to describe the device control meaning corresponding to this type of gesture); feature template (represented by a 27-dimensional feature vector mean template, recording the typical numerical range and temporal variation patterns of joint angles and finger flexion during the execution of this type of gesture); and a sample index list (binding the storage path of all samples for this type of gesture, with each sample index associated with a corresponding time-series data file, and recording the timestamp information of the start frame, peak frame, and end frame). The knowledge base supports bidirectional queries: when the input is a gesture number, the retrieved outputs the gesture category name, semantic description, and standard feature template; when the input is a semantic keyword, the retrieved outputs the corresponding gesture number and feature template. The knowledge base also reserves extended fields for each type of gesture to record variant execution methods of that gesture or the index range of newly added samples. Through the above structured organization, the knowledge base achieves a reliable mapping from gesture perception features to device control semantics, providing a standardized semantic reference for the interpretation and verification of recognition results. Moreover, when adding new gesture types or supplementing samples, it is only necessary to add new entries to the index file according to the established field format and store the data file in the corresponding directory, without reconstructing the overall structure, thus possessing good scalability.

[0041] S7, Control Command Mapping and Target Device Control Equipment control commands include one or more of the following: motion direction control, motion speed control, motion mode switching, and equipment start / stop. Control modes include single-command control mode and continuous-command control mode. In single-command control mode, a single gesture corresponds to a single equipment control command. For example, a "clenched fist" gesture corresponds to stopping the equipment, a "pointing index finger forward" gesture corresponds to moving the equipment forward, a "palm left swing" gesture corresponds to turning the equipment left, and a "palm right swing" gesture corresponds to turning the equipment right. In continuous-command control mode, a sequence of gestures corresponds to a device motion trajectory or a complex action. The system takes continuously input multi-segment gesture timing data as input, automatically segments the gesture sequence, and outputs the corresponding sequence of equipment control commands sequentially. For example, the gesture sequence "forward → left turn → forward → stop" controls the equipment to move along an L-shaped trajectory.

[0042] like Figure 5As shown, the device control module converts the identified commands into control signals executable by the target device and sends them to the target device via wired or wireless means. Target devices include, but are not limited to, unmanned ground vehicles, unmanned aerial vehicles, robotic arms, and intelligent equipment.

[0043] In continuous interaction scenarios, testers execute multiple standard dynamic gestures consecutively, forming "gesture statements" with a clear semantic order. The system takes the continuously input multi-segment gesture sequence data as input and automatically parses the gesture sequence without manual segmentation, outputting the corresponding device control commands for each gesture in sequence. For example, the continuous gesture sequence "Forward → Obstacle Detected → Stop → Left Turn → Forward" is parsed by the system as follows: Command 1 is Forward (device moves forward), Command 2 is Obstacle Detected (triggers obstacle avoidance mode), Command 3 is Stop (device pauses), Command 4 is Left Turn (device turns 90 degrees to the left), and Command 5 is Forward (device continues moving forward). This function enables operators to issue complex multi-step commands through continuous gestures, greatly improving the efficiency and naturalness of human-computer interaction.

[0044] Through the coordinated efforts of the aforementioned steps, the gestures performed by the operator while wearing data gloves undergo data collection, feature extraction, network recognition, semantic mapping, and command conversion, ultimately driving the target device to produce a corresponding motion response. Throughout the entire process, the operator does not need to rely on visual sensors, be in a specific lighting environment, or learn a complex operating interface; they can complete real-time command and control of the equipment solely through natural gestures, forming a complete human-machine interaction closed loop of "human → gesture → equipment".

[0045] like Figure 6 As shown, another embodiment of the present invention provides a dynamic gesture recognition and device control system based on data gloves and spatiotemporal graph convolution, comprising: The data acquisition module 100 includes a data glove equipped with a high-precision inertial measurement unit (IMU) for acquiring spatial coordinates, joint angles, and motion trajectory time-series data of 21 joint nodes of the hand. The specific process has been described above and will not be repeated here. The hand topology modeling module 200 is used to construct a hand topology structure containing 21 joint nodes, extract joint angle, finger flexion and finger opening features, and reduce the original sensing data to a one-dimensional array containing 27 features. The specific process has been described above and will not be repeated here. The kinematic feature extraction module 300 is used to calculate joint velocity vectors, acceleration vectors and trajectory curvature features based on the joint position changes between consecutive frames, and to construct a kinematic feature set. The specific process has been described above and will not be repeated here. The spatiotemporal graph convolutional network module 400 is used to construct a spatiotemporal graph structure of the hand. It fuses position and motion features in the spatial graph convolutional layer, dynamically adjusts the weights of the convolutional kernels using acceleration features in the temporal convolutional layer, and dynamically allocates the weights of each joint in feature aggregation through a motion intensity attention mechanism. The specific process has been described above and will not be repeated here. The network training module 500 is used to train the network model through supervised learning to output gesture recognition results. The specific process has been described above and will not be repeated here. The gesture semantic knowledge base construction module 600 is used to associate the recognition results with predefined semantic labels to build a structured gesture semantic knowledge base. The specific process has been described above and will not be repeated here. The device control module 700 is used to map the recognized gesture semantics into device control commands, generate control signals and send them to the target device. The specific process has been described above and will not be repeated here.

[0046] The hand topology modeling module 200, kinematic feature extraction module 300, spatiotemporal graph convolutional network module 400, network training module 500, gesture semantic knowledge base construction module 600, and device control module 700 can be integrated into an electronic device (e.g., a circuit board or controller). This electronic device can be integrated or detachably mounted on the data glove and electrically connected to the high-precision inertial measurement unit of the data glove. The electronic device has a processor and a memory, the memory storing a computer program, wherein the computer program, when executed by the processor, implements the steps of the method described above.

[0047] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to perform the method of the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0048] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0049] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store the aforementioned various data, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0050] Through the collaborative work of the various modules mentioned above, the gestures performed by the operator while wearing data gloves are processed through stages such as data collection, feature extraction, network recognition, semantic mapping, and command conversion, ultimately driving the target device to produce a corresponding motion response. Throughout the entire process, the operator does not need to rely on visual sensors, be in a specific lighting environment, or learn a complex operating interface; they can complete real-time command and control of the equipment solely through natural gestures, forming a complete human-machine interaction closed loop of "human → gesture → equipment".

[0051] The foregoing has shown and described the basic principles, main features, and application effects of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dynamic gesture recognition and device control method based on data gloves and spatiotemporal graph convolution, characterized in that, The steps are as follows: S1. The operator wears data gloves to perform preset gesture actions and collects spatial coordinates, joint angles and motion trajectory time data of 21 joint nodes of the hand through the data gloves; S2. Construct the hand topology and extract features such as joint angle, finger flexion, and finger opening and closing, and reduce the original sensor data to a one-dimensional array containing 27 features; S3. Calculate velocity vectors, acceleration vectors, and trajectory curvature features based on joint position changes between consecutive frames, and construct a set of kinematic features; S4. Construct a spatiotemporal graph convolutional network, fuse position and velocity features in the spatial graph convolutional layer, dynamically adjust the convolutional kernel weights using acceleration features in the temporal convolutional layer, and introduce an attention mechanism based on joint motion intensity. S5. Train the network model through supervised learning and output the gesture recognition results; S6. Associate the recognition results with predefined semantic labels to construct a structured gesture semantic knowledge base; S7. Map gesture semantics to device control commands to achieve real-time gesture control of the target device.

2. The method according to claim 1, characterized in that, Of the 21 joint nodes, the wrist joint is joint node 0, the thumb includes joint nodes 1-4, the index finger includes joint nodes 5-8, the middle finger includes joint nodes 9-12, the ring finger includes joint nodes 13-16, and the little finger includes joint nodes 17-20.

3. The method according to claim 2, characterized in that, S2 include: S21. For the index finger, middle finger, ring finger and little finger, the degree of finger bending is characterized by the spatial triangle formed by three adjacent points and the ratio relationship from the wrist joint to the endpoint of the triangle. S22. For the thumb, the ratio relationship between joint node 17 to joint node 2 and joint node 4 is used to characterize the spatial position of the thumb in order to overcome the detection error of the non-visual motion joints of the thumb. S23. Project the spatial angle onto the reference plane to characterize the finger opening and closing degree; S24. Finally, the original gesture sensing data is reduced to a one-dimensional array containing 27 features.

4. The method according to claim 3, characterized in that, S21 specifically refers to, For the index finger, based on spaces △5,6,7 and △6,7,8, the space △5,6,7 is defined by ∠5,6,7 and the ratio l0,5:l0,7 from the wrist joint to the two endpoints, the space △6,7,8 is defined by ∠6,7,8 and the ratio l0,6:l0,8 from the wrist joint to the two endpoints, and the spatial position of the index finger is defined by ∠0,5,6. The definition of a spatial triangle for the middle, ring, and little fingers is similar to that for the index finger.

5. The method according to claim 3, characterized in that, S23 specifically refers to: For the angle between the index and middle fingers, ∠5, 6, 9, 10 represent the degree of opening and closing between the index and middle fingers. The spatial angle is projected onto the plane 0, 5, 6 to obtain the plane angle, so as to eliminate the data error caused by the observation angle and coordinate reconstruction. The methods for representing the angles between the other fingers and the degree of finger opening and closing are similar to those for the angle between the index and middle fingers.

6. The method according to claim 1, characterized in that, S3 include: S31. Calculate the velocity vector , ; S32. Calculate the acceleration vector ; S33, Calculate the trajectory curvature ; in, Represents the cross product of vectors. Denotes the Euclidean norm. It is the position vector for three consecutive frames.

7. The method according to claim 6, characterized in that, The step S4, which involves fusing position and velocity features in the spatial graph convolutional layer, specifically involves concatenating the joint position coordinates and motion velocity vectors in the spatial graph convolutional layer as the initial input to the spatial graph convolutional layer. ; in, These are the initial input features for the spatial graph convolutional layer.

8. The method according to claim 6, characterized in that, The step S4, which describes dynamically adjusting the convolution kernel weights in the temporal convolutional layer using acceleration features, specifically involves... ; in, The original temporal convolution kernel, The adjusted convolutional kernel weights, Joint acceleration characteristics, For learnable projection matrices, For bias terms, This is the Sigmoid activation function.

9. The method according to claim 6, characterized in that, In S4, the attention mechanism based on joint motion intensity is specifically as follows: S41, Define Joints During the period The average motion intensity within is ; S42. Attention weights are generated using the Softmax function. ; in, The total number of joints. For joints At any moment The velocity vector.

10. The method according to claim 1, characterized in that, In S5, the supervised learning uses a weighted decaying cross-entropy loss function. ; in, The total number of samples, For the sample The true label, The probability distribution predicted by the network. For network parameters, This is the hyperparameter for regularization intensity.

11. A dynamic gesture recognition and device control system based on data gloves and spatiotemporal graph convolution, characterized in that, include: The data acquisition module includes a data glove equipped with a high-precision inertial measurement unit, used to acquire spatial coordinates, joint angles, and motion trajectory time-series data of 21 joint nodes of the hand; The hand topology modeling module is used to construct a hand topology structure containing 21 joint nodes, extract joint angle, finger flexion and finger opening features, and reduce the original sensor data to a one-dimensional array containing 27 features. A kinematic feature extraction module is used to calculate joint velocity vectors, acceleration vectors, and trajectory curvature features based on the joint position changes between consecutive frames, and to construct a kinematic feature set. The spatiotemporal graph convolutional network module is used to construct a spatiotemporal graph structure of the hand. It integrates position and motion features in the spatial graph convolutional layer, dynamically adjusts the weights of the convolutional kernels using acceleration features in the temporal convolutional layer, and dynamically allocates the weights of each joint in feature aggregation through a motion intensity attention mechanism. The network training module is used to train a network model through supervised learning and output gesture recognition results. A gesture semantic knowledge base construction module is used to associate recognition results with predefined semantic labels to construct a structured gesture semantic knowledge base; The device control module is used to map the recognized gesture semantics into device control commands, generate control signals, and send them to the target device.

12. The system according to claim 11, characterized in that, The device control module supports the following human-machine interaction control modes: Single command control mode: A single gesture corresponds to a single device control command; Continuous command control mode: Gesture sequences correspond to device motion trajectories or compound actions.

Images