A deep learning-based dexterous hand action recognition method, device and medium

Abstract

The application discloses a dexterous hand action recognition method and device based on deep learning and a medium, relates to the technical field of intelligent action perception, and comprises the following steps: performing space-time structural modeling on multi-modal action perception data to generate an action space-time graph; inputting the action space-time graph into a multi-modal space-time graph attention network, assigning attention weights to graph nodes according to the time sequence change state of hand actions, and performing attention weighted fusion on visual features and inertial features to obtain space-time fusion feature representation; inputting the space-time fusion feature representation into a time sequence convolution network, simultaneously modeling short-term change features and long-term evolution features of gesture actions in the time dimension by using multi-scale dilated convolution, outputting time sequence action features, and obtaining dexterous hand action recognition results through gesture category discrimination. The application obtains action features with good discriminability, and realizes stable recognition of dexterous hand actions.

Description

Technical Field

[0001] This invention relates to the field of intelligent motion perception technology, and in particular to a method, device and medium for dexterous hand motion recognition based on deep learning. Background Technology

[0002] In recent years, with the continuous advancement of deep learning technology, especially in the fields of computer vision and intelligent perception, deep learning has achieved remarkable success in gesture recognition and human motion capture. Gesture recognition, as an important branch of human motion recognition, is widely used in human-computer interaction, virtual reality, and intelligent robot control. Traditional gesture recognition methods largely rely on vision-based information, such as using cameras to capture images and extracting hand feature points through image processing algorithms for subsequent classification and recognition. With the deepening application of deep learning algorithms in image and sensor data analysis, deep learning-based gesture recognition technology has gradually become mainstream, especially driven by deep convolutional neural networks (CNNs) and temporal models (such as LSTM and GRU), resulting in significant improvements in gesture recognition accuracy and robustness.

[0003] Despite advancements in existing deep learning-based gesture recognition technologies, these techniques typically rely on a single modality for data acquisition, such as visual information or inertial sensor data alone. This can significantly impact recognition performance in complex environments, especially under poor lighting conditions or with significant background interference. Secondly, while demonstrating strong capabilities in processing time-series data, effectively modeling the spatiotemporal characteristics of complex gestures, particularly in the short and long-term time dimensions, remains challenging. These issues mean that existing technologies may not be accurate enough when handling complex dexterity hand movements, especially in recognizing rapid gesture changes or long-duration continuous actions. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a dexterous hand action recognition method based on deep learning to solve the problems of insufficient environmental adaptability and difficulty in uniformly modeling the spatiotemporal features of complex hand actions in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a dexterous hand action recognition method based on deep learning, which includes: acquiring visual images, inertial data and depth information of a dexterous hand; converting the visual images into three-dimensional coordinates of key points of the hand; performing noise reduction and correction on the inertial data; performing illumination compensation and depth map processing on the depth information; and obtaining multimodal action perception data.

[0008] Perform spatiotemporal structured modeling on multimodal motion perception data to generate a motion spatiotemporal graph;

[0009] The action spatiotemporal graph is input into the multimodal spatiotemporal graph attention network. Attention weights are assigned to graph nodes according to the temporal changes of hand movements. Attention weighted fusion is performed on visual features and inertial features to obtain spatiotemporal fusion feature representations.

[0010] The spatiotemporal fusion feature representation is input into the temporal convolutional network. Multi-scale dilated convolution is used to simultaneously model the short-term change features and long-term evolution features of hand gestures in the time dimension, outputting temporal action features. The dexterity hand action recognition results are obtained by judging the gesture category.

[0011] As a preferred embodiment of the deep learning-based dexterity hand movement recognition method of the present invention, the specific steps for acquiring visual images, inertial data, and depth information of the dexterity hand, and converting the visual images into three-dimensional coordinates of key hand points, are as follows.

[0012] Visual images, inertial data, and depth information of the dexterous hand are collected, and the visual images, inertial data, and depth information are time-aligned. The hand region is located in the time-aligned visual images, and the two-dimensional position information of each key point of the hand is extracted.

[0013] Based on the depth information at the corresponding time, the two-dimensional position information of the hand key points is mapped to the three-dimensional coordinates of the hand key points in the spatial coordinate system.

[0014] As a preferred embodiment of the deep learning-based dexterity hand action recognition method of the present invention, the steps of performing denoising and correction on inertial data, and performing illumination compensation and depth map processing on depth information to obtain multimodal action perception data are as follows:

[0015] The time-aligned inertial data is processed by outlier removal and signal smoothing to eliminate random noise, and zero-bias correction and drift correction are performed to obtain the corrected inertial data.

[0016] The time-aligned depth information is reduced by suppressing brightness interference to minimize the impact of ambient light changes, and noise reduction and smoothing are performed to fill in the missing depth areas, thus obtaining continuous and stable depth map data.

[0017] The three-dimensional coordinates of key hand points, the corrected inertial data, and the continuous stable depth map data are uniformly organized to output multimodal motion perception data.

[0018] As a preferred embodiment of the deep learning-based dexterity hand action recognition method of the present invention, the specific steps for performing spatiotemporal structured modeling on multimodal action perception data to generate a spatiotemporal action graph are as follows:

[0019] The three-dimensional coordinates of hand key points in multimodal motion perception data are used as graph nodes, and the corrected inertial data is added as graph node attributes to form a preliminary graph structure.

[0020] The connection relationship between each pair of adjacent graph nodes is defined according to the hand skeleton structure. Spatial edges are constructed by calculating the Euclidean distance and relative motion angle between graph nodes.

[0021] The initial spatial graph is generated by connecting the graph nodes in the preliminary graph structure using spatial edges and attaching continuous and stable depth map data.

[0022] In the initial spatial graph, time edges are formed by connecting graph nodes at different time frames, and these time edges are integrated into the initial spatial graph to obtain the action spatiotemporal graph.

[0023] As a preferred embodiment of the deep learning-based dexterity hand action recognition method of the present invention, the steps include: inputting the action spatiotemporal graph into a multimodal spatiotemporal graph attention network, assigning attention weights to graph nodes according to the temporal changes of hand actions, and performing attention-weighted fusion of visual features and inertial features to obtain a spatiotemporal fusion feature representation. The specific steps are as follows:

[0024] The action spatiotemporal graph is input into the multimodal spatiotemporal graph attention network. Based on the spatiotemporal edges in the action spatiotemporal graph, the change state of each graph node in the time series is calculated to obtain the temporal change state of hand actions.

[0025] Based on the temporal changes of hand movements, the changes of each graph node in the time series are analyzed, attention weights are assigned to the graph nodes, and dynamic feature fusion is performed on visual features and inertial features according to the assigned attention weights to obtain spatiotemporal fusion feature representations.

[0026] As a preferred embodiment of the deep learning-based dexterity hand action recognition method of the present invention, the construction process of the multimodal spatiotemporal graph attention network includes the following specific steps:

[0027] The graph neural network is used to propagate information and update nodes on the spatial and temporal edges of the action spatiotemporal graph, and aggregate spatiotemporal information.

[0028] The aggregated temporal information is passed to the attention mechanism, which dynamically allocates attention weights between graph nodes, adjusts the degree of feature fusion, and optimizes information transmission based on spatiotemporal changes and node importance to obtain a multimodal spatiotemporal graph attention network.

[0029] As a preferred embodiment of the deep learning-based dexterity hand gesture recognition method of the present invention, the method involves inputting spatiotemporal fusion feature representations into a temporal convolutional network, employing multi-scale dilated convolution to simultaneously model the short-term variation features and long-term evolution features of hand gestures in the time dimension, and outputting temporal action features. The specific steps are as follows:

[0030] The spatiotemporal fusion feature representation is organized into a time series format and input into a temporal convolutional network. Through convolutional kernels with different dilation rates, in each dilated convolutional layer, a smaller dilation rate is used to capture subtle changes in the short term to obtain short-term change features, while a larger dilation rate is used to capture long-term evolution features.

[0031] Short-term change features and long-term evolution features are passed to the fully connected layer, and then converted into fixed-dimensional temporal action features through the fully connected layer.

[0032] As a preferred embodiment of the dexterity hand action recognition method based on deep learning described in this invention, the method of obtaining dexterity hand action recognition results refers to inputting temporal action features into a Softmax classifier, calculating the probability value of each gesture category based on the temporal action features, and selecting the gesture category with the highest probability value.

[0033] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the deep learning-based dexterous hand action recognition method described in the first aspect of the present invention.

[0034] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the deep learning-based dexterity hand action recognition method described in the first aspect of the present invention.

[0035] The beneficial effects of this invention are as follows: by fusing visual, inertial, and depth information to construct multimodal motion perception data, and using spatiotemporal structured modeling to generate a motion spatiotemporal graph, the spatial structural relationship and temporal evolution process of dexterous hand movements can be uniformly expressed; furthermore, by utilizing a multimodal spatiotemporal graph attention network to adaptively allocate attention weights according to the temporal change state of the movement and perform feature fusion, and combining multi-scale temporal convolution to jointly model the short-term changes and long-term evolution of the movement, action features with good discriminative power are obtained, thereby achieving stable recognition of dexterous hand movements. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a deep learning-based dexterity hand action recognition method.

[0038] Figure 2 A flowchart for obtaining the 3D coordinates of key points on the hand.

[0039] Figure 3 A flowchart for constructing an action spacetime graph.

[0040] Figure 4 This is a flowchart of a multimodal spatiotemporal graph attention network and feature fusion.

[0041] Figure 5 This is a spatiotemporal graph attention weight allocation analysis diagram.

[0042] Figure 6 This is a performance comparison chart showing the impact of different convolutional kernel sizes on recognition accuracy. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0046] Reference Figures 1-6 This is one embodiment of the present invention, which provides a method for recognizing dexterous hand movements based on deep learning, including the following steps:

[0047] S1. Collect visual images, inertial data, and depth information of the dexterous hand, convert the visual images into three-dimensional coordinates of key hand points, perform noise reduction and correction on the inertial data, and perform illumination compensation and depth map processing on the depth information to obtain multimodal motion perception data.

[0048] S1.1 Acquire visual images, inertial data, and depth information of the dexterous hand, perform time alignment processing on the visual images, inertial data, and depth information, locate the hand region on the time-aligned visual images, and extract the two-dimensional position information of each key point of the hand.

[0049] Furthermore, visual images, inertial data, and depth information of the dexterous hand are acquired by an RGB camera, an inertial measurement unit, and a depth camera, respectively, and timestamps are uniformly written to the three raw data streams. A reference timestamp sequence is established based on the visual image timestamps, interpolation matching is performed on the inertial data, and the most recent timestamp matching is performed on the depth information to achieve time alignment of the three raw data streams. The time-aligned visual image is then located and cropped to obtain the hand region image. Hand keypoint detection is performed on the hand region image, and the two-dimensional pixel coordinates of the detected keypoints are mapped back to the original visual image coordinate system to obtain the two-dimensional position information of each hand keypoint.

[0050] S1.2 Based on the depth information at the corresponding time, the two-dimensional position information of the hand key points is mapped to the three-dimensional coordinates of the hand key points in the spatial coordinate system.

[0051] Furthermore, the depth value corresponding to the two-dimensional position information of each hand key point is queried; the two-dimensional pixel coordinates and depth values ​​are back-projected together with the depth camera intrinsic parameters to obtain the three-dimensional coordinates of the hand in the depth camera coordinate system; and then the three-dimensional coordinates of the hand are transformed to a unified spatial coordinate system using the camera extrinsic parameters to obtain the three-dimensional coordinates of the hand key points.

[0052] S1.3. After time alignment, the inertial data is processed by outlier removal and signal smoothing to eliminate random noise in the inertial data, and zero bias correction and drift correction are performed to obtain the corrected inertial data.

[0053] Furthermore, outlier removal is first performed on the time-aligned inertial data, identifying and deleting or replacing abrupt change data based on sliding window statistical criteria; then, signal smoothing is performed on the outlier-removed inertial data, using low-pass filtering or moving average filtering to suppress high-frequency random noise; subsequently, zero-bias acceleration and zero-bias angular velocity are calculated based on stationary segment data or calibration parameters and zero-bias correction is performed; finally, drift correction is performed on the zero-bias-corrected inertial data, using integral constraints to compensate for the offset accumulated over time, and the corrected inertial data is output.

[0054] It should be noted that integral constraints refer to the constraints imposed on the integration results during the integration of inertial data, in combination with known physical conditions or prior motion, in order to suppress the accumulation of errors over time. Integral constraints are usually established and used to correct the integral results through known motion states (such as zero velocity and continuous attitude changes during the stationary phase) or reference information provided by external sensors (such as pose change information acquired by vision sensors, spatial distance constraint information provided by depth sensors, or reference motion state information output by other position and attitude measurement devices).

[0055] The sliding window statistical criterion refers to a set of rules for identifying outliers in inertial data time series. It involves using a fixed-length or adaptive-length time window to perform statistical characteristic calculations on the data samples within the window and judging whether the current data point deviates from the normal distribution range based on the statistical results.

[0056] The sliding window statistical criterion is formed by setting a fixed-length sliding time window in the inertial data time series, calculating the statistical characteristics of the inertial data samples within the window, and combining the noise distribution characteristics of the inertial sensor and the data fluctuation range under normal motion conditions to set the corresponding deviation tolerance.

[0057] S1.4. After aligning the time, the depth information is reduced by suppressing brightness interference to minimize the impact of ambient light changes. Denoising and smoothing processes are then performed to fill in the missing depth areas and obtain continuous and stable depth map data.

[0058] Furthermore, the time-aligned depth information is first subjected to luminance interference suppression processing, which reduces the impact of ambient light changes on depth measurement by depth value normalization or luminance-related term compensation. Then, the depth information after luminance interference suppression is subjected to depth denoising and smoothing processing, which uses median filtering or bilateral filtering to eliminate discrete noise and preserve depth edges. Finally, the invalid pixels or hole regions in the processed depth map are filled with missing depth regions, and the missing depth values ​​are restored by neighborhood interpolation or spatial continuity constraints to obtain continuous and stable depth map data.

[0059] S1.5. The three-dimensional coordinates of the key hand points, the corrected inertial data, and the continuous stable depth map data are uniformly organized and output as multimodal motion perception data.

[0060] Furthermore, using a unified timestamp as an index, the three-dimensional coordinates of key hand points, the corrected inertial data, and the continuous stable depth map data at the same moment are matched accordingly. The three types of data are then standardized, organized, and encapsulated to form multimodal motion perception data.

[0061] S2. Perform spatiotemporal structured modeling on multimodal motion perception data to generate a motion spatiotemporal graph.

[0062] S2.1. Use the three-dimensional coordinates of the hand key points in the multimodal motion perception data as graph nodes, and add the corrected inertial data as graph node attributes to form a preliminary graph structure.

[0063] Furthermore, a "graph node set" is established using the single-frame timestamp of the multimodal motion perception data as the unit. The three-dimensional coordinates of each hand key point under the timestamp are instantiated as a graph node, and a unique key point number and corresponding three-dimensional coordinate value are assigned to each graph node. Then, the corrected inertial data under the same timestamp are appended to the graph node attribute vector according to the preset association rules (for example, sharing the same inertial attribute for all graph nodes, or mapping to the corresponding key point node according to the local area of ​​the hand), thereby obtaining a preliminary graph structure containing graph nodes and graph node attributes.

[0064] It should be noted that the association rules are a set of rules used to describe the correspondence between the corrected inertial data and the key point graph nodes of the hand, and are used to determine the way inertial data is attached as a graph node attribute; the association rules are generated based on the fixed installation position of the inertial measurement unit, the measurement axis direction of the inertial measurement unit, and the geometric correspondence between it and the spatial structure of the hand.

[0065] S2.2. Define the connection relationship between each pair of adjacent graph nodes according to the hand skeleton structure, and construct the spatial edge by calculating the Euclidean distance and relative motion angle between graph nodes.

[0066] Furthermore, a "skeleton adjacency list" is established based on the predefined hand skeleton structure. Each pair of adjacent graph nodes is enumerated in the skeleton adjacency list and the connection relationship is determined. For each pair of adjacent graph nodes, the Euclidean distance is calculated based on the three-dimensional coordinates of the hand key points of the two graph nodes as the spatial edge length feature, and the relative motion angle is calculated based on the displacement vector of the adjacent graph nodes under adjacent timestamps as the spatial edge direction feature. The connection relationship, Euclidean distance and relative motion angle are encapsulated together as a "spatial edge".

[0067] It should be noted that the hand skeleton structure is a structured representation of the anatomical connections between key points of the hand, including the numbering information of each key point and its adjacent connections. This hand skeleton structure is generated based on a pre-stored topological definition table of key points set according to the hand's anatomical structure and the spatial arrangement rules of key points. Among them, the spatial arrangement rules of key points are a set of rules used to describe the relative spatial position and topological order of each key point in the overall hand structure, which are used to constrain the spatial adjacency and hierarchical relationships between key points.

[0068] The spatial arrangement rules of key points are defined and generated based on the connection sequence between joints and phalanges in the anatomical structure of the hand and the relative positional relationship of key points in actual space.

[0069] The Euclidean distance is calculated based on the 3D coordinates of the hand keypoints in the two graphs, using the following expression:

[0070] ;

[0071] In the formula, Graph nodes With graph nodes The Euclidean distance of the corresponding hand key points in the spatial coordinate system; Graph nodes The corresponding hand key points in the spatial coordinate system Axis coordinate components; Graph nodes The corresponding hand key points in the spatial coordinate system Axis coordinate components; Graph nodes The corresponding hand key points in the spatial coordinate system Axis coordinate components; Graph nodes The corresponding hand key points in the spatial coordinate system Axis coordinate components; Graph nodes The corresponding hand key points in the spatial coordinate system Axis coordinate components; Graph nodes The corresponding hand key points in the spatial coordinate system Axis coordinate components; It is the first in the hand skeletal structure The index identifier of each graph node; Graph nodes The adjacent connected first The index identifier of each graph node;

[0072] The relative motion angle is calculated based on the displacement vectors of adjacent graph nodes at adjacent timestamps, and the expression is:

[0073] ;

[0074] In the formula, Graph nodes With graph nodes The relative motion angle at adjacent timestamps; Graph nodes The displacement vector at adjacent timestamps, the displacement vector is determined by graph nodes. The difference in three-dimensional coordinates of key hand points corresponding to adjacent timestamps is calculated. Graph nodes The displacement vector at adjacent timestamps, the displacement vector is determined by graph nodes. The difference in three-dimensional coordinates of key hand points corresponding to adjacent timestamps is calculated. Graph nodes The magnitude of the displacement vector is used to represent the graph node. The displacement magnitude between adjacent timestamps; Graph nodes The magnitude of the displacement vector is used to represent the graph node. The displacement magnitude between adjacent timestamps;

[0075] When a graph node is in a near-static state between adjacent timestamps, its displacement vector magnitude may be close to zero. To avoid numerical instability or undefinable situations during the calculation of relative motion angles, a minimum effective displacement threshold is set for the displacement amplitude (by statistically analyzing the amplitude range of key point displacement noise when the hand is in a static state, and combining it with the sensor measurement accuracy, it is set to a minimum effective displacement value that is higher than the upper limit of static noise and lower than the displacement amplitude of normal movement). When the displacement amplitude is less than the minimum effective displacement threshold, the calculation of the corresponding angle feature is skipped to ensure the stability of the spatial edge feature calculation process.

[0076] S2.3 Connect the graph nodes in the preliminary graph structure using spatial edges, and attach continuous and stable depth map data to generate the initial spatial graph.

[0077] Furthermore, based on spatial edges, each pair of graph nodes that satisfy the adjacency relationship of the hand skeleton structure, along with their corresponding Euclidean distance and relative motion angle features, are uniformly numbered and encapsulated. A set of spatial edges containing node pair indices and spatial feature parameters is formed according to the node index pairs. The graph nodes in the initial graph structure are connected one by one according to the connection relationship in the set of spatial edges to form a graph topology containing graph nodes, graph node attributes, and spatial edges. The continuous stable depth graph data corresponding to the timestamp is written into the data field of the initial spatial graph (for example, as a full-graph-level attribute or as local depth feature input of graph nodes), thereby generating an initial spatial graph containing spatial connection relationships and depth information.

[0078] S2.4 In the initial spatial graph, time edges are formed by connecting graph nodes in different time frames, and the time edges are integrated into the initial spatial graph to obtain the action spatiotemporal graph.

[0079] Furthermore, using the baseline timestamp sequence as a time index, a "time edge set" is established between adjacent timestamps in the initial spatial graph, and the same key point is numbered at the timestamp. Graph nodes and timestamps The corresponding graph nodes are connected, and time-related features such as time interval and node displacement are written for each time edge. Then, the time edge set and the spatial edge set in the initial spatial graph are merged into the same graph data structure, so that the graph contains both spatial edges and time edges, thus obtaining the action spatiotemporal graph.

[0080] S3. Input the action spatiotemporal graph into the multimodal spatiotemporal graph attention network, assign attention weights to graph nodes according to the temporal change state of hand actions, and perform attention-weighted fusion of visual features and inertial features to obtain spatiotemporal fusion feature representation.

[0081] It should be noted that the multimodal spatiotemporal graph attention network is composed of a graph neural network and an attention mechanism in a hierarchical structure. The graph neural network is located in the preceding layer, receiving the action spatiotemporal graph and performing spatiotemporal aggregation of graph node features based on spatial and temporal edges, outputting a graph node feature sequence containing spatial relationship information and temporal evolution information. The attention mechanism is located in the subsequent layer, taking the graph node feature sequence output by the graph neural network of the previous layer as input, and performing attention weight calculation and weighted fusion on the graph node feature sequence without changing the topology of the action spatiotemporal graph. This allows the spatiotemporal features output by the graph neural network to further distinguish the importance of nodes and emphasize temporal changes in the hierarchical connection, thereby forming a unified multimodal spatiotemporal graph attention network through the connection of the preceding and following layers.

[0082] S3.1. Information propagation and node updates are performed on the spatial and temporal edges of the action spatiotemporal graph through a graph neural network, and spatiotemporal information is aggregated.

[0083] Furthermore, after inputting the action-temporal graph into the graph neural network, the graph neural network uses the spatial edges and temporal edges in the action-temporal graph as information transmission paths to perform information propagation and node updating for each graph node: In the information propagation stage, graph node attributes and edge association features of adjacent graph nodes are collected from graph nodes connected at both ends of the spatial edge, and graph node attributes and edge association features of graph nodes corresponding to adjacent time frames are collected from graph nodes connected at both ends of the temporal edge. In the information propagation stage, neighborhood aggregation operations are performed on the spatial edge neighborhood information and the temporal edge neighborhood information respectively to obtain spatial edge aggregation features and temporal edge aggregation features; In the node updating stage, the spatial edge aggregation features and temporal edge aggregation features are fused with the original graph node attributes to perform feature fusion operations and output the updated graph node features. The updated graph node features continue to perform multiple rounds of information propagation and node updating within the action-temporal graph according to the temporal order and spatial adjacency relationship, thereby completing the joint aggregation of spatial edge information and temporal edge information and obtaining aggregated spatiotemporal information.

[0084] It should be noted that the action spatiotemporal graph is input into the graph neural network. The graph neural network completes the graph node feature propagation and node update based on spatial and temporal edges, and outputs spatiotemporal feature representation. The spatiotemporal feature representation is input into the Softmax classifier to obtain the gesture category probability distribution. The cross-entropy loss is calculated between the gesture category probability distribution and the gesture category labeled in the training samples. The network parameters of the graph neural network and the Softmax classifier are updated according to the backpropagation mechanism. The above process is repeated until the loss function converges and the graph neural network training is completed.

[0085] S3.2. The aggregated temporal information is passed to the attention mechanism, which dynamically allocates the attention weights between graph nodes, adjusts the degree of feature fusion, and optimizes information transmission according to spatiotemporal changes and node importance to obtain a multimodal spatiotemporal graph attention network.

[0086] Furthermore, the aggregated spatiotemporal information output by the graph neural network includes temporal information obtained by aggregating across time edges. This temporal information is input into the attention mechanism, which then calculates attention weights for each pair of graph nodes in the action-spatiotemporal graph that have spatial or temporal edge connections. The attention weight calculation process consists of graph node feature similarity calculation and normalization calculation. Graph node feature similarity calculation obtains a similarity score through dot product or cosine similarity operations between graph node feature vectors. The normalization calculation converts the similarity score into attention weights using the Softmax function, with values ​​falling within the example (0,1) for the same neighborhood / same time frame. The weights sum to 1; the attention mechanism performs weighted summation and aggregation on the spatial edge neighborhood graph node features and the temporal edge neighborhood graph node features based on the attention weights, to obtain attention-weighted aggregated features, and then performs feature fusion operation with the original graph node features to form updated graph node features; the attention mechanism repeatedly performs attention weight calculation, attention weight aggregation and feature fusion operation on the continuous time series of the action spatiotemporal graph, so that the attention weight changes with the time information, and makes the attention weight reflect the importance of graph nodes under the difference in the contribution of graph node features, thereby optimizing the information transmission in the action spatiotemporal graph and obtaining a multimodal spatiotemporal graph attention network.

[0087] It should be noted that training the multimodal spatiotemporal graph attention network involves inputting the action spatiotemporal graph into a graph neural network to obtain aggregated spatiotemporal information. This aggregated spatiotemporal information is then input into the attention mechanism to obtain updated graph node features. These updated graph node features are then input into a softmax classifier to obtain the gesture category probability distribution. The gesture category probability distribution and the gesture categories labeled in the training samples are used to calculate the cross-entropy loss. The cross-entropy loss is used to update the parameters of the graph neural network and the attention mechanism through backpropagation, and the parameters of the softmax classifier are updated simultaneously. The training process iterates until the cross-entropy loss converges, completing the training of the multimodal spatiotemporal graph attention network.

[0088] S3.3 Input the action spatiotemporal graph into the multimodal spatiotemporal graph attention network. Based on the spatiotemporal edges in the action spatiotemporal graph, calculate the change state of each graph node in the time series to obtain the temporal change state of hand actions.

[0089] Furthermore, after the action spatiotemporal graph is input into the multimodal spatiotemporal graph attention network, a time series association is established between graph nodes with the same keypoint number at adjacent timestamps based on the time edge set of the action spatiotemporal graph. The graph node feature sequence with the same keypoint number is extracted from the graph node features output by the multimodal spatiotemporal graph attention network in time-stamp order. Subsequently, difference operations or displacement vector calculations are performed on the graph node feature sequences at adjacent timestamps to obtain the feature changes of the graph nodes in the time series. The feature changes of the graph nodes (such as the difference magnitude, direction of change, or rate of change of the feature vectors of graph nodes at adjacent timestamps, and the change state of the corresponding graph node is formed by performing aggregation operations such as summation, averaging, or weighted accumulation in the time dimension) are aggregated in the time dimension to form the change state of the graph nodes. Finally, the change states of the graph nodes corresponding to all keypoint numbers are summarized to obtain the temporal change state of the hand action.

[0090] S3.4. Based on the temporal changes of hand movements, analyze the changes of each graph node in the time series, assign attention weights to the graph nodes, and perform dynamic feature fusion on visual features and inertial features according to the assigned attention weights to obtain spatiotemporal fusion feature representations.

[0091] Furthermore, based on the temporal changes in hand movements, the change amplitude and trend of the graph node corresponding to each key point number are calculated to obtain the graph node importance measurement result. This graph node importance measurement result is used as the weight modulation input for the attention mechanism. Within the graph node connection relationship defined by the spatial edge set and temporal edge set of the action spatiotemporal graph, the attention mechanism updates the attention weights between graph nodes based on the graph node importance measurement result. Statistical amplitude indicators (such as the mean, maximum, or energy value of the change) and trend indicators (such as the consistency of the change direction in consecutive time steps or the cumulative change direction) are calculated for the feature changes of graph nodes in the time series. When the change amplitude and change direction of a graph node remain consistent in multiple adjacent time steps, it is determined that the graph node has a large change amplitude or a significant change trend, thus assigning it a higher weight in the attention weight allocation. Subsequently, the updated attention weights are applied to the visual features and inertial features corresponding to the graph nodes, respectively, and attention weighted summation and fusion are performed to obtain a fused feature vector. The fused feature vector is then aggregated in the time dimension according to the timestamp order to finally obtain the spatiotemporal fused feature representation.

[0092] like Figure 5 The figure shows the weighting of different sensor data at various time points. As can be seen, the attention mechanism adaptively adjusts the importance of each sensor data source according to the temporal changes of the action. As the action progresses, visual and inertial data acquire different attention weights at different time periods, indicating that the most discriminative features can be adaptively focused on during recognition. This further verifies the contribution of the attention mechanism in this invention to improving the accuracy of dexterity hand action recognition.

[0093] S4. Input the spatiotemporal fusion feature representation into the temporal convolutional network, and use multi-scale dilated convolution to simultaneously model the short-term change features and long-term evolution features of gesture actions in the time dimension, output temporal action features, and obtain dexterity hand action recognition results through gesture category discrimination.

[0094] It should be noted that temporal convolutional networks are neural network structures that take time-series format features as input and use time-dimensional convolution operations as the core computation method. By performing one-dimensional convolution operations on the time axis, they model the feature changes of adjacent or cross-time steps in the sequence, thereby achieving effective representation of temporal patterns, trends and evolution laws.

[0095] Temporal convolutional networks consist of multiple layers of one-dimensional convolutional layers stacked in chronological order. Each convolutional layer uses dilated convolutional kernels with different dilation rates to form multi-scale temporal receptive fields. The lower convolutional layers focus on capturing local change features within short time windows in the time series, while the higher convolutional layers cover longer time spans with larger dilation rates to model long-term evolutionary features. The outputs of each layer are passed step by step in the network to form a hierarchical representation of temporal features.

[0096] During the training phase, the spatiotemporal fusion feature representations corresponding to the samples are organized into a time series format and input into a temporal convolutional network. The temporal convolutional network outputs temporal action features through forward propagation. The temporal action features are input into a Softmax classifier to obtain the gesture category probability distribution. The gesture category probability distribution and the sample labeled gesture category are used to calculate the cross-entropy loss. The parameters of the temporal convolutional network and the Softmax classifier are updated through the backpropagation mechanism. The above process is repeated until the loss function converges and the training of the temporal convolutional network is completed.

[0097] S4.1. Organize the spatiotemporal fusion feature representation into a time series format and input it into a temporal convolutional network. Through convolutional kernels with different dilation rates, in each dilated convolutional layer, a smaller dilation rate is used to capture subtle changes in the short term to obtain short-term change features, while a larger dilation rate is used to capture long-term evolution features.

[0098] Furthermore, the spatiotemporal fusion feature representation is sorted according to the baseline timestamp sequence and spliced ​​frame by frame according to the timestamp to form a time-series format input tensor with the time dimension as the main axis. After the time-series format input tensor is input into the temporal convolutional network, the temporal convolutional network performs one-dimensional convolution operation on the time-series format input tensor in each layer of dilated convolution, and expands the sampling interval of the convolution kernel in the time dimension by setting different dilation rates. Dilated convolution with a smaller dilation rate performs convolution accumulation on adjacent or near neighboring time steps to quantify feature differences within a short time window and output short-term change features. Dilated convolution with a larger dilation rate performs convolution accumulation over a longer time span to quantify feature evolution within a long time window and output long-term evolution features. Short-term change features and long-term evolution features are jointly represented in the hierarchical output of the temporal convolutional network and used as input for subsequent temporal action feature extraction.

[0099] like Figure 6 As shown, by comparing temporal convolutional networks using small convolutional kernels (short-term features) and large convolutional kernels (long-term features), we can see the impact of kernel size on the modeling of short-term and long-term action features. An appropriate kernel size can simultaneously capture both short-term action changes and long-term action evolution, improving recognition accuracy. The different kernel configurations in the figure demonstrate the advantages of temporal convolutional networks in handling action features at different time scales in dexterity hand action recognition.

[0100] S4.2. Transmit short-term change features and long-term evolution features to the fully connected layer, and convert them into fixed-dimensional temporal action features through the fully connected layer.

[0101] Furthermore, short-term change features and long-term evolution features are concatenated along the feature dimension to form a unified temporal feature vector. The temporal feature vector is then input into a fully connected layer, which performs linear transformations and nonlinear mapping operations on the temporal feature vector to compress or map the original high-dimensional temporal feature space to a dimensional space, outputting fixed-dimensional temporal action features.

[0102] S4.3 Input the temporal action features into the Softmax classifier, calculate the probability value of each gesture category based on the temporal action features, and select the gesture category with the highest probability value to obtain the dexterity hand action recognition result.

[0103] Furthermore, fixed-dimensional temporal action features are passed as input to the Softmax classifier. A linear weighted summation operation is performed on the temporal action features and the classification weight matrix to obtain the original classification score for each gesture category. Subsequently, an exponential operation is performed on the original classification scores of each gesture category and normalized to calculate the probability value of each gesture category under the current temporal action feature conditions. The probability values ​​of all gesture categories are compared, and the gesture category with the highest probability value is selected as the dexterity hand action recognition result.

[0104] Calculate the probability value of each gesture category under the current temporal action feature conditions, as expressed by:

[0105] ;

[0106] In the formula, Under the current temporal action characteristics, the first The probability values ​​corresponding to each gesture category; It is a gesture category index, used to identify the gesture category whose probability is currently being calculated; It is the first Original classification scores for each gesture category Exponentialized score after transformation by the natural exponential function; It is the first Original classification scores for each gesture category Exponentialized score after transformation by the natural exponential function; It is a gesture category index, used to traverse all gesture categories; It is the total number of gesture categories;

[0107] It should be noted that the Softmax classifier is a classification method for multi-class discrimination. It converts the input features into probability values ​​corresponding to each class, so that the output results of each class have comparable probabilistic meanings, and the class with the highest probability value is taken as the final discrimination result.

[0108] This embodiment also provides a computer device applicable to the dexterous hand action recognition method based on deep learning, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the dexterous hand action recognition method based on deep learning as proposed in the above embodiment.

[0109] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0110] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the deep learning-based dexterity hand gesture recognition method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0111] In summary, this invention constructs multimodal motion perception data by fusing visual, inertial, and depth information, and generates a spatiotemporal motion graph using spatiotemporal structured modeling, thereby unifying the spatial structural relationships and temporal evolution of dexterous hand movements. Furthermore, it utilizes a multimodal spatiotemporal graph attention network to adaptively allocate attention weights based on the temporal changes of the motion and perform feature fusion, combined with multi-scale temporal convolution to jointly model short-term changes and long-term evolution of the motion, obtaining motion features with good discriminative power, and achieving stable recognition of dexterous hand movements.

[0112] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for dexterous hand action recognition based on deep learning, characterized in that: include, The system collects visual images, inertial data, and depth information of the dexterous hand, converts the visual images into three-dimensional coordinates of key hand points, performs noise reduction and correction on the inertial data, performs illumination compensation and depth map processing on the depth information, and obtains multimodal motion perception data. Spatiotemporal structured modeling is performed on multimodal motion perception data to generate a motion spatiotemporal graph. The specific steps are as follows. The three-dimensional coordinates of hand key points in multimodal motion perception data are used as graph nodes, and the corrected inertial data is added as graph node attributes to form a preliminary graph structure. The connection relationship between each pair of adjacent graph nodes is defined according to the hand skeleton structure. Spatial edges are constructed by calculating the Euclidean distance and relative motion angle between graph nodes. The initial spatial graph is generated by connecting the graph nodes in the preliminary graph structure using spatial edges and attaching continuous and stable depth map data. In the initial spatial graph, time edges are formed by connecting graph nodes at different time frames, and these time edges are integrated into the initial spatial graph to obtain the action spatiotemporal graph; The action spatiotemporal graph is input into the multimodal spatiotemporal graph attention network. Attention weights are assigned to graph nodes according to the temporal changes of hand movements. Attention weighted fusion is performed on visual features and inertial features to obtain spatiotemporal fusion feature representations. The spatiotemporal fusion feature representation is input into the temporal convolutional network. Multi-scale dilated convolution is used to simultaneously model the short-term change features and long-term evolution features of hand gestures in the time dimension, outputting temporal action features. The dexterity hand action recognition results are obtained through gesture category discrimination. The process involves inputting spatiotemporal fusion feature representations into a temporal convolutional network, employing multi-scale dilated convolutions to simultaneously model both short-term and long-term evolutionary features of hand gestures in the temporal dimension, and outputting temporal action features. The specific steps are as follows: The spatiotemporal fusion feature representation is organized into a time series format and input into a temporal convolutional network. In each layer of dilated convolution, the temporal convolutional network performs one-dimensional convolution operation on the time series format input tensor. By setting different dilation rates, the sampling interval of the convolution kernel in the time dimension is expanded. The dilated convolution with a smaller dilation rate performs convolution accumulation on adjacent time steps to quantify the feature differences within a short time window and output short-term change features. The dilated convolution with a larger dilation rate performs convolution accumulation on a longer time span to quantify the feature evolution within a long time window and output long-term evolution features. Short-term change features and long-term evolution features are passed to the fully connected layer. The fully connected layer performs linear transformation and nonlinear mapping operations and compresses the space into a dimensional space to output temporal action features.

2. The deep learning-based dexterity hand action recognition method as described in claim 1, characterized in that: The process of acquiring visual images, inertial data, and depth information of the dexterous hand, and converting the visual images into three-dimensional coordinates of key hand points, involves the following steps. Visual images, inertial data, and depth information of the dexterous hand are collected, and the visual images, inertial data, and depth information are time-aligned. The hand region is located in the time-aligned visual images, and the two-dimensional position information of each key point of the hand is extracted. Based on the depth information at the corresponding time, the two-dimensional position information of the hand key points is mapped to the three-dimensional coordinates of the hand key points in the spatial coordinate system.

3. The dexterous hand action recognition method based on deep learning as described in claim 1, characterized in that: The process of performing denoising and correction on the inertial data, and performing illumination compensation and depth map processing on the depth information to obtain multimodal motion perception data, is detailed below. The time-aligned inertial data is processed by outlier removal and signal smoothing to eliminate random noise, and zero-bias correction and drift correction are performed to obtain the corrected inertial data. The time-aligned depth information is reduced by suppressing brightness interference to minimize the impact of ambient light changes, and noise reduction and smoothing are performed to fill in the missing depth areas, thus obtaining continuous and stable depth map data. The three-dimensional coordinates of key hand points, the corrected inertial data, and the continuous stable depth map data are uniformly organized to output multimodal motion perception data.

4. The deep learning-based dexterity hand action recognition method as described in claim 1, characterized in that: The process involves inputting the action spatiotemporal graph into a multimodal spatiotemporal graph attention network, assigning attention weights to graph nodes based on the temporal changes in hand movements, and performing attention-weighted fusion of visual and inertial features to obtain a spatiotemporal fusion feature representation. The specific steps are as follows: The action spatiotemporal graph is input into the multimodal spatiotemporal graph attention network. Based on the spatiotemporal edges in the action spatiotemporal graph, the change state of each graph node in the time series is calculated to obtain the temporal change state of hand actions. Based on the temporal changes of hand movements, the changes of each graph node in the time series are analyzed, attention weights are assigned to the graph nodes, and dynamic feature fusion is performed on visual features and inertial features according to the assigned attention weights to obtain spatiotemporal fusion feature representations.

5. The deep learning-based dexterity hand action recognition method as described in claim 4, characterized in that: The construction process of the multimodal spatiotemporal graph attention network includes the following specific steps: The graph neural network is used to propagate information and update nodes on the spatial and temporal edges of the action spatiotemporal graph, and aggregate spatiotemporal information. The aggregated temporal information is passed to the attention mechanism, which dynamically allocates attention weights between graph nodes, adjusts the degree of feature fusion, and optimizes information transmission based on spatiotemporal changes and node importance to obtain a multimodal spatiotemporal graph attention network.

6. The deep learning-based dexterity hand action recognition method as described in claim 1, characterized in that: The process of obtaining dexterous hand gesture recognition results involves inputting temporal action features into a Softmax classifier, calculating the probability value of each gesture category based on the temporal action features, and selecting the gesture category with the highest probability value.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the deep learning-based dexterity hand action recognition method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the deep learning-based dexterity hand action recognition method according to any one of claims 1 to 6.

Images