A hand motion recognition method for human-computer interaction

Abstract

A hand action recognition method for human-computer interaction first fully exploits the visual features of video frame sequences, dynamically extracting inter-frame motion information and frame appearance features through an inter-frame motion attention mechanism. It then enhances temporal features by combining intra-frame static attention with inter-frame dynamic attention. Finally, it uses a CTC beam search decoding method to predict the corresponding machine instructions from the video, transmits these instructions to a robot, and the robot performs the corresponding actions according to the instructions. This method considers both inter-frame motion information and frame appearance features in hand action recognition, fully utilizing the temporal information of hand action videos, and improves the applicability and accuracy of hand action recognition for human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the field of motion recognition technology, and in particular to a hand motion recognition method oriented to human-computer interaction. Background Art

[0002] With the rapid development of artificial intelligence and robotics, the ways of interaction between humans and robots have become more and more extensive. Among them, using human movements to control robots has become one of the key issues in the current field of human-computer interaction.

[0003] Traditional gesture recognition typically relies on contact-based recognition, using data gloves or embedded sensors installed on the human body to capture motion information. While this method offers good recognition accuracy, it requires the user to wear a sensor device, resulting in limited flexibility and high equipment costs. With the rise of contactless recognition technology, machine vision-based motion recognition is becoming a mainstream approach. As an important interaction method, hand gesture recognition enables more intuitive control commands by capturing and understanding user gestures or movements. For robots, gesture recognition technology not only enhances the user experience but also increases the robot's adaptability and flexibility in complex environments.

[0004] In the field of computer vision, hand motion recognition faces numerous challenges. The hand is complex in structure, possessing numerous joints and degrees of freedom. Its movement patterns are varied and rapid, and hand motion habits and morphology vary between individuals. Existing hand motion recognition technologies often suffer from low recognition accuracy and poor real-time performance when processing complex gestures or movements. This is especially true in dynamic environments, where rapid hand movements or interference often lead to suboptimal recognition results. Therefore, improving hand motion recognition accuracy and real-time responsiveness has become a pressing technical challenge in the field of robotics for human-computer interaction.

[0005] This paper proposes a new hand motion recognition method that, by combining deep learning and image processing technology, can accurately and efficiently identify complex hand motions, providing robots with a more intelligent and flexible operation mode when performing tasks. Summary of the Invention

[0006] In order to overcome the current shortcomings of low accuracy in recognizing complex hand movement patterns and low efficiency in analyzing the continuous changes of hand movements, and targeting the problems of temporal dynamics and association complexity in hand movement recognition tasks, the present invention proposes a hand movement recognition method for human-computer interaction, which combines the motion information and timing features between video frames and uses cross-frame motion trajectories to improve recognition accuracy.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A hand action recognition method for human-computer interaction first fully exploits the visual features of video frame sequences and dynamically extracts inter-frame motion information and appearance features through the inter-frame motion attention mechanism. Then, it enhances the temporal features by combining intra-frame static attention and inter-frame dynamic attention. Finally, it uses the CTC beam search decoding method to predict the corresponding machine instructions in the video, transmits the machine instructions to the robot, and the robot performs relevant actions according to the instructions.

[0009] Furthermore, the method comprises the following steps:

[0010] Step 1: A hand action video is represented as V motion ={I1, I2, ..., I i ,…,I T}, where I1 represents the first frame of the hand motion video, I2 represents the second frame of the hand motion video, and I i Indicates the i-th frame of the hand action video, I T Indicates the Tth frame of the hand action video, where T represents the number of frames; I i ∈R C×H×W , R represents Euclidean space, C represents the number of channels, H×W represents the spatial size of the frame; Z is the robot instruction data corresponding to the video;

[0011] Step 2: Hand action video V motion Perform downsampling data preprocessing to obtain processed video data V′ motion At the same time, the robot instruction data Z is processed by word segmentation to generate a processed robot instruction text set Z′={z1,z2,...,z i ,…,z t}, where z1 represents the first robot instruction corresponding to the hand motion video, z2 represents the second robot instruction corresponding to the hand motion video, and z i represents the i-th robot instruction corresponding to the hand action video, z t represents the t-th robot instruction corresponding to the hand action video;

[0012] Step 3: Use 2D convolutional neural network to extract video frame sequence V′ motion The initial characteristics After the pooling layer, the video frame feature set is obtained

[0013] Step 4: V motion Each frame in the video is divided into a region consisting of several neighborhood windows, and then the similarity of the corresponding regions between adjacent frames is calculated to obtain Vmotion The inter-frame attention map for each pair of adjacent frames in the video,

[0014]

[0015] Where k = 1, 2, .., T-1, Represents the similarity between the (i, j) point of the kth frame and the (i, j) point of the adjacent k+1th frame, i∈H, j∈W, N×N is the neighborhood window size, Represents the features of point (i, j) in the kth frame, represents the spatial dimension of Q and K, W Q , W K ∈R C×C is an adjustable linear matrix, n i，j is the coordinate of the neighborhood window where the adjacent frame (i, j) is located;

[0016] Step 5: Calculate V motion The similar appearance features of point (i, j) in the kth frame and point (i, j) in the adjacent k+1th frame in the video,

[0017]

[0018] in, is an adjustable linear matrix;

[0019] Step 6. Create an adjacent frame coordinate map B∈R 2×H×W , where the coordinate value of each position represents its relative position in the entire frame, calculate I k and I k+1 The motion vector between the point (i, j) in the kth frame and the point (i, j) in the adjacent k+1th frame,

[0020]

[0021] in, isI k The mapping coordinates of (i, j) in is the adjacent frame I k+1 The weighted mapping coordinates of the point in ;

[0022] Step 7: Calculate the initial frame I1 to the last frame I of the video T The complete motion trajectory of

[0023]

[0024] Step 8. Calculate the fusion feature f fused =αf appearance +βf trajectory, where α and β are fusion weight hyperparameters, f appearance By similar appearance features Calculated by 1D convolutional neural network, f trajectory From the complete motion trajectory MT i，j Calculated by BiLSTM network;

[0025] Step 9: Calculate based on fusion feature f fused Static attention,

[0026] SA=Conv 1×1 (DW_D Conv(DWConv(f fused )));

[0027] Among them, DW Conv represents small kernel depth convolution, DW_D Conv represents dilated depth convolution, Conv 1×1 Represents 1×1 convolution;

[0028] Step 10: Calculate based on fusion feature f fused Dynamic attention,

[0029] DA=FC(AvgPool(f fused ));

[0030] Among them, AvgPool represents average pooling, and FC represents the fully connected layer;

[0031] Step 11: Calculate the final attention.

[0032]

[0033] Step 12: Calculate the predicted distribution of hand movement video mapping to machine instructions.

[0034] P=softmax(FC(f fused ′));

[0035] Among them, the shape of P is T×C, C is the number of categories of machine instructions;

[0036] Step 13: Calculate the matching loss function between the predicted sequence P and the target robot instruction text set Z′.

[0037] L CTC = -logP(Z′|P);

[0038] Step 14: Repeat steps 3 to 14. CTC When it drops to the specified threshold, the calculation ends and the CTC beam search decoding method is used to give the currently predicted machine instruction.

[0039] MI = Decode(P);

[0040] Among them, Decode is the CTC beam search decoding method; then the machine instructions are transmitted to the robot, and the robot performs relevant actions according to the instructions.

[0041] The technical concept of the present invention is as follows: In the recognition of hand movements in human-computer interaction, the inter-frame motion information and appearance features are fully exploited, and the intra-frame static attention and inter-frame dynamic attention are used to enhance the timing features, thereby improving the accuracy of hand movement recognition.

[0042] The beneficial effects of the present invention are: it can improve the accuracy of hand movement recognition, comprehensively consider the cross-frame motion trajectory correlation information and the temporal feature enhancement in long-sequence tasks, and improve the applicability and accuracy of hand movement recognition for human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a hand motion recognition method for human-computer interaction; DETAILED DESCRIPTION

[0044] The present invention will be further described below with reference to the accompanying drawings.

[0045] Reference Figure 1 , a hand motion recognition method for human-computer interaction, comprising the following steps:

[0046] Step 1: A hand action video is represented as V motion ={I1, I2, ..., I i ,…,I T}, where I1 represents the first frame of the hand motion video, I2 represents the second frame of the hand motion video, and I i Indicates the i-th frame of the hand action video, I T Indicates the Tth frame of the hand action video, where T represents the number of frames; I i ∈R C×H×W , R represents Euclidean space, C represents the number of channels, H×W represents the spatial size of the frame; Z is the robot instruction data corresponding to the video;

[0047] Step 2: Hand action video V motion Perform downsampling data preprocessing to obtain processed video data V′ motion At the same time, the robot instruction data Z is processed by word segmentation to generate a processed robot instruction text set Z′={z1,z2,...,z i ,…,z t}, where z1 represents the first robot instruction corresponding to the hand motion video, z2 represents the second robot instruction corresponding to the hand motion video, and z irepresents the i-th robot instruction corresponding to the hand action video, z t represents the t-th robot instruction corresponding to the hand action video;

[0048] Step 3: Use 2D convolutional neural network to extract video frame sequence V′ motion The initial characteristics After the pooling layer, the video frame feature set is obtained

[0049] Step 4: V motion Each frame in the video is divided into a region consisting of several neighborhood windows, and then the similarity of the corresponding regions between adjacent frames is calculated to obtain V motion The inter-frame attention map for each pair of adjacent frames in the video,

[0050]

[0051] Where k = 1, 2, .., T-1, Represents the similarity between the (i, j) point of the kth frame and the (i, j) point of the adjacent k+1th frame, i∈H, j∈W, N×N is the neighborhood window size, Represents the features of point (i, j) in the kth frame, represents the spatial dimensions of Q and K, is an adjustable linear matrix, n i，j is the coordinate of the neighborhood window where the adjacent frame (i, j) is located;

[0052] Step 5: Calculate V motion The similar appearance features of point (i, j) in the kth frame and point (i, j) in the adjacent k+1th frame in the video,

[0053]

[0054] in, is an adjustable linear matrix;

[0055] Step 6. Create an adjacent frame coordinate map B∈R 2×H×W , where the coordinate value of each position represents its relative position in the entire frame, calculate I k and I k+1 The motion vector between the point (i, j) in the kth frame and the point (i, j) in the adjacent k+1th frame,

[0056]

[0057] in, isI k The mapping coordinates of (i, j) in is the adjacent frame Ik+1 The weighted mapping coordinates of the point in ;

[0058] Step 7: Calculate the initial frame I1 to the last frame I of the video T The complete motion trajectory of

[0059]

[0060] Step 8. Calculate the fusion feature f fused =αf appearance +βf trajectory , where α and β are fusion weight hyperparameters, f appearance By similar appearance features Calculated by 1D convolutional neural network, f trajectory From the complete motion trajectory MT i，j Calculated by BiLSTM network;

[0061] Step 9: Calculate based on fusion feature f fused Static attention,

[0062] SA=Conv 1×1 (DW_D Conv(DW Conv(f fused )));

[0063] Among them, DW Conv represents small kernel depth convolution, DW_D Conv represents dilated depth convolution, Conv 1×1 Represents 1×1 convolution;

[0064] Step 10: Calculate based on fusion feature f fused Dynamic attention,

[0065] DA=FC(AvgPool(f fused ));

[0066] Among them, AvgPool represents average pooling, and FC represents the fully connected layer;

[0067] Step 11: Calculate the final attention.

[0068]

[0069] Step 12: Calculate the predicted distribution of hand movement video mapping to machine instructions.

[0070] P=softmax(FC(f fused ′));

[0071] Among them, the shape of P is T×C, C is the number of categories of machine instructions;

[0072] Step 13: Calculate the matching loss function between the predicted sequence P and the target robot instruction text set Z′.

[0073] L CTC = -logP(Z′|P);

[0074] Step 14: Repeat steps 3 to 14. CTC When it drops to the specified threshold, the calculation ends and the CTC beam search decoding method is used to give the currently predicted machine instruction.

[0075] MI = Decode(P);

[0076] Among them, Decode is the CTC beam search decoding method; then the machine instructions are transmitted to the robot, and the robot performs relevant actions according to the instructions.

[0077] In this embodiment, for a hand action video representing "grabbing an object on the table with the right hand", the data is first preprocessed using a downsampling method, and the robot instruction data corresponding to the video is subjected to word segmentation text processing operations to generate a processed robot instruction text; the video frame features are obtained through a 2D convolutional neural network and a pooling layer; the inter-frame motion attention mechanism is used to dynamically calculate the motion vectors and similar appearance features of two adjacent frames, and the motion vectors are accumulated to obtain the complete motion trajectory of the entire video; the fusion features are calculated using the similar appearance features processed by the 1D convolutional neural network and the motion trajectory processed by BiLSTM; the dynamic attention and static attention are calculated based on the fusion features, and the two are combined to obtain the final attention; the final attention is used to calculate the predicted distribution of the hand action video mapped to the machine instruction, and the CTC beam search decoding method is used to give the current predicted machine instruction; finally, the identified machine instruction is transmitted to the robot, and the robot performs the action of "grabbing the object on the table with the right hand" according to the instruction.

[0078] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A hand motion recognition method for human-computer interaction, characterized in that: First, the visual features of the video frame sequence are fully exploited, and the inter-frame motion information and appearance features of the frames are dynamically extracted through the inter-frame motion attention mechanism. Then, the temporal features are enhanced by combining intra-frame static attention and inter-frame dynamic attention. Finally, the CTC beam search decoding method is used to predict the corresponding machine instructions of the video, and the machine instructions are transmitted to the robot, which performs the relevant actions according to the instructions. The method comprises the following steps: Step 1: A hand action video is represented as V motion ={I1,I2,…,I i ,…,I T }, where I1 represents the first frame of the hand motion video, I2 represents the second frame of the hand motion video, and I i Indicates the i-th frame of the hand action video, I T Indicates the Tth frame of the hand action video, where T represents the number of frames; I i ∈R C×H×W , R represents Euclidean space, C represents the number of channels, H×W represents the spatial size of the frame; Z is the robot instruction data corresponding to the video; Step 2: Hand action video V motion Perform downsampling data preprocessing to obtain processed video data V′ motion At the same time, the robot instruction data Z is processed by word segmentation to generate a processed robot instruction text set Z′={z1,z2,…,z i ,…,z t }, where z1 represents the first robot instruction corresponding to the hand motion video, z2 represents the second robot instruction corresponding to the hand motion video, and z i represents the i-th robot instruction corresponding to the hand action video, z t represents the t-th robot instruction corresponding to the hand action video; Step 3: Use 2D convolutional neural network to extract video frame sequence V′ motion The initial characteristics After the pooling layer, the video frame feature set is obtained Step 4: V motion Each frame in the video is divided into a region consisting of several neighborhood windows, and then the similarity of the corresponding regions between adjacent frames is calculated to obtain V motion The inter-frame attention map for each pair of adjacent frames in the video, Where k = 1, 2, .., T-1, Represents the similarity between the (i, j) point of the kth frame and the (i, j) point of the adjacent k+1th frame, i∈H, j∈W, N×N is the neighborhood window size, Represents the features of point (i, j) in the kth frame, represents the spatial dimensions of Q and K, is an adjustable linear matrix, n i,j is the coordinate of the neighborhood window where the adjacent frame (i, j) is located; Step 5: Calculate V motion The similar appearance features of point (i, j) in the kth frame and point (i, j) in the adjacent k+1th frame in the video, in, is an adjustable linear matrix; Step 6. Create an adjacent frame coordinate map B∈R 2×H×W , where the coordinate value of each position represents its relative position in the entire frame, calculate I k and I k+1 The motion vector between the point (i, j) in the kth frame and the point (i, j) in the adjacent k+1th frame, in, isI k The mapping coordinates of (i, j) in , is the adjacent frame I k+1 The weighted mapping coordinates of the point in ; Step 7: Calculate the initial frame I1 to the last frame I of the video T The complete motion trajectory of Step 8. Calculate the fusion feature f fused =αf appearance +βf trajectory , where α and β are fusion weight hyperparameters, f appearance By similar appearance features Calculated by 1D convolutional neural network, f trajectory From the complete motion trajectory MT i,j Calculated by BiLSTM network; Step 9: Calculate based on fusion feature f fused Static attention, SA=Conv 1×1 (DW_DConv(DWConv(f fused ))); Among them, DW Conv represents small kernel depth convolution, DW_D Conv represents dilated depth convolution, Conv 1×1 Represents 1×1 convolution; Step 10: Calculate based on fusion feature f fused Dynamic attention, DA=FC(AvgPool(f fused )); Among them, AvgPool represents average pooling, and FC represents the fully connected layer; Step 11: Calculate the final attention. Step 12: Calculate the predicted distribution of hand movement video mapping to machine instructions. P=softmax(FC(f fused ′)); Among them, the shape of P is T×C, C is the number of categories of machine instructions; Step 13: Calculate the matching loss function between the predicted sequence P and the target robot instruction text set Z′. L CTC =-logP(Z′|P); Step 14: Repeat steps 3 to 14. CTC When it drops to the specified threshold, the calculation ends and the CTC beam search decoding method is used to give the currently predicted machine instruction. MI = Decode(P); Among them, Decode is the CTC beam search decoding method; then the machine instructions are transmitted to the robot, and the robot performs relevant actions according to the instructions.

Images