Hand-object interaction recognition method, system and equipment
By constructing gesture description data sets and integrating skeleton, visual and linguistic information, the limitations of traditional hand-object interaction technology in identifying complex hand postures and object interaction details are solved, and more efficient hand-object interaction recognition performance is achieved.
Patent Information
- Application Number
- CN202411969011.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-05-02
AI Technical Summary
When traditional hand-object interaction technology recognizes the physical interaction between hand movements and objects, it lacks accurate descriptions of finger spatial positions, joint angles and subtle movements, resulting in limitations in identifying and understanding complex hand postures and object interaction details.
By constructing a gesture description data set, combining skeleton information, language and visual information, an initial hand-object interaction recognition model is established, and the skeleton features are processed and aligned through a multi-head self-attention mechanism and a multi-layer perception machine to achieve the fusion of hand skeleton mode, visual mode and language mode.
It improves the recognition ability of hand-object interaction tasks, and can better understand hand details, thereby improving the performance of hand-object interaction tasks.
Smart Images
Figure CN119920001A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of hand-object interaction technology, and in particular to a hand-object interaction recognition method, system and device. Background Art
[0002] Hand-object interaction technology enables natural interaction between people and virtual or physical devices by capturing hand movements or gestures. It is widely used in virtual reality, augmented reality, medical rehabilitation, smart home, Internet of Things, gaming and entertainment and other fields.
[0003] Traditional hand-object interaction technology uses visual language models to identify the physical interactions between hand movements and objects, such as identifying whether the hand is picking up, putting down, pushing or rotating an object. However, the traditional methods lack an accurate description of the spatial position, joint angles and subtle movements of the fingers, resulting in limitations in the model in recognizing and understanding complex hand postures and the details of the interaction with objects. Summary of the invention
[0004] The technical problem to be solved by the present invention is how to improve the recognition ability in hand-object interaction, and to provide a hand-object interaction recognition method, system and device that can combine skeleton information, language and visual information to better understand hand details, thereby improving the performance of hand-object interaction tasks.
[0005] In order to solve the above technical problems, an embodiment of the present invention provides a method for identifying hand-object interaction, comprising:
[0006] Constructing a gesture description data set; the gesture description data set includes skeleton description texts of a plurality of gesture postures;
[0007] According to the traditional visual language model, the hand skeleton modality, visual modality and language modality are integrated to establish the initial hand-object interaction recognition model;
[0008] Using the gesture description data set to train the initial hand-object interaction recognition model to obtain a hand-object interaction recognition model;
[0009] The image to be recognized is input into the hand-object interaction recognition model to obtain a description of the hand-object interaction action in the image to be recognized.
[0010] As an improvement of the above solution, the step of constructing a gesture description dataset includes:
[0011] Collecting a plurality of gesture images, and identifying palms, fingers, and finger joints in the plurality of gesture images;
[0012] Obtaining a first skeleton description text of the gesture image according to the direction of the palm;
[0013] Obtaining a second skeleton description text of the gesture image according to the positional relationship between the fingers;
[0014] Obtaining a third skeleton description text of the gesture image according to the flexion angle of the finger joint;
[0015] The first skeleton description text, the second skeleton description text, and the third skeleton description text are used to annotate the gesture image to obtain a gesture description data set.
[0016] As an improvement of the above solution, the method of fusing the hand skeleton modality, visual modality and language modality according to the traditional visual language model to establish an initial hand-object interaction recognition model includes:
[0017] Split the traditional visual language model into a visual processing submodule and a language processing submodule;
[0018] Input the image data to be recognized into the visual processing submodule to obtain entity data and skeleton data;
[0019] A multi-head self-attention mechanism is used to extract skeleton features from the skeleton data;
[0020] Mapping the skeleton features to the text dimension through a multi-layer perceptron to obtain skeleton alignment features;
[0021] The entity data and the skeleton alignment features are input into the language processing submodule to achieve the fusion of the hand skeleton modality, visual modality and language modality to obtain an initial hand-object interaction recognition model.
[0022] As an improvement of the above solution, the multi-head self-attention mechanism is used to extract skeleton features from the skeleton data, including:
[0023] According to the skeleton data, obtaining joint space coordinates;
[0024] According to the joint space coordinates, the skeleton data is segmented using a preset sliding window to obtain a plurality of space segments;
[0025] A multi-head self-attention mechanism is adopted to calculate the spatial labels of the several spatial segments through 3D convolution, normalization and activation function to obtain skeleton features.
[0026] As an improvement of the above scheme, the multi-head self-attention mechanism is used to calculate the spatial labels of the plurality of spatial segments through 3D convolution, normalization and activation function to obtain skeleton features, including:
[0027] Establishing a multi-head self-attention model, wherein the multi-head self-attention model includes a plurality of self-attention heads with different linear transformation parameters;
[0028] Input the plurality of spatial segments into the multi-head self-attention model, and compute the query, key, and value of the spatial segment by 3D convolution in each attention head;
[0029] Calculate the similarity between the query and the key to obtain an attention score;
[0030] According to the attention score, weighting the value to obtain a weighted feature vector;
[0031] Taking the activation value calculated by the activation function as a target within a preset activation value range, the weighted feature vector is normalized to obtain a normalized feature;
[0032] Performing residual connection and convolution on the normalized features output by each of the attention heads to obtain updated features;
[0033] The updated features are fed back to each of the attention heads through a feedforward neural network to obtain spatial labels of the plurality of spatial segments as skeleton features.
[0034] As an improvement of the above solution, the skeleton features are mapped to the text dimension through a multi-layer perceptron to obtain skeleton alignment features, including:
[0035] A linear transformation algorithm is performed on the skeleton features through a multi-layer perceptron to map the skeleton features to a high-level dimension to obtain a linear high-dimensional skeleton feature;
[0036] The linear high-dimensional skeleton features are nonlinearly transformed through the GELU activation function to obtain skeleton alignment features; the dimension of the skeleton alignment features is the same as the text dimension of the traditional visual language model.
[0037] As an improvement of the above solution, the method of using the gesture description data set to train the initial hand-object interaction recognition model to obtain the hand-object interaction recognition model includes:
[0038] According to the gesture description data set, a plurality of image-text pairs are obtained; each of the image-text pairs comprises a gesture image and a skeleton description text corresponding thereto;
[0039] Extracting skeleton points in the gesture image according to the gesture image to obtain a first skeleton feature;
[0040] Describing the text according to the skeleton, obtaining a first text feature;
[0041] Calculating the similarity between the first skeleton feature and the first text feature in the image-text pair to obtain a similarity matrix;
[0042] Calculating a bidirectional loss value between the first skeleton feature and the first text feature by using a cross entropy loss function;
[0043] The parameters of the multi-head self-attention mechanism and the multi-layer perceptron in the initial hand-object interaction recognition model are adjusted according to the bidirectional loss value to obtain a hand-object interaction recognition model.
[0044] As an improvement of the above solution, the calculating the similarity between the first skeleton feature and the first text feature in the image-text pair to obtain a similarity matrix includes:
[0045] The temperature parameter is introduced to adjust the smoothness of the similarity distribution;
[0046] Calculating the cosine similarity between the first skeleton feature and the first text feature according to the adjusted similarity distribution;
[0047] A similarity matrix is obtained according to the cosine similarity between the first skeleton feature and the first text feature in the same image-text pair.
[0048] The embodiment of the present invention further provides a hand-object interaction recognition system, including:
[0049] A data set construction module is used to construct a gesture description data set; the gesture description data set includes skeleton description texts of a plurality of gesture postures;
[0050] The initial model building module is used to fuse the hand skeleton modality, visual modality and language modality according to the traditional visual language model to establish an initial hand-object interaction recognition model;
[0051] A model training module, used to train the initial hand-object interaction recognition model using the gesture description data set to obtain a hand-object interaction recognition model;
[0052] The model application module is used to input the image to be recognized into the hand-object interaction recognition model to obtain the hand-object interaction action description in the image to be recognized.
[0053] An embodiment of the present invention also provides a hand-object interaction recognition device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the hand-object interaction recognition method as described above.
[0054] Compared with the prior art, the present invention discloses a method, system and device for identifying hand-object interaction, which constructs a gesture description dataset; the gesture description dataset includes skeleton description texts of several gesture postures; according to the traditional visual language model, the hand skeleton modality, visual modality and language modality are fused to establish an initial hand-object interaction recognition model; the gesture description dataset is used to train the initial hand-object interaction recognition model to obtain the hand-object interaction recognition model; the image to be recognized is input into the hand-object interaction recognition model to obtain the description of the hand-object interaction action in the image to be recognized. By adopting the embodiments of the present invention, the skeleton information, language and visual information in the image to be recognized can be combined to better understand the hand details, thereby improving the performance of the hand-object interaction task. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a schematic diagram of the steps of a method for identifying hand-object interaction provided by an embodiment of the present invention;
[0056] Figure 2 It is a structural diagram of a hand-object interaction recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] In the description of the specification and claims, it is to be understood that the terms first, second, etc. in the specification and claims are only used for the purpose of describing the same technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor necessarily describing the order or time sequence. The terms are interchangeable where appropriate. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features.
[0059] Existing hand-object interaction technology relies on the development of cutting-edge technologies such as computer vision, sensor technology, machine learning and artificial intelligence, which can track hand movements in real time and accurately identify them. Computer vision combined with deep learning algorithms are used to accurately identify gestures and movements, while sensor technology achieves fine control by detecting hand movement, pressure and angle. At the same time, machine learning algorithms enhance the dynamic interaction capabilities of gesture recognition through big data training models, while tactile feedback technology further enhances the user's immersion and interactive experience. With the development of artificial intelligence, hand-object interaction will develop in a more natural, intelligent and device-free direction. In the future, direct gesture control without wearable devices may be realized, and combined with multimodal interactions such as voice and eye tracking to provide a personalized interactive experience.
[0060] Visual-Language Large Models (VLLMs) significantly improve the intelligence and flexibility of hand-object interaction tasks by combining visual information with natural language. VLLMs are mainly used in natural language instruction execution, semantic understanding and environmental perception, and reinforcement learning in hand-object interaction. They are widely used in scenarios such as intelligent robots, virtual assistants, augmented reality, and virtual reality. The core technologies include visual perception, cross-modal alignment and fusion, and fine-tuning through large-scale pre-trained models, which enable the system to have high generalization capabilities.
[0061] Traditional VLLMs usually focus on the semantic information of the global image and lack sufficient sensitivity to the subtle interactions between hands and objects. In order to compensate for the lack of detail capture, the model structure becomes more complex, resulting in increased computational costs and resource consumption, which is not conducive to real-time applications.
[0062] Based on the above considerations, an embodiment of the present invention provides a method for identifying hand-object interaction. Figure 1 In this embodiment, the hand-object interaction recognition method is specifically performed through steps S1 to S4:
[0063] S1. Constructing a gesture description dataset; the gesture description dataset includes skeleton description texts of a plurality of gesture postures.
[0064] It should be noted that the data set constructed in the embodiment of the present invention describes the hand gestures from the skeleton dimension. When dealing with hand-object interaction tasks, traditional visual language models can only capture the two-dimensional plane information of the hand, which leads to limitations in the model in recognizing and understanding complex hand gestures and interaction details with objects. By constructing a gesture description data set, the hand gestures and movements can be more accurate.
[0065] S2. Based on the traditional visual language model, the hand skeleton modality, visual modality and language modality are integrated to establish an initial hand-object interaction recognition model.
[0066] Traditional visual language models only process visual modalities and language modalities, and often lack accurate descriptions of the spatial position, joint angles, and subtle movements of fingers. The present application scheme integrates the hand skeleton modality into the traditional visual language model to obtain precise structural information of the hand in three-dimensional space, such as the position information and motion trajectory of each joint.
[0067] S3. Use the gesture description data set to train the initial hand-object interaction recognition model to obtain a hand-object interaction recognition model.
[0068] It should be noted that when training the initial hand-object interaction recognition model, a preferred solution is to only train the data processing and data fusion process of the hand skeleton modality, while freezing the original parameters in the traditional visual language model, which can effectively improve the model training speed.
[0069] S4. Input the image to be recognized into the hand-object interaction recognition model to obtain a description of the hand-object interaction action in the image to be recognized.
[0070] In the above scheme, an initial hand-object interaction recognition model of hand skeleton-vision-language tri-modality is formed by integrating the hand skeleton modality into the traditional visual language model, and the initial hand-object interaction recognition model is trained using the constructed gesture description dataset. The obtained hand-object interaction recognition model can combine the skeleton information, language and visual information in the image to be recognized to better understand the hand details, thereby improving the performance of hand-object interaction tasks.
[0071] As a preferred implementation, step S1, constructing a gesture description dataset, includes:
[0072] Collecting a plurality of gesture images, and identifying palms, fingers, and finger joints in the plurality of gesture images;
[0073] Obtaining a first skeleton description text of the gesture image according to the direction of the palm;
[0074] Obtaining a second skeleton description text of the gesture image according to the positional relationship between the fingers;
[0075] Obtaining a third skeleton description text of the gesture image according to the flexion angle of the finger joint;
[0076] The first skeleton description text, the second skeleton description text, and the third skeleton description text are used to annotate the gesture image to obtain a gesture description data set.
[0077] In the above scheme, the gesture description dataset describes the gesture posture through detailed skeleton description text, providing data support for the subsequent training of the initial hand-object interaction recognition model.
[0078] The embodiment of the present invention adopts the description of each skeleton joint to replace the traditional description of the overall hand posture, so that the skeleton description text has a higher degree of detail. Exemplarily, based on the analysis of the topological structure of the hand, the hand is divided into the palm and fingers, and the fingers are further subdivided into proximal joints and distal joints. The skeleton description text is composed by specifically describing the state of each finger joint and the relationship between the fingers. The description of the gesture posture includes a detailed description of each finger joint and the relationship between the fingers based on the flexion angle of the finger joints. In addition, it also includes a description of the palm direction and the overall gesture posture. In a specific embodiment, the skeleton text description is "The index finger is fully extended, while the base joints of the other four fingers are bent relative to the palm, and their tips are also bent. The overall posture resembles a pointing gesture.", which describes the relationship between the various parts of the hand and the overall hand posture.
[0079] As a preferred implementation, step S2, based on the traditional visual language model, fuses the hand skeleton modality, visual modality and language modality to establish an initial hand-object interaction recognition model, which is specifically performed through steps S21-S25:
[0080] S21, split the traditional visual language model into a visual processing submodule and a language processing submodule;
[0081] S22, inputting the image data to be recognized into the visual processing submodule to obtain entity data and skeleton data;
[0082] S23, using a multi-head self-attention mechanism to extract skeleton features from the skeleton data;
[0083] S24, mapping the skeleton features to the text dimension through a multi-layer perceptron to obtain skeleton alignment features;
[0084] S25. Input the entity data and the skeleton alignment features into the language processing submodule to achieve the fusion of the hand skeleton modality, visual modality and language modality to obtain an initial hand-object interaction recognition model.
[0085] It should be noted that the visual processing submodule corresponds to the visual modality, and the language processing submodule corresponds to the language modality. In abstract terms, the visual processing submodule can perform environmental perception and entity discrimination, and the language processing submodule is used to execute natural language instructions and semantic understanding. However, the traditional visual language model lacks sufficient sensitivity to the subtle interactions between hands and objects, and cannot perform fine processing on the image to be recognized.
[0086] It should also be noted that since the text and skeleton data come from different modalities and the text is a description of the skeleton, the text usually has a higher level of abstraction and requires unification of the dimensions of the skeleton features and the text modality. In an embodiment of the present invention, the dimensions of the skeleton features and the text dimensions are aligned through a multi-layer perceptron, and then the skeleton alignment features and entity data are input into the language processing submodule, which can achieve effective fusion between modalities.
[0087] Further, preferably, step S23, using a multi-head self-attention mechanism to extract skeleton features from the skeleton data, includes:
[0088] According to the skeleton data, obtaining joint space coordinates;
[0089] According to the joint space coordinates, the skeleton data is segmented using a preset sliding window to obtain a plurality of space segments;
[0090] A multi-head self-attention mechanism is adopted to calculate the spatial labels of the several spatial segments through 3D convolution, normalization and activation function to obtain skeleton features.
[0091] Furthermore, preferably, the multi-head self-attention mechanism is used to calculate the spatial labels of the plurality of spatial segments through 3D convolution, normalization and activation function to obtain skeleton features, including:
[0092] Establishing a multi-head self-attention model, wherein the multi-head self-attention model includes a plurality of self-attention heads with different linear transformation parameters;
[0093] Input the plurality of spatial segments into the multi-head self-attention model, and compute the query, key, and value of the spatial segment by 3D convolution in each attention head;
[0094] Calculate the similarity between the query and the key to obtain an attention score;
[0095] According to the attention score, weighting the value to obtain a weighted feature vector;
[0096] Taking the activation value calculated by the activation function as a target within a preset activation value range, the weighted feature vector is normalized to obtain a normalized feature;
[0097] Performing residual connection and convolution on the normalized features output by each of the attention heads to obtain updated features;
[0098] The updated features are fed back to each of the attention heads through a feedforward neural network to obtain spatial labels of the plurality of spatial segments as skeleton features.
[0099] In an embodiment of the present invention, in order to better extract information between skeleton joints, multiple multi-head self-attention modules are used. Preferably, the input of each module is the output of the previous layer, and the initial input is the preprocessed skeleton data. Each layer is first position-encoded and normalized to enhance the spatial relationship of feature expression. Position encoding encodes each mark through an encoding function so that the model can capture the relative position of each joint in the skeleton.
[0100] In each self-attention module, the query, key, and value are first calculated through 3D convolution. Then, the attention score is obtained by calculating the similarity between the query and the key, and the value vector is further weighted to highlight the more relevant parts of the input sequence. The weighted feature representation is obtained by weighted summation. In order to prevent gradient explosion, normalization operations are applied to the output of each layer during the calculation process to ensure that the activation values are within a reasonable range.
[0101] The output of each multi-head self-attention module is weighted and updated through residual connections and convolution operations. The updated feature representation is further extracted through a feed-forward neural network (FFN) and finally outputs enhanced skeleton features.
[0102] Preferably, step S24, mapping the skeleton features to text dimensions through a multi-layer perceptron to obtain skeleton alignment features, includes:
[0103] A linear transformation algorithm is performed on the skeleton features through a multi-layer perceptron to map the skeleton features to a high-level dimension to obtain a linear high-dimensional skeleton feature;
[0104] The linear high-dimensional skeleton features are nonlinearly transformed through the GELU activation function to obtain skeleton alignment features; the dimension of the skeleton alignment features is the same as the text dimension of the traditional visual language model.
[0105] It should be noted that the multilayer perceptron maps the skeleton features to a new feature space through linear transformation, while the GELU activation function further aligns the linear high-dimensional skeleton features with the dimensions of the text modality through nonlinear transformation, which can transform the skeleton features from low-level features to higher-level representations.
[0106] In an embodiment of the present invention, feature alignment is achieved through contrastive learning, which is different from traditional classification methods. Although traditional classification methods can also align text with other modalities to a certain extent, they cannot fully utilize the information contained in the text, nor can they effectively utilize unlabeled data. The contrastive learning method adopted by the present invention aligns by measuring the similarity between text and skeleton data in a shared feature space. This method does not rely on absolute labels, and can efficiently align skeleton features with text with similar features even when facing unseen data.
[0107] As a preferred implementation, step S3, using the gesture description data set to train the initial hand-object interaction recognition model to obtain the hand-object interaction recognition model, includes:
[0108] According to the gesture description data set, a plurality of image-text pairs are obtained; each of the image-text pairs comprises a gesture image and a skeleton description text corresponding thereto;
[0109] Extracting skeleton points in the gesture image according to the gesture image to obtain a first skeleton feature;
[0110] Describing the text according to the skeleton, obtaining a first text feature;
[0111] Calculating the similarity between the first skeleton feature and the first text feature in the image-text pair to obtain a similarity matrix;
[0112] Calculating a bidirectional loss value between the first skeleton feature and the first text feature by using a cross entropy loss function;
[0113] The parameters of the multi-head self-attention mechanism and the multi-layer perceptron in the initial hand-object interaction recognition model are adjusted according to the bidirectional loss value to obtain a hand-object interaction recognition model.
[0114] It should be noted that the traditional visual language model in the embodiment of the present invention does not participate in the alignment process. In order to maintain the original visual and language cognitive capabilities of the model, the parameters of the traditional visual language model are frozen during the training phase of the initial hand-object interaction recognition model.
[0115] It should also be noted that when training the model, the method of obtaining the first skeleton feature based on the skeleton points is also obtained by using the multi-head self-attention mechanism. More specifically, the skeleton points are obtained through a data set or a posture estimation algorithm to obtain three-dimensional coordinates, and the input skeleton sequence of each hand is represented as a three-dimensional array, the dimensions of which include the coordinate dimension, the time step, and the number of joints. When processing single-frame data, we simplify the input data into a two-dimensional form, that is, only consider the joint data of each frame. Then the sliding window method is used to segment the input data along the spatial dimension to form multiple spatial segments. The first skeleton feature is obtained through 3D convolution, normalization, and activation function.
[0116] In a preferred embodiment of the present invention, the input skeleton features are converted into text features through alignment, and then these converted text features are input into the initial hand-object interaction recognition model together with the visual features as additional prompt information to increase the amount of information input to the model.
[0117] Further, as a preferred implementation, the calculating the similarity between the first skeleton feature and the first text feature in the image-text pair to obtain a similarity matrix includes:
[0118] The temperature parameter is introduced to adjust the smoothness of the similarity distribution;
[0119] Calculating the cosine similarity between the first skeleton feature and the first text feature according to the adjusted similarity distribution;
[0120] A similarity matrix is obtained according to the cosine similarity between the first skeleton feature and the first text feature in the same image-text pair.
[0121] In the above scheme, when calculating the similarity between the first skeleton feature and the first text feature, the temperature parameter is introduced to adjust the smoothness of the similarity distribution to balance the calculation of the similarity, and then the difference between the similarity matrix and the true label distribution is measured by the cross-entropy loss function. After the losses from skeleton to text and text to skeleton are calculated separately, the total loss is finally obtained to ensure that the model learns the matching relationship in the bidirectional alignment.
[0122] A hand-object interaction recognition method provided by an embodiment of the present invention is adopted. An initial hand-object interaction recognition model of hand skeleton-vision-language tri-modality is formed by integrating the hand skeleton modality into the traditional visual language model, and the initial hand-object interaction recognition model is trained using a constructed gesture description dataset. The obtained hand-object interaction recognition model can combine the skeleton information, language and visual information in the image to be recognized to better understand the hand details, thereby improving the performance of the hand-object interaction task.
[0123] The present invention provides a hand-object interaction recognition system. Figure 2 The hand-object interaction recognition system includes a data set construction module 11, an initial model construction module 12, a model training module 13 and a model application module 14, wherein:
[0124] A data set construction module 11 is used to construct a gesture description data set; the gesture description data set includes skeleton description texts of a plurality of gesture postures;
[0125] An initial model building module 12 is used to fuse the hand skeleton modality, visual modality and language modality according to the traditional visual language model to establish an initial hand-object interaction recognition model;
[0126] A model training module 13, configured to train the initial hand-object interaction recognition model using the gesture description data set to obtain a hand-object interaction recognition model;
[0127] The model application module 14 is used to input the image to be recognized into the hand-object interaction recognition model to obtain the hand-object interaction action description in the image to be recognized.
[0128] As a preferred implementation, the data set construction module 11 is specifically used for:
[0129] Collecting a plurality of gesture images, and identifying palms, fingers, and finger joints in the plurality of gesture images;
[0130] Obtaining a first skeleton description text of the gesture image according to the direction of the palm;
[0131] Obtaining a second skeleton description text of the gesture image according to the positional relationship between the fingers;
[0132] Obtaining a third skeleton description text of the gesture image according to the flexion angle of the finger joint;
[0133] The first skeleton description text, the second skeleton description text, and the third skeleton description text are used to annotate the gesture image to obtain a gesture description data set.
[0134] As a preferred implementation, the initial model building module 12 includes:
[0135] A traditional model splitting unit, used to split the traditional visual language model into a visual processing submodule and a language processing submodule;
[0136] The first submodule processing unit is used to input the image data to be recognized into the visual processing submodule to obtain entity data and skeleton data;
[0137] A skeleton feature extraction unit, configured to extract skeleton features from the skeleton data using a multi-head self-attention mechanism;
[0138] A feature mapping unit, used for mapping the skeleton features to text dimensions through a multi-layer perceptron to obtain skeleton alignment features;
[0139] The modality fusion unit is used to input the entity data and the skeleton alignment features into the language processing submodule to achieve the fusion of the hand skeleton modality, visual modality and language modality to obtain an initial hand-object interaction recognition model.
[0140] Further, preferably, the skeleton feature extraction unit is specifically used for:
[0141] According to the skeleton data, obtaining joint space coordinates;
[0142] According to the joint space coordinates, the skeleton data is segmented using a preset sliding window to obtain a plurality of space segments;
[0143] A multi-head self-attention mechanism is adopted to calculate the spatial labels of the several spatial segments through 3D convolution, normalization and activation function to obtain skeleton features.
[0144] Furthermore, preferably, the multi-head self-attention mechanism is used to calculate the spatial labels of the plurality of spatial segments through 3D convolution, normalization and activation function to obtain skeleton features, including:
[0145] Establishing a multi-head self-attention model, wherein the multi-head self-attention model includes a plurality of self-attention heads with different linear transformation parameters;
[0146] Input the plurality of spatial segments into the multi-head self-attention model, and compute the query, key, and value of the spatial segment by 3D convolution in each attention head;
[0147] Calculate the similarity between the query and the key to obtain an attention score;
[0148] According to the attention score, weighting the value to obtain a weighted feature vector;
[0149] Taking the activation value calculated by the activation function as a target within a preset activation value range, the weighted feature vector is normalized to obtain a normalized feature;
[0150] Performing residual connection and convolution on the normalized features output by each of the attention heads to obtain updated features;
[0151] The updated features are fed back to each of the attention heads through a feedforward neural network to obtain spatial labels of the plurality of spatial segments as skeleton features.
[0152] Preferably, the feature mapping unit is specifically used for:
[0153] A linear transformation algorithm is performed on the skeleton features through a multi-layer perceptron to map the skeleton features to a high-level dimension to obtain a linear high-dimensional skeleton feature;
[0154] The linear high-dimensional skeleton features are nonlinearly transformed through the GELU activation function to obtain skeleton alignment features; the dimension of the skeleton alignment features is the same as the text dimension of the traditional visual language model.
[0155] As a preferred implementation, the model training module 13 includes:
[0156] An image-text pair generating unit, used to obtain a plurality of image-text pairs according to the gesture description data set; each of the image-text pairs includes a gesture image and a skeleton description text corresponding thereto;
[0157] A first skeleton feature acquisition unit, configured to extract skeleton points in the gesture image according to the gesture image to obtain a first skeleton feature;
[0158] A first text feature acquisition unit, configured to obtain a first text feature according to the skeleton description text;
[0159] A similarity calculation unit, used for calculating the similarity between the first skeleton feature and the first text feature in the image-text pair to obtain a similarity matrix;
[0160] A loss value calculation unit, used for calculating a bidirectional loss value between the first skeleton feature and the first text feature by using a cross entropy loss function;
[0161] A parameter optimization unit is used to adjust the parameters of the multi-head self-attention mechanism and the parameters of the multi-layer perceptron in the initial hand-object interaction recognition model according to the bidirectional loss value to obtain the hand-object interaction recognition model.
[0162] Further, preferably, the similarity calculation unit is specifically used to:
[0163] The temperature parameter is introduced to adjust the smoothness of the similarity distribution;
[0164] Calculating the cosine similarity between the first skeleton feature and the first text feature according to the adjusted similarity distribution;
[0165] A similarity matrix is obtained according to the cosine similarity between the first skeleton feature and the first text feature in the same image-text pair.
[0166] A hand-object interaction recognition system provided by an embodiment of the present invention forms an initial hand-object interaction recognition model of hand skeleton-vision-language tri-modality by integrating the hand skeleton modality into the traditional visual language model, and uses a constructed gesture description dataset to train the initial hand-object interaction recognition model. The obtained hand-object interaction recognition model can combine the skeleton information, language and visual information in the image to be recognized to better understand the hand details, thereby improving the performance of the hand-object interaction task.
[0167] An embodiment of the present invention also provides a hand-object interaction recognition device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a hand-object interaction recognition method as described above. The working principles and beneficial effects of the two correspond one to one, and thus will not be elaborated upon.
[0168] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0169] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for identifying hand-object interaction, characterized in that: include: Constructing a gesture description data set; the gesture description data set includes skeleton description texts of a plurality of gesture postures; According to the traditional visual language model, the hand skeleton modality, visual modality and language modality are integrated to establish the initial hand-object interaction recognition model; Using the gesture description data set to train the initial hand-object interaction recognition model to obtain a hand-object interaction recognition model; The image to be recognized is input into the hand-object interaction recognition model to obtain a description of the hand-object interaction action in the image to be recognized.
2. The method for identifying hand-object interaction according to claim 1, characterized in that: The step of constructing a gesture description dataset includes: Collecting a plurality of gesture images, and identifying palms, fingers, and finger joints in the plurality of gesture images; Obtaining a first skeleton description text of the gesture image according to the direction of the palm; Obtaining a second skeleton description text of the gesture image according to the positional relationship between the fingers; Obtaining a third skeleton description text of the gesture image according to the flexion angle of the finger joint; The first skeleton description text, the second skeleton description text, and the third skeleton description text are used to annotate the gesture image to obtain a gesture description data set.
3. The method for identifying hand-object interaction according to claim 1, characterized in that: According to the traditional visual language model, the hand skeleton modality, visual modality and language modality are integrated to establish an initial hand-object interaction recognition model, including: Split the traditional visual language model into a visual processing submodule and a language processing submodule; Input the image data to be recognized into the visual processing submodule to obtain entity data and skeleton data; A multi-head self-attention mechanism is used to extract skeleton features from the skeleton data; Mapping the skeleton features to the text dimension through a multi-layer perceptron to obtain skeleton alignment features; The entity data and the skeleton alignment features are input into the language processing submodule to achieve the fusion of the hand skeleton modality, visual modality and language modality to obtain an initial hand-object interaction recognition model.
4. The method for identifying hand-object interaction according to claim 3, characterized in that: The multi-head self-attention mechanism is used to extract skeleton features from the skeleton data, including: According to the skeleton data, obtaining joint space coordinates; According to the joint space coordinates, the skeleton data is segmented using a preset sliding window to obtain a plurality of space segments; A multi-head self-attention mechanism is adopted to calculate the spatial labels of the several spatial segments through 3D convolution, normalization and activation function to obtain skeleton features.
5. The method for identifying hand-object interaction according to claim 4, characterized in that: The multi-head self-attention mechanism is used to calculate the spatial labels of the plurality of spatial segments through 3D convolution, normalization and activation function to obtain skeleton features, including: Establishing a multi-head self-attention model, wherein the multi-head self-attention model includes a plurality of self-attention heads with different linear transformation parameters; Input the plurality of spatial segments into the multi-head self-attention model, and compute the query, key, and value of the spatial segment by 3D convolution in each attention head; Calculate the similarity between the query and the key to obtain an attention score; According to the attention score, weighting the value to obtain a weighted feature vector; Taking the activation value calculated by the activation function as a target within a preset activation value range, the weighted feature vector is normalized to obtain a normalized feature; Performing residual connection and convolution on the normalized features output by each of the attention heads to obtain updated features; The updated features are fed back to each of the attention heads through a feedforward neural network to obtain spatial labels of the plurality of spatial segments as skeleton features.
6. The method for identifying hand-object interaction according to claim 3, characterized in that: The skeleton features are mapped to the text dimension by a multi-layer perceptron to obtain skeleton alignment features, including: A linear transformation algorithm is performed on the skeleton features through a multi-layer perceptron to map the skeleton features to a high-level dimension to obtain a linear high-dimensional skeleton feature; The linear high-dimensional skeleton features are nonlinearly transformed through the GELU activation function to obtain skeleton alignment features; the dimension of the skeleton alignment features is the same as the text dimension of the traditional visual language model.
7. The method for identifying hand-object interaction according to claim 1, characterized in that: The step of using the gesture description data set to train the initial hand-object interaction recognition model to obtain the hand-object interaction recognition model includes: According to the gesture description data set, a plurality of image-text pairs are obtained; each of the image-text pairs comprises a gesture image and a skeleton description text corresponding thereto; Extracting skeleton points in the gesture image according to the gesture image to obtain a first skeleton feature; Describing the text according to the skeleton, obtaining a first text feature; Calculating the similarity between the first skeleton feature and the first text feature in the image-text pair to obtain a similarity matrix; Calculating a bidirectional loss value between the first skeleton feature and the first text feature by using a cross entropy loss function; The parameters of the multi-head self-attention mechanism and the multi-layer perceptron in the initial hand-object interaction recognition model are adjusted according to the bidirectional loss value to obtain a hand-object interaction recognition model.
8. The method for identifying hand-object interaction according to claim 7, characterized in that: The calculating the similarity between the first skeleton feature and the first text feature in the image-text pair to obtain a similarity matrix includes: The temperature parameter is introduced to adjust the smoothness of the similarity distribution; Calculating the cosine similarity between the first skeleton feature and the first text feature according to the adjusted similarity distribution; A similarity matrix is obtained according to the cosine similarity between the first skeleton feature and the first text feature in the same image-text pair.
9. A hand-object interaction recognition system, characterized in that: include: A data set construction module is used to construct a gesture description data set; the gesture description data set includes skeleton description texts of a plurality of gesture postures; The initial model building module is used to fuse the hand skeleton modality, visual modality and language modality according to the traditional visual language model to establish an initial hand-object interaction recognition model; A model training module, used to train the initial hand-object interaction recognition model using the gesture description data set to obtain a hand-object interaction recognition model; The model application module is used to input the image to be recognized into the hand-object interaction recognition model to obtain the hand-object interaction action description in the image to be recognized.
10. A hand-object interaction recognition device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the hand-object interaction recognition method as described in any one of claims 1 to 8 when executing the computer program.