Gesture recognition method and device

Abstract

The invention discloses a gesture recognition method and device, and relates to the technical field of computer vision and human-computer interaction. A specific embodiment of the method comprises the following steps: acquiring multi-modal visual data of a to-be-recognized gesture, and performing feature extraction on the multi-modal visual data to obtain visual features; constructing cue words of the gestures based on the cue word template, and performing feature extraction on the cue words of the gestures to obtain text features; and calculating the similarity between the multi-modal visual data and the cue word of each gesture according to the visual features and the text features, and determining a gesture recognition result according to the similarity. According to the embodiment, the accuracy, robustness and expandability of gesture recognition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and human-computer interaction, and in particular to a method and device for gesture recognition. Background Art

[0002] Gesture recognition is an important research direction in the field of computer vision and human-computer interaction. By recognizing and understanding human gestures, computer systems can achieve new ways of interaction. For example, in the field of AR (Augmented Reality) / VR (Virtual Reality), gesture recognition is an important technology for achieving an immersive experience. Users can directly manipulate virtual objects through gestures to navigate, select and operate, which greatly improves the intuitiveness and immersion of the interaction. Existing gesture recognition technology usually adopts a computer vision-based solution, inputs single-modal information such as RGB images or bone node coordinates, and realizes the recognition of specific gestures through architectures such as rule systems and deep learning networks.

[0003] In the process of implementing the present invention, the inventors found that the input data of the existing gesture recognition technology solutions is unimodal data, which leads to certain limitations in the accuracy and robustness of the gesture recognition results. At the same time, the existing technology solutions are mostly simple conditional judgments or use simple deep learning network structures without knowledge reserves and reasoning capabilities for gesture recognition, so the robustness and scalability of gesture recognition are poor. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a method and device for gesture recognition, which can combine multimodal visual data and prompt words to perform gesture recognition. Multimodal visual data ensures the richness of data content and avoids the problem of inaccurate recognition caused by interference such as ambient light, thereby improving the accuracy and robustness of gesture recognition. At the same time, for gestures to be recognized that have not been trained, it is only necessary to collect multimodal visual data and generate prompt words for new gestures to be recognized based on prompt word templates. The gesture recognition result can be obtained based on the similarity between the multimodal visual data and the prompt words, realizing the zero-sample prediction function and improving the robustness and scalability of gesture recognition.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for gesture recognition is provided, comprising: Acquire multimodal visual data of a gesture to be recognized, and perform feature extraction on the multimodal visual data to obtain visual features; Constructing prompt words for each gesture based on the prompt word template, and performing feature extraction on the prompt words for each gesture to obtain text features; The similarity between the multimodal visual data and the prompt words of each gesture is calculated according to the visual features and the text features, and the gesture recognition result is determined according to the similarity.

[0006] Optionally, the multimodal visual data includes ordinary image data, depth image data and position data of palm joints; feature extraction is performed on the multimodal visual data to obtain visual features, including: for ordinary image data and depth image data, the entire image is divided into a specified number of small pieces, and a position code of each small piece is generated according to the position of each small piece in the entire image; for each small piece, a linear transformation is performed on the small piece and the resultant is added to the position code of the small piece to obtain first preprocessed data; for the position data of the palm joints, the three-dimensional coordinate information in the position data is encoded to obtain second preprocessed data; and the first preprocessed data and the second preprocessed data are spliced ​​and input into a visual transformer network for feature extraction to obtain visual features.

[0007] Optionally, the position data of the palm joint points are encoded according to the three-dimensional coordinate information in the position data to obtain second preprocessed data, including: the position data of the palm joint points are encoded by sine function and cosine function according to the three-dimensional coordinate information in the position data to convert the data format to obtain second preprocessed data.

[0008] Optionally, the visual transformer network is trained based on multiple frames of ordinary image data with continuous temporal information, depth image data corresponding to each frame of ordinary image data, and position data of palm joints.

[0009] Optionally, the prompt word template includes a fixed gesture name prompt word and a plurality of learnable prompt word vectors, and the prompt word template is obtained by pre-training sample prompt words of each collected gesture.

[0010] Optionally, extracting features from the prompt words of each gesture to obtain text features includes: inputting the prompt words of each gesture into a pre-trained language model to extract features to obtain text features.

[0011] Optionally, the similarity between the multimodal visual data and the prompt words of the various gestures is calculated according to the visual features and the text features, and the gesture recognition result is determined according to the similarity, including: calculating the similarity between the multimodal visual data and the prompt words of the various gestures according to the visual features and the text features, and taking the prompt word of the gesture with the highest similarity to the multimodal visual data as the prompt word of the gesture to be recognized; and obtaining the gesture recognition result according to the gesture name prompt word in the prompt word of the gesture to be recognized.

[0012] According to another aspect of an embodiment of the present invention, there is provided a gesture recognition device, comprising: A visual feature extraction module, used to obtain multimodal visual data of the gesture to be recognized, and perform feature extraction on the multimodal visual data to obtain visual features; A text feature extraction module, used to construct a prompt word for each gesture based on a prompt word template, and perform feature extraction on the prompt word for each gesture to obtain a text feature; The recognition result determination module is used to calculate the similarity between the multimodal visual data and the prompt words of each gesture according to the visual features and the text features, and determine the gesture recognition result according to the similarity.

[0013] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the gesture recognition method provided by an embodiment of the present invention.

[0014] According to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for gesture recognition provided by the embodiments of the present invention is implemented.

[0015] According to another aspect of the embodiments of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method for gesture recognition provided by the embodiments of the present invention is implemented.

[0016] One embodiment of the above invention has the following advantages or beneficial effects: by acquiring multimodal visual data of gestures to be recognized, and extracting features from the multimodal visual data to obtain visual features; constructing prompt words for each gesture based on prompt word templates, and extracting features from the prompt words for each gesture to obtain text features; calculating the similarity between the multimodal visual data and the prompt words for each gesture based on the visual features and text features, and determining the gesture recognition result based on the similarity, the technical solution can combine multimodal visual data and prompt words for gesture recognition, and the multimodal visual data ensures the richness of the data content, avoiding the problem of inaccurate recognition caused by interference such as ambient light, thereby improving the accuracy and robustness of gesture recognition. At the same time, for gestures to be recognized that have not been trained, it is only necessary to collect multimodal visual data and generate prompt words for new gestures to be recognized based on the prompt word template, and the gesture recognition result can be obtained based on the similarity between the multimodal visual data and the prompt words, realizing the zero-sample prediction function, and improving the robustness and scalability of gesture recognition.

[0017] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Figure 1 is a schematic diagram of main steps of a method for gesture recognition according to an embodiment of the present invention; Figure 2 is a structural diagram of a gesture recognition model according to an embodiment of the present invention; Figure 3 is a schematic diagram of main modules of a gesture recognition device according to an embodiment of the present invention; Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied; Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0021] In the introduction to the embodiments of the present invention, the professional terms involved and their definitions are as follows.

[0022] 1. Transformer is a neural network architecture originally proposed by Vaswani et al. in their 2017 paper “Attention is All You Need”. It is mainly used to process sequence data (such as natural language) and performs well in tasks such as machine translation and text generation. The core component of Transformer is the “attention mechanism”, especially the “self-attention mechanism”, which allows the model to focus on other elements in the sequence when processing each input element, thereby capturing long-distance dependencies.

[0023] 2. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model proposed by Google. BERT uses a bidirectional Transformer encoder to generate contextual representations of text. It has achieved remarkable results in various natural language processing tasks by performing unsupervised pre-training on a large-scale corpus and then fine-tuning on specific tasks. The innovation of BERT lies in its bidirectionality, that is, it considers both the left and right contexts when generating the representation of each word.

[0024] 3. CLIP (Contrastive Language-Image Pre-Training) is a model proposed by OpenAI that aims to jointly train the embedding representation of images and text through contrastive learning. CLIP uses a large number of image-text pairs to train the model so that it can understand and match the semantic relationship between images and text. After training, CLIP can perform various visual tasks such as image classification and image search in a "zero-shot" setting.

[0025] 4. Prompt, in natural language processing (NLP), prompt refers to an input format or template used to guide the pre-trained language model to generate specific output. By designing different prompts, the knowledge of the pre-trained model on various tasks can be effectively utilized. For example, in zero-shot or few-shot learning, designing a suitable prompt can help the model understand the task requirements and generate corresponding answers.

[0026] 5. Embedding is a method of mapping discrete objects (such as words, sentences, images, etc.) to continuous vector space. In natural language processing, word embedding is to convert words into real vectors of fixed dimensions so that words with similar semantics are closer in the vector space. Common word embedding methods include Word2Vec, GloVe, and BERT.

[0027] 6. Few-shot Learning is a machine learning method that aims to use a very small number of training samples (usually 1 to 5 per class) to train the model so that it can perform well on new tasks. Few-shot Learning achieves rapid adaptation to new tasks by leveraging the knowledge of pre-trained models and a small amount of task-specific data. Common methods include meta-learning and fine-tuning using pre-trained models.

[0028] 7. Zero-shot Learning is a machine learning method that aims to enable the model to handle tasks or categories that have never been seen before. Zero-shot Learning achieves recognition of new categories by leveraging the features of known categories and the descriptions of new categories. For example, the CLIP model performs well in unseen image classification tasks by jointly training image and text embeddings.

[0029] In the existing gesture recognition technology solutions, the input data is single-mode time-series continuous or non-time-series data, such as common inputs including: RGB (RGB color mode is a color standard in the industry. It is obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other to obtain various colors. RGB represents the colors of the three channels of red, green, and blue) image input, palm bone node coordinate input, etc. After obtaining the input data, the next technical solution is usually conditional judgment based on the rule system, LSTM (Long Short-Term Memory, long short-term memory network), Transformer converter and other non-multimodal architectures. Since the existing technology does not use multimodal data and multimodal architecture, the accuracy and robustness of gesture recognition results have certain limitations. For example, when only RGB image data is used as input, when the image is affected by light, gesture recognition errors are likely to occur. Correspondingly, if only palm bone node input is used, when the node data is lost or offset, the accuracy of gesture recognition results will also be reduced. At the same time, the solutions in the existing technology are mostly simple conditional judgments or use simple deep learning network structures without knowledge reserves and reasoning capabilities, so the robustness and scalability of gesture recognition are poor.

[0030] In order to solve the technical problems existing in the prior art, the present invention proposes a gesture recognition method based on multimodal information fusion and prompt word learning. First, the input data of this method is multimodal fusion data, such as inputting multiple frames of continuous RGB images, corresponding multiple frames of continuous depth images and multiple frames of continuous palm joint coordinate point information; then, a gesture recognition architecture based on the fusion of the vision transformer Vision Transformer and the pre-trained model of natural language processing is constructed. By performing modal fusion on multimodal input information such as images, coordinate points, and texts, and combining the knowledge reserve, reasoning, Few shot (few sample tasks), Zero shot (zero sample tasks) and other capabilities of the pre-trained large model, a gesture recognition system with high accuracy, strong robustness and good scalability is finally realized.

[0031] Figure 1FIG. 1 is a schematic diagram of the main steps of the method for gesture recognition according to an embodiment of the present invention. Figure 1 As shown, the gesture recognition method of the embodiment of the present invention mainly includes the following steps S101 to S103.

[0032] Step S101: Acquire multimodal visual data of the gesture to be recognized, and extract features from the multimodal visual data to obtain visual features. In the embodiment of the present invention, a multimodal fusion large model technical solution is used to implement the task of gesture recognition, thereby improving the accuracy and robustness of gesture recognition results.

[0033] According to one embodiment of the present invention, multimodal visual data may include, for example, ordinary image data, depth image data, and position data of palm joints. Among them, ordinary image data is, for example, RGB image data collected by a camera or a camera. A depth image, also known as a range image, refers to an image that uses the distance (depth) from the image collector to each point in the scene as a pixel value, which directly reflects the geometry of the visible surface of the scene. Among them, multimodal visual data can be collected separately by hardware and software devices such as cameras, depth cameras, head-mounted "spatial computing" display devices, and machine learning pipelines for computer vision applications.

[0034] Feature extraction is performed on the multimodal visual data to obtain visual features, which may specifically include: for ordinary image data and depth image data, the entire image is divided into a specified number of small pieces, and a position code of each small piece is generated according to the position of each small piece in the entire image; for each small piece, a linear transformation is performed on the small piece, and the resultant is added to the position code of the small piece to obtain first preprocessed data; for the position data of the palm joints, the three-dimensional coordinate information in the position data is encoded to obtain second preprocessed data; and the first preprocessed data and the second preprocessed data are spliced ​​and input into a visual transformer network for feature extraction to obtain visual features.

[0035] According to one embodiment of the present invention, the position data of the palm joint points are encoded according to the three-dimensional coordinate information in the position data to obtain the second preprocessed data, which can specifically include: the position data of the palm joint points are encoded according to the three-dimensional coordinate information in the position data by using sine function and cosine function to convert the data format to obtain the second preprocessed data.

[0036] In order to achieve the fusion of the three kinds of visual data, the present invention needs to process them separately. For RGB image data and depth image data, the whole image is divided into KxK small patches respectively. The typical value of K is 16, that is, a whole image is cut into 256 small patches. At the same time, in order to retain the position information of the gesture in the image, the learnable position encoding Position Embedding is used to describe the position of these small patches in the whole image. After that, each patch is linearly transformed and added to the position encoding Position Embedding. In this way, the processing of RGB image data and depth map data is completed to obtain the first preprocessed data. For the position data of each joint point of the palm, their X, Y, and Z three-dimensional coordinate information is encoded by two periodic functions, sine function sin and cosine function cos, to realize the conversion of data format, where the conversion formula can be the formula commonly used in algorithms such as pre-trained language model Bert, as shown below: ; Among them, PE is the abbreviation of Position Embedding, that is, position encoding, p represents the position of the small piece, d model is the dimension of the model encoding, that is, the size of the model output data, i represents the size of the small piece in d model The i-th dimension in the model encoding.

[0037] After completing the position data processing of the palm joints and obtaining the second preprocessed data, the processed data of the three modes are spliced ​​together and input into the pre-trained vision transformer network Vision Transformer network for feature extraction to obtain visual features.

[0038] According to an embodiment of the present invention, the visual transformer network is trained based on multiple frames of ordinary image data with continuous temporal information, depth image data corresponding to each frame of ordinary image data, and position data of palm joints.

[0039] Specifically, continuous frame images, depth images, palm joint coordinate points and other information of different gestures can be collected through hardware and software devices such as cameras, depth cameras, head-mounted "spatial computing" display devices, and machine learning pipelines for computer vision applications. Since the present invention involves model training, for several gestures to be recognized, each gesture needs to collect M segments of continuous data in different environments. In one embodiment of the present invention, it is assumed that 6 gestures are to be recognized, namely, correct (ok), like (good), victory (victory), love (love), pray (pray) and wave (wave). Then for each gesture, 1000 segments are collected in different lighting environments (such as morning, noon, evening, indoors, and outdoors), each segment is 10 seconds long, about 300 frames. At the same time, in order to test the reasoning performance of the model, the same scheme is adopted, and 100 segments are collected for each gesture as a test set.

[0040] After completing the data collection work, the model training will begin. The gesture recognition model of the present invention is a technical solution for gesture recognition after a set of multimodal data fusion, which mainly includes two branches, namely, the Vision Transformer branch for processing ordinary images, depth images, and the position data of palm joints, and the pre-trained language branch for processing prompt word text. The network parameters of the Vision Transformer branch and the pre-trained language Bert branch are pre-trained in advance through multimodal comparative learning methods such as CLIP. After pre-training on general large-scale data (on the order of 1 billion), the model has the capabilities of knowledge reserve, reasoning, Few shot, Zero shot, etc., which plays an important role in the accuracy, robustness and scalability of subsequent gesture recognition.

[0041] For the Vision Transformer branch, unlike the gesture recognition method, the present invention uses three modal data at the same time, namely RGB image data, depth map data and palm joint position data. And all three types of data contain continuous time series information. For example, for RGB image data, there are 300 frames of images for each gesture. Correspondingly, there are also 300 frames of depth images corresponding to the RGB image, and 300 sets of corresponding palm joint position data will also be collected. Afterwards, these three types of data can be processed and fused separately, and the processing results can be spliced ​​together and input into the pre-trained Vision Transformer network to adjust the parameters of the Vision Transformer network to obtain the visual transformer network.

[0042] In an embodiment of the present invention, by collecting multiple frames of image data with continuous time series information to train the visual transformer network, the characteristics and characteristic changes of the gesture can be better obtained, so as to more accurately perform gesture recognition in combination with the characteristic changes of the gesture.

[0043] Step S102: construct prompt words for each gesture based on prompt word templates, and extract features of the prompt words for each gesture to obtain text features. In an embodiment of the present invention, prompt word templates for various gestures may be pre-set so as to construct prompt words for each gesture based on the prompt word templates.

[0044] According to an embodiment of the present invention, the prompt word template includes a fixed gesture name prompt word and a plurality of learnable prompt word vectors, and the prompt word template is obtained by pre-training the sample prompt words of each collected gesture.

[0045] According to an embodiment of the present invention, extracting features from the prompt words of each gesture to obtain text features may specifically include: inputting the prompt words of each gesture into a pre-trained language model to extract features to obtain text features.

[0046] In the embodiment of the present invention, for each gesture to be recognized, a prompt word template can be constructed respectively. In order to improve the accuracy and robustness of gesture recognition, the present invention adopts a learnable prompt word scheme, namely Prompt Learning. The constructed prompt word template is in the following form: {ok} [V1] [V2]…[Vn] {good} [V1] [V2]…[Vn] {victory}[V1] [V2]…[Vn] {love} [V1] [V2]…[Vn] {pray}[V1] [V2]…[Vn] {wave}[V1] [V2]…[Vn] Among them, the words in {} are fixed gesture name prompt words, V1...Vn are learnable prompt word vectors, and the number of n can be 16 or 24, etc. After constructing the prompt word template, you can also use the pre-processing method of the pre-trained language model Bert, and then input it into the pre-trained language model for feature extraction to obtain text features. Among them, the Bert network has also been pre-trained. For the construction of the prompt word template, you can collect a large amount of description text corresponding to the gesture data, extract features from the text to obtain the prompt word vector, and train and learn based on the gesture name and prompt word vector of each gesture, you can update the initial value of the n prompt word vectors in the prompt word template to obtain the final prompt word template.

[0047] When the prompt words of each gesture are subjected to feature extraction to obtain text features, the prompt words of each gesture may be input into the trained pre-trained language model for feature extraction.

[0048] Step S103: calculating the similarity between the multimodal visual data and the prompt words of each gesture according to the visual features and the text features, and determining the gesture recognition result according to the similarity.

[0049] According to an embodiment of the present invention, the similarity between the multimodal visual data and the prompt words of each gesture is calculated according to the visual features and the text features, and the gesture recognition result is determined according to the similarity. Specifically, it can include: calculating the similarity between the multimodal visual data and the prompt words of each gesture according to the visual features and the text features, and taking the prompt words of the gesture with the highest similarity to the multimodal visual data as the prompt words of the gesture to be recognized; obtaining the gesture recognition result according to the gesture name prompt words in the prompt words of the gesture to be recognized. Wherein, when calculating the similarity between the visual features and the text features, cosine similarity, Pearson correlation coefficient, Euclidean distance, Manhattan distance, and other algorithms can be selected, and the present invention does not limit this. For each gesture prompt word, it corresponds to a text feature. By calculating the similarity between the visual features and the text features, the similarity between the multimodal visual data of the gesture to be recognized and the prompt words of each gesture can be obtained, and the gesture name corresponding to the gesture prompt word with the highest similarity is taken as the gesture name of the gesture to be recognized, and the gesture recognition can be completed.

[0050] In one embodiment of the present invention, after the data of the Vision Transformer branch and the Bert branch are subjected to feature extraction by their respective models, visual feature visual embedding and text feature text embedding are obtained, and then the similarity calculation formula is used to calculate the correlation degree (i.e., similarity) between the visual feature and the text feature. The similarity formula may be cosine similarity, etc. After calculating the similarity between the visual feature and the text feature, the gesture name corresponding to the gesture prompt word with the highest similarity may be used as the gesture name of the gesture to be recognized, and the gesture recognition result may be obtained. In an embodiment of the present invention, when performing gesture recognition, batch recognition may be performed, or only one gesture may be recognized at a time, and the present invention is not limited to this.

[0051] In an embodiment of the present invention, when performing gesture recognition, a gesture recognition model may be trained, and then multimodal data corresponding to one or a batch of gestures to be recognized is input into the gesture recognition model to obtain a recognition result. And when training the gesture recognition model, a batch processing method is adopted, that is, multiple gestures are trained at one time. The training data of each gesture includes an RGB image sequence, a depth image sequence, a position coordinate sequence of each joint point of the palm, and a corresponding text prompt word Prompt. According to the hardware performance of GPU (graphics processing unit, graphics processor), the batch size of each processing can be adjusted, such as 4 or 8, etc.

[0052] Figure 2 FIG. 1 is a schematic diagram of the structure of a gesture recognition model according to an embodiment of the present invention. Figure 2 As shown, in one embodiment of the present invention, the input of the gesture recognition model mainly includes two parts, one part is the multimodal visual data with continuous time series information of each gesture, and the multimodal data includes RGB image data, depth image data and the position data of the palm joint points; the other part is the gesture prompt word corresponding to each gesture, wherein the prompt word includes information such as the gesture name prompt word and the prompt word vector, and represents each text word in the prompt word and its position information. For the multimodal data, after multimodal data fusion processing is performed through the visual transformer encoder layer, feature extraction is performed to obtain visual features; for each gesture prompt word, feature extraction is performed through the bidirectional encoder layer based on the transformer to obtain text features. Then, the visual features and text features are compared for similarity through the similarity comparison layer, and the obtained similarity is normalized through the normalization processing layer Softmax layer, and the data is normalized to between 0 and 1 to facilitate model training and result comparison. In this way, the gesture recognition result can be obtained according to the similarity after normalization processing of the output.

[0053] Figure 3 FIG. 1 is a schematic diagram of the main modules of a gesture recognition device according to an embodiment of the present invention. Figure 3 As shown, the gesture recognition device 300 according to the embodiment of the present invention mainly includes a visual feature extraction module 301 , a text feature extraction module 302 and a recognition result determination module 303 .

[0054] A visual feature extraction module 301 is used to obtain multimodal visual data of a gesture to be recognized, and extract features from the multimodal visual data to obtain visual features; A text feature extraction module 302, used to construct a prompt word for each gesture based on a prompt word template, and extract features from the prompt word for each gesture to obtain text features; The recognition result determination module 303 is used to calculate the similarity between the multimodal visual data and the prompt words of each gesture according to the visual features and the text features, and determine the gesture recognition result according to the similarity.

[0055] According to one embodiment of the present invention, the multimodal visual data includes ordinary image data, depth image data and position data of palm joints; the visual feature extraction module 301 can also be used to: for ordinary image data and depth image data, divide the entire image into a specified number of small pieces, and generate a position code for each small piece according to the position of each small piece in the entire image; for each small piece, perform a linear transformation on the small piece and add the resultant to the position code of the small piece to obtain first preprocessed data; for the position data of the palm joints, encode the three-dimensional coordinate information in the position data to obtain second preprocessed data; splice the first preprocessed data and the second preprocessed data and input them into the visual transformer network for feature extraction to obtain visual features.

[0056] According to another embodiment of the present invention, when the visual feature extraction module 301 encodes the position data of the palm joint points according to the three-dimensional coordinate information in the position data to obtain the second preprocessed data, it can be specifically used to: encode the position data of the palm joint points according to the three-dimensional coordinate information in the position data by using sine function and cosine function to convert the data format to obtain the second preprocessed data.

[0057] According to another embodiment of the present invention, the visual transformer network is trained based on multiple frames of ordinary image data with continuous temporal information, depth image data corresponding to each frame of ordinary image data, and position data of palm joints.

[0058] According to yet another embodiment of the present invention, the prompt word template includes a fixed gesture name prompt word and a plurality of learnable prompt word vectors, and the prompt word template is obtained by pre-training sample prompt words of each collected gesture.

[0059] According to yet another embodiment of the present invention, the text feature extraction module 302 may also be used to: input the prompt words of each gesture into a pre-trained language model to perform feature extraction to obtain text features.

[0060] According to another embodiment of the present invention, the recognition result determination module 303 can also be used to: calculate the similarity between the multimodal visual data and the prompt words of each gesture based on the visual features and the text features, and use the prompt word of the gesture with the highest similarity to the multimodal visual data as the prompt word of the gesture to be recognized; and obtain the gesture recognition result based on the gesture name prompt word in the prompt word of the gesture to be recognized.

[0061] According to the technical solution of the embodiment of the present invention, by acquiring multimodal visual data of the gesture to be recognized, and extracting features from the multimodal visual data to obtain visual features; constructing prompt words for each gesture based on prompt word templates, and extracting features from the prompt words for each gesture to obtain text features; calculating the similarity between the multimodal visual data and the prompt words for each gesture based on the visual features and text features, and determining the gesture recognition result based on the similarity, the technical solution can combine multimodal visual data and prompt words for gesture recognition, and the multimodal visual data ensures the richness of the data content, avoiding the problem of inaccurate recognition caused by interference such as ambient light, thereby improving the accuracy and robustness of gesture recognition. At the same time, for gestures to be recognized that have not been trained, it is only necessary to collect multimodal visual data and generate prompt words for new gestures to be recognized based on the prompt word template, and the gesture recognition result can be obtained based on the similarity between the multimodal visual data and the prompt words, realizing the zero-sample prediction function, and improving the robustness and scalability of gesture recognition.

[0062] Figure 4 An exemplary system architecture 400 is shown to which the method or apparatus for gesture recognition according to an embodiment of the present invention may be applied.

[0063] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0064] The user can use the terminal devices 401, 402, 403 to interact with the server 405 through the network 404 to receive or send messages, etc. Various client applications can be installed on the terminal devices 401, 402, 403, such as shopping applications, gesture recognition applications, image acquisition applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0065] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0066] Server 405 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using terminal devices 401, 402, and 403 (only as an example). The background management server may obtain multimodal visual data of the gesture to be recognized from the received gesture recognition request and other data, and perform feature extraction on the multimodal visual data to obtain visual features; construct prompt words for each gesture based on the prompt word template, and perform feature extraction on the prompt words for each gesture to obtain text features; calculate the similarity between the multimodal visual data and the prompt words for each gesture based on the visual features and the text features, and determine the gesture recognition result based on the similarity, and feed back the processing result (such as the gesture recognition result - only as an example) to the terminal device.

[0067] It should be noted that the gesture recognition method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the gesture recognition device is generally disposed in the server 405 .

[0068] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0069] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 5 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0070] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0071] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0072] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0073] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0074] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0075] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be arranged in a processor. For example, they may be described as: a processor includes a visual feature extraction module, a text feature extraction module, and a recognition result determination module. In some cases, the names of these units or modules do not constitute limitations on the units or modules themselves. For example, the recognition result determination module may also be described as "a module for calculating the similarity between the multimodal visual data and the prompt words of each gesture based on the visual features and the text features, and determining the gesture recognition result based on the similarity".

[0076] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining multimodal visual data of the gesture to be recognized, and extracting features from the multimodal visual data to obtain visual features; constructing prompt words for each gesture based on a prompt word template, and extracting features from the prompt words for each gesture to obtain text features; calculating the similarity between the multimodal visual data and the prompt words for each gesture based on the visual features and the text features, and determining the gesture recognition result based on the similarity.

[0077] According to the technical solution of the embodiment of the present invention, by acquiring multimodal visual data of the gesture to be recognized, and extracting features from the multimodal visual data to obtain visual features; constructing prompt words for each gesture based on prompt word templates, and extracting features from the prompt words for each gesture to obtain text features; calculating the similarity between the multimodal visual data and the prompt words for each gesture based on the visual features and text features, and determining the gesture recognition result based on the similarity, the technical solution can combine multimodal visual data and prompt words for gesture recognition, and the multimodal visual data ensures the richness of the data content, avoiding the problem of inaccurate recognition caused by interference such as ambient light, thereby improving the accuracy and robustness of gesture recognition. At the same time, for gestures to be recognized that have not been trained, it is only necessary to collect multimodal visual data and generate prompt words for new gestures to be recognized based on the prompt word template, and the gesture recognition result can be obtained based on the similarity between the multimodal visual data and the prompt words, realizing the zero-sample prediction function, and improving the robustness and scalability of gesture recognition.

[0078] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for gesture recognition, characterized in that: include: Acquire multimodal visual data of a gesture to be recognized, and perform feature extraction on the multimodal visual data to obtain visual features; Constructing prompt words for each gesture based on the prompt word template, and performing feature extraction on the prompt words for each gesture to obtain text features; The similarity between the multimodal visual data and the prompt words of each gesture is calculated according to the visual features and the text features, and the gesture recognition result is determined according to the similarity.

2. The method according to claim 1, characterized in that The multimodal visual data includes ordinary image data, depth image data and position data of palm joints; Extracting features from the multimodal visual data to obtain visual features includes: For normal image data and depth image data, the entire image is divided into a specified number of small pieces, and a position code of each small piece is generated according to the position of each small piece in the entire image; For each small piece, perform a linear transformation on the small piece and add the result to the position code of the small piece to obtain first preprocessed data; Encoding the position data of the palm joint points according to the three-dimensional coordinate information in the position data to obtain second preprocessed data; The first preprocessed data and the second preprocessed data are spliced ​​and input into a visual transformer network for feature extraction to obtain visual features.

3. The method according to claim 2, characterized in that The position data of the palm joints are encoded according to the three-dimensional coordinate information in the position data to obtain second preprocessed data, including: The position data of the palm joints are encoded by sine function and cosine function according to the three-dimensional coordinate information in the position data to convert the data format and obtain the second pre-processed data.

4. The method according to claim 2, characterized in that: The visual transformer network is trained based on multiple frames of ordinary image data with continuous time sequence information, depth image data corresponding to each frame of ordinary image data, and position data of palm joints.

5. The method according to claim 1, characterized in that The prompt word template includes a fixed gesture name prompt word and a plurality of learnable prompt word vectors, and the prompt word template is obtained by pre-training the sample prompt words of each collected gesture.

6. The method according to claim 1 or 5, characterized in that: Feature extraction is performed on the prompt words of each gesture to obtain text features, including: The prompt words of each gesture are input into a pre-trained language model to perform feature extraction to obtain text features.

7. The method according to claim 5, characterized in that Calculating the similarity between the multimodal visual data and the prompt words of each gesture according to the visual features and the text features, and determining the gesture recognition result according to the similarity, including: Calculating the similarity between the multimodal visual data and the prompt words of each gesture according to the visual features and the text features, and using the prompt word of the gesture with the highest similarity to the multimodal visual data as the prompt word of the gesture to be recognized; A gesture recognition result is obtained according to the gesture name prompt word in the prompt word of the gesture to be recognized.

8. A gesture recognition device, characterized in that: include: A visual feature extraction module, used to obtain multimodal visual data of the gesture to be recognized, and perform feature extraction on the multimodal visual data to obtain visual features; A text feature extraction module, used to construct a prompt word for each gesture based on a prompt word template, and perform feature extraction on the prompt word for each gesture to obtain a text feature; The recognition result determination module is used to calculate the similarity between the multimodal visual data and the prompt words of each gesture according to the visual features and the text features, and determine the gesture recognition result according to the similarity.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images