Gesture recognition method and apparatus

Abstract

The application discloses a gesture recognition method and device. The method comprises the following steps: acquiring a first image vector and a weight of the first image vector; the first image vector is an image vector of a key frame in an image sequence generated according to a video image of a gesture to be recognized, and the weight is a weight of the key frame in the image sequence; generating a first gesture feature vector according to the first image vector and the weight of the first image vector; determining a gesture recognition result of the gesture to be recognized according to the first gesture feature vector and a first gesture feature vector set; the first gesture feature vector set is a set composed of gesture feature vectors whose gesture recognition results are known. By converting the image vector of the key frame of the image sequence of the gesture to be recognized into a gesture feature vector with a fixed length, the gesture recognition process is converted into a retrieval process of the gesture feature vector, and the problems of introducing noise or losing feature information and increasing device overhead can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminal device, and particularly relates to a gesture recognition method and device. BACKGROUND

[0002] With the development of human-computer interaction technology and computer vision technology, the interaction between human and computer gradually becomes an important part of people's daily life. For example, gesture recognition based on computer vision technology is a very friendly human-computer interaction mode.

[0003] Gesture recognition can include dynamic gesture recognition, that is, the recognition of a dynamic gesture made by a user by a computer. A dynamic gesture is usually composed of multiple hand actions. At present, a dynamic gesture can be recognized by using a deep learning model, or can be recognized by a template matching manner.

[0004] When a dynamic gesture is recognized by using a deep learning model, feature extraction needs to be performed on the dynamic gesture. At present, a fixed feature collection window is usually used to perform feature extraction on dynamic gestures made by different users to represent the same meaning. However, since the action frequency, amplitude and hand shape of dynamic gestures made by different users usually have great differences, if a fixed feature collection window is used to perform feature extraction on dynamic gestures made by different users, noise data will be introduced for dynamic gestures made by users with relatively fast action frequency, and part of feature information will be lost for dynamic gestures made by users with relatively slow action frequency, which affects the accuracy of gesture recognition.

[0005] When a dynamic gesture is recognized by a template matching manner, since the action frequency, amplitude and hand shape of dynamic gestures made by different users to represent the same meaning have great differences, and the dynamic gestures made by the same user to represent the same meaning have random variability, a large number of templates need to be provided, which not only affects the efficiency of gesture recognition, but also increases the device overhead, and has poor applicability. SUMMARY

[0006] The present application provides a gesture recognition method and device to solve the problems of low gesture recognition accuracy and efficiency and large device overhead in current dynamic gesture recognition.

[0007] In a first aspect, the present application provides a gesture recognition method, comprising: obtaining a first image vector and a weight of the first image vector; the first image vector being an image vector of a key frame in a first image sequence; the weight of the first image vector being a weight of the key frame in the first image sequence; the first image sequence being an image sequence generated according to a video image of a gesture to be recognized; the key frame being an image carrying hand information in the first image sequence; generating a first gesture feature vector according to the first image vector and the weight of the first image vector; determining a gesture recognition result of the gesture to be recognized according to the first gesture feature vector and a first gesture feature vector set; the first gesture feature vector set being a set composed of gesture feature vectors whose gesture recognition results are known.

[0008] In the present implementation, first, a first image vector of a key frame in a first image sequence of a gesture to be recognized is extracted, and a weight of the first image vector used to represent the importance of the key frame in the first image sequence is obtained, then a first gesture feature vector is generated according to the first image vector and the weight of the first image vector, finally, a gesture recognition result of the gesture to be recognized is determined according to the first gesture feature vector and a first gesture feature vector set containing gesture feature vectors whose gesture recognition results are known. By using the scheme of the present implementation, in the case that gestures made by different users for the same meaning are different, and in the case that gestures made by the same user for the same meaning are randomly changed, the information of important gesture features of the corresponding gesture to be recognized can be obtained through the extraction of the key frame, and the problem of noise introduction or gesture feature information loss caused by the fixation of the feature collection window can be avoided, so that the accuracy of gesture recognition can be improved. Secondly, by fusing the image vector of the key frame of the gesture to be recognized and the corresponding weight to generate the first gesture feature vector, the image vectors of the key frames corresponding to the gestures to be recognized made by the same user multiple times, with changes, and representing the same meaning can all be converted into first gesture feature vectors of fixed length, so that the number of matching templates can be reduced, and the overhead of the terminal device can be reduced. In addition, the gesture recognition result of the gesture to be recognized can be determined through the matching of the first gesture feature vector and the first gesture feature vector set containing gesture feature vectors whose gesture recognition results are known, the gesture recognition process is converted into a retrieval process of the first gesture feature vector, the gesture recognition process is simpler, the gesture to be recognized can be recognized more quickly, and the efficiency of gesture recognition is improved.

[0009] In a first possible implementation manner of the first aspect, the obtaining the first image vector and the weight of the first image vector comprises: obtaining a first image vector set; the first image vector set is a set composed of image vectors of images included in the first image sequence; calculating an attention weight of each image vector in the first image vector set according to an attention model; determining an image vector and the attention weight of which are greater than an attention threshold as the first image vector and the weight of the first image vector.

[0010] In the implementation manner, the attention weight of each frame of image in the first image sequence can be determined according to the importance of each frame of image in the first image sequence, that is, the importance of each frame of image to the gesture recognition result, and the key frame, that is, the first image vector, of the first image sequence can be extracted by comparing the attention weight with the attention threshold, and the weight of the first image vector is determined, so that the first image vector and the weight of the first image vector are more accurate.

[0011] In a second possible implementation manner of the first aspect, the generating the first gesture feature vector according to the first image vector and the weight of the first image vector comprises: performing encoding processing on the first image vector to obtain an encoded first image vector; and generating the first gesture feature vector according to the encoded first image vector and the weight of the first image vector.

[0012] In the implementation manner, the first image vector can be converted into a first gesture feature vector with a fixed length through encoding, so that the number of matching templates in gesture recognition can be reduced, and the cost of the terminal device can be reduced.

[0013] In a third possible implementation manner of the first aspect, the first gesture feature vector set comprises gesture feature vectors corresponding to at least one known gesture category, and one known gesture category corresponds to one known gesture recognition result; and the determining the gesture recognition result of the gesture to be recognized according to the first gesture feature vector and the first gesture feature vector set comprises: determining a first gesture category; the first gesture category is a known gesture category with the greatest similarity to the first gesture feature vector in the at least one known gesture category; and if the similarity between the first gesture feature vector and the first gesture category is greater than a preset similarity threshold, determining a known gesture recognition result corresponding to the first gesture category as the gesture recognition result of the gesture to be recognized.

[0014] In the present implementation, the gesture recognition result of the to-be-recognized gesture is determined by performing similarity matching between the first gesture feature vector and the gesture category corresponding to the known gesture feature vector contained in the first gesture feature vector set, thereby converting the gesture recognition process into a process of performing similarity matching between the first gesture feature vector and the known gesture category, making the gesture recognition process simpler and enabling the to-be-recognized gesture to be recognized more quickly, and improving the efficiency of gesture recognition.

[0015] With reference to the first aspect, in a fourth possible implementation of the first aspect, the method further includes: if the similarity between the first gesture feature vector and the first gesture category is less than or equal to a preset similarity threshold, obtaining a first gesture recognition result input by a user; and determining the first gesture recognition result as the gesture recognition result of the to-be-recognized gesture.

[0016] In the present implementation, for the to-be-recognized gesture corresponding to the first gesture feature vector for which the similarity matching is unsuccessful, the gesture recognition result input by the user is directly determined as the gesture recognition result of the to-be-recognized gesture, without the need for re-modeling and re-detection of the to-be-recognized gesture, thereby making the gesture recognition process simpler and more applicable.

[0017] With reference to the first aspect, in a fifth possible implementation of the first aspect, the method further includes: updating the first gesture feature vector set according to the first gesture feature vector and the gesture recognition result of the to-be-recognized gesture.

[0018] In the present implementation, when the first gesture feature vector set is updated, i.e., when the known gesture feature vector is updated, the new gesture feature vector can be directly updated to the first gesture feature vector set, without the need for identifying the newly added gesture through a model, thereby making the updating process of the known gesture feature vector simpler.

[0019] With reference to the first aspect, in a sixth possible implementation of the first aspect, the attention model is preset.

[0020] In the present implementation, the attention model is preset, which can enable the attention model to be obtained more simply and quickly, and is more applicable.

[0021] With reference to the first aspect, in a seventh possible implementation of the first aspect, the first gesture feature vector set is preset.

[0022] In the present implementation, the first gesture feature vector set is preset, which can enable the first gesture feature vector set to be obtained more simply and quickly, and is more applicable.

[0023] In a second aspect, the present application provides a gesture recognition device, comprising: an acquisition module, configured to acquire a first image vector and a weight of the first image vector; the first image vector being an image vector of a key frame in a first image sequence; the weight of the first image vector being a weight of the key frame in the first image sequence; the first image sequence being an image sequence generated according to a video image of a gesture to be recognized; the key frame being an image carrying hand information in the first image sequence; a processing module, configured to generate a first gesture feature vector according to the first image vector and the weight of the first image vector; and determine a gesture recognition result of the gesture to be recognized according to the first gesture feature vector and a first gesture feature vector set; the first gesture feature vector set being a set composed of gesture feature vectors whose gesture recognition results are known.

[0024] The device of the present implementation can extract a first image vector of a key frame in a first image sequence of a gesture to be recognized, acquire a weight of the first image vector representing the importance of the key frame in the first image sequence, then generate a first gesture feature vector according to the first image vector and the weight of the first image vector, and finally determine a gesture recognition result of the gesture to be recognized according to the first gesture feature vector and a first gesture feature vector set containing gesture feature vectors whose gesture recognition results are known. The device can extract the key frame and acquire information of important gesture features of the corresponding gesture to be recognized in the case of gestures made by different users for the same meaning and in the case of gestures made by the same user for the same meaning and randomly changing, and can improve the accuracy of gesture recognition without introducing noise or losing gesture feature information due to the fixed feature collection window. Secondly, the image vector of the key frame of the gesture to be recognized and the corresponding weight are fused to generate the first gesture feature vector, which can convert the image vectors of the key frames of the gestures to be recognized made by the same user multiple times, with changes, and representing the same meaning into first gesture feature vectors of a fixed length, thereby reducing the number of matching templates and reducing the overhead of the terminal device. Furthermore, the gesture recognition result of the gesture to be recognized can be determined by matching the first gesture feature vector with the first gesture feature vector set containing gesture feature vectors whose gesture recognition results are known, the gesture recognition process is converted into a retrieval process of the first gesture feature vector, the gesture recognition process is simpler, the gesture to be recognized can be recognized more quickly, and the efficiency of gesture recognition is improved.

[0025] In a first possible implementation manner of the second aspect, the obtaining module is specifically configured to: obtain a first image vector set; the first image vector set is a set composed of image vectors of images included in the first image sequence; calculate attention weights of each image vector in the first image vector set according to the attention model; and determine an image vector with an attention weight greater than an attention threshold as the first image vector and the attention weight as the weight of the first image vector.

[0026] The device of the present implementation can determine the attention weight of each frame of image in the first image sequence according to the importance of each frame of image in the first image sequence, that is, the importance of each frame of image to the gesture recognition result, extract the key frame of the first image sequence, that is, the first image vector, by comparing the attention weight with the attention threshold, and determine the weight of the first image vector, so that the obtained first image vector and the weight of the first image vector are more accurate.

[0027] In a second possible implementation manner of the second aspect, the processing module is specifically configured to: perform encoding processing on the first image vector to obtain an encoded first image vector; and generate a first gesture feature vector according to the encoded first image vector and the weight of the first image vector.

[0028] The device of the present implementation can convert the first image vector into a first gesture feature vector with a fixed length through encoding, so as to reduce the number of matching templates in gesture recognition and reduce the cost of the terminal device.

[0029] In a third possible implementation manner of the second aspect, the first gesture feature vector set includes gesture feature vectors with known gesture recognition results corresponding to at least one known gesture category, and one known gesture category corresponds to one known gesture recognition result; and the processing module is specifically configured to: determine a first gesture category; the first gesture category is a known gesture category with the greatest similarity to the first gesture feature vector in the at least one known gesture category; and if the similarity between the first gesture feature vector and the first gesture category is greater than a preset similarity threshold, determine that the known gesture recognition result corresponding to the first gesture category is the gesture recognition result of the gesture to be recognized.

[0030] The device of the present implementation can determine the gesture recognition result of the gesture to be recognized by performing similarity matching between the first gesture feature vector and the gesture category corresponding to the known gesture feature vector included in the first gesture feature vector set, so as to convert the gesture recognition process into a process of performing similarity matching between the first gesture feature vector and the known gesture category, make the gesture recognition process simpler, and more quickly recognize the gesture to be recognized, thereby improving the efficiency of gesture recognition.

[0031] In a fourth possible implementation manner of the second aspect, the processing module is further configured to: if the similarity between the first gesture feature vector and the first gesture category is less than or equal to a preset similarity threshold, obtain a first gesture recognition result input by a user; and determine the first gesture recognition result as the gesture recognition result of the gesture to be recognized.

[0032] The device of the present implementation can directly determine the gesture recognition result input by the user as the gesture recognition result of the gesture to be recognized for the gesture to be recognized corresponding to the first gesture feature vector for which the similarity matching is unsuccessful, without the need to re-model and re-detect the gesture to be recognized, so that the gesture recognition process is simpler and more applicable.

[0033] In a fifth possible implementation manner of the second aspect, the processing module is further configured to: update the first gesture feature vector set according to the first gesture feature vector and the gesture recognition result of the gesture to be recognized.

[0034] The device of the present implementation can directly update the new gesture feature vector into the first gesture feature vector set when updating the first gesture feature vector set, i.e., updating the known gesture feature vector, without the need to identify the new gesture through the model, so that the updating process of the known gesture feature vector is simpler.

[0035] In a sixth possible implementation manner of the second aspect, the attention model is preset.

[0036] The device of the present implementation can obtain the attention model more simply and quickly, and is more applicable.

[0037] In a seventh possible implementation manner of the second aspect, the first gesture feature vector set is preset.

[0038] The device of the present implementation can obtain the first gesture feature vector set more simply and quickly, and is more applicable.

[0039] In a third aspect, the embodiments of the present application provide a device, which includes a processor. When the processor executes a computer program or instructions in a memory, the method as described in the first aspect is executed.

[0040] In a fourth aspect, the embodiments of the present application provide a device, which includes a processor and a memory. The memory is configured to store a computer program or instructions. The processor is configured to execute the computer program or instructions stored in the memory, so that the device executes the corresponding method as shown in the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides an apparatus, the apparatus comprising a processor, a memory and a transceiver; the transceiver is configured to receive or send signals; the memory is configured to store computer programs or instructions; the processor is configured to call the computer programs or instructions from the memory to execute the method according to the first aspect.

[0042] In a sixth aspect, an embodiment of the present application provides an apparatus, the apparatus comprising a processor and an interface circuit; the interface circuit is configured to receive computer programs or instructions and transmit to the processor; the processor executes the computer programs or instructions to execute the corresponding method according to the first aspect.

[0043] In a seventh aspect, an embodiment of the present application provides a computer storage medium, the computer storage medium is configured to store computer programs or instructions, when the computer programs or instructions are executed, the method according to the first aspect is implemented.

[0044] In an eighth aspect, an embodiment of the present application provides a computer program product comprising computer programs or instructions, when the computer programs or instructions are executed, the method according to the first aspect is implemented.

[0045] To address the current issues of low gesture recognition accuracy and efficiency, as well as high equipment overhead, in dynamic gesture recognition, the present application provides a gesture recognition method and apparatus. In this method, a first image vector of a key frame in a first image sequence of the gesture to be recognized is first extracted, and a weight for the first image vector representing the importance of the key frame in the first image sequence is obtained. A first gesture feature vector is then generated based on the first image vector and the weight of the first image vector. Finally, a gesture recognition result for the gesture to be recognized is determined based on the first gesture feature vector and a first gesture feature vector set containing gesture feature vectors for which gesture recognition results are known. This implementation allows for the extraction of key frames to obtain important gesture feature information for the gesture to be recognized, even when gestures performed by different users with the same meaning differ, or when gestures performed by the same user with the same meaning vary randomly. This eliminates the problem of noise or loss of gesture feature information due to a fixed feature acquisition window, thereby improving gesture recognition accuracy. Secondly, by fusing the image vectors of the key frames of the gesture to be recognized and the corresponding weights to generate a first gesture feature vector, the image vectors of the key frames corresponding to gestures to be recognized that are performed multiple times by the same user and have different meanings but represent the same meaning can all be converted into first gesture feature vectors of fixed length, thereby reducing the number of matching templates and reducing the overhead of the terminal device. In addition, the gesture recognition result of the gesture to be recognized can be determined by matching the first gesture feature vector with a first gesture feature vector set containing gesture feature vectors with known gesture recognition results, converting the gesture recognition process into a retrieval process of the first gesture feature vector, making the gesture recognition process simpler, allowing the gesture to be recognized to be recognized more quickly, and improving the efficiency of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of an embodiment of the gesture recognition method provided in this application;

[0047] Figure 2 This is a structural block diagram of an embodiment of the gesture recognition device provided by this application;

[0048] Figure 3 This is a structural block diagram of an embodiment of the chip provided in this application. DETAILED DESCRIPTION

[0049] The technical solution of this application is described below in conjunction with the accompanying drawings.

[0050] In the description of the present application, unless otherwise specified, " / " means the meaning of "or", for example, A / B can mean A or B. "And / or" in this paper is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean: A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different.

[0051] It should be noted that in this application, "exemplary" or "for example" means to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0052] Embodiments of the present application provide a gesture recognition method and device. The technical solution provided by the present application can be applied to a terminal device, and the terminal device of the present application can be mobile or stationary. The terminal device can be a user equipment (UE), and exemplary UEs can be mobile phones, portable android devices (Pads), personal digital assistants (PDAs), etc. The terminal device can be a device carrying an operating system such as Android, iOS, Windows, or other operating systems. Or other operating system devices.

[0053] The embodiments of the gesture recognition method provided by the present application are described below.

[0054] Referring to Figure 1 , Figure 1 The flowchart of one embodiment of the gesture recognition method provided by the present application can include the following steps:

[0055] Step S101, acquiring a first image vector and a weight of the first image vector.

[0056] The first image vector is an image vector of a key frame in a first image sequence. The first image sequence can include one key frame or multiple key frames, and thus the number of first image vectors can be one or multiple. The weight of each first image vector is the weight of the key frame corresponding to the first image vector in the first image sequence, which indicates the importance of the key frame in the first image sequence. The first image sequence is an image sequence generated according to a video image of a gesture to be recognized.

[0057] When the terminal device recognizes the gesture to be recognized, the terminal device first captures the gesture to be recognized made by the user through the camera, and obtains a video image of the gesture to be recognized. The gesture to be recognized can be a dynamic gesture, and accordingly, the video image of the gesture to be recognized can include multiple images. Some of the multiple images carry hand information, and some of the multiple images do not carry hand information, that is, the multiple images include images carrying hand information and images not carrying hand information. The images carrying hand information are key frames in the multiple images. Generally, each key frame corresponds to a hand action, and the hand actions corresponding to all the key frames are connected together to form the gesture to be recognized.

[0058] The key frame in this embodiment can also be understood as follows: a sequence of actions includes multiple images, and a frame without information change is a normal frame, and a frame with information change or ending of change is a key frame. Specifically, the beginning or end of an action and the process of intermediate movement are key frames in the sequence. In addition, the key frame is often used in the field of images, and has some differences in different scenarios. The present application does not limit this, and the scheme of the present application can also refer to the known key frame recognition or extraction method in the field in different scenarios.

[0059] In a possible implementation manner, the first image sequence can be composed of all images included in the video image of the gesture to be recognized. In a possible implementation manner, a part of the images included in the video image of the gesture to be recognized can be selected to form the first image sequence.

[0060] The terminal device captures a video image of a to-be-recognized gesture, and obtains a first image sequence according to the video image of the to-be-recognized gesture. Each frame image in the first image sequence is converted into an image vector to obtain an image vector of the frame image. A first image vector set is formed by the image vectors of all the images in the first image sequence. The first image vector set is input into an attention model to calculate an attention weight of each image vector in the first image vector set. An image vector in the first image vector set with an attention weight greater than an attention threshold is determined as a first image vector, and the attention weight of the first image vector is determined as the weight of the first image vector. The image corresponding to the first image vector is a key frame of the first image sequence. The attention threshold can be set according to the requirements of an actual application scenario.

[0061] The attention model can be trained by a deep learning model based on an attention mechanism. The attention model can be used to extract key information contained in a plurality of original information. For example, the attention model can be used to extract keywords, key sentences or abstracts of a document. For another example, the attention model can also be used to extract a key frame of an image sequence, and the specific implementation can refer to the content of the above embodiments.

[0062] By extracting the image vector (i.e., the first image vector) of the key frame in the first image sequence through the attention model, the important gesture feature information of the to-be-recognized gesture can be obtained. Subsequently, the gesture recognition result of the to-be-recognized gesture can be accurately determined according to the important gesture feature information of the to-be-recognized gesture, so that the problem of inaccurate gesture recognition result caused by the loss or introduction of noise of the gesture feature information of the to-be-recognized gesture due to the fixed feature collection window can be avoided.

[0063] As can be known from the content of the above embodiments, in one possible implementation, the attention model needs to be obtained before the first image vector and the weight of the first image vector are obtained. The attention model can be obtained in various ways, for example:

[0064] In one possible implementation, the attention model can be generated by training. For example, a deep learning model based on an attention mechanism can be trained using a pre-set known image sequence set to generate the attention model. In this embodiment of this application, the known image sequence set used to train and generate the attention model is referred to as a first known image sequence set.

[0065] The known image sequence set includes a plurality of known image sequences. The known image sequence is an image sequence generated according to a video image of a known dynamic gesture. The way of generating the known image sequence according to the video image of the known dynamic gesture can refer to the way of generating the first image sequence in the foregoing embodiments, which will not be described here in detail.

[0066] The known dynamic gesture can be considered as a dynamic gesture whose gesture category is known (i.e., the gesture category is determined) and whose gesture meaning is known (i.e., the gesture meaning is determined), or can also be considered as a dynamic gesture whose gesture category is known and whose gesture recognition result is known (i.e., the gesture recognition result is determined). The gesture category refers to a category to which the dynamic gesture belongs. For example, the gesture category can be waving left, waving right, drawing a circle clockwise, drawing a circle counterclockwise, etc.

[0067] The gesture category corresponding to the known dynamic gesture can be defined as a known gesture category, the gesture recognition result corresponding to the known dynamic gesture can be defined as a known gesture recognition result, and the gesture meaning corresponding to the known dynamic gesture can be defined as a known gesture meaning. The known gesture category has a corresponding relationship with the known gesture recognition result or the known gesture meaning. In a possible implementation, the known gesture category can correspond to the known gesture recognition result or the known gesture meaning one by one. In a possible implementation, a plurality of known gesture categories can correspond to the same known gesture recognition result or known gesture meaning.

[0068] When the attention mechanism-based deep learning model is trained using the first known image sequence set, each known image sequence included in the first known image sequence set has a label recording a known gesture category corresponding to the known image sequence.

[0069] For example, when the attention mechanism-based deep learning model is trained using the first known image sequence set to generate an attention model, the following implementation can be used: a known image sequence is selected from the first known image sequence set one by one, each frame of image included in the known image sequence is subjected to image vectorization conversion to obtain an image vector of the frame of image, the image vectors of all images included in the known image sequence form a second image vector set, the second image vector set is input into the attention mechanism-based deep learning model, the attention mechanism-based deep learning model is trained until a loss function corresponding to the attention mechanism-based deep learning model converges, and the attention mechanism-based deep learning model obtained by the current training is determined as the attention model.

[0070] In a possible implementation, the trained attention model can also be preconfigured in the terminal device, and when the attention model is acquired, the preconfigured attention model can be acquired. In this way, the attention model can be quickly acquired, the acquisition process of the attention model is simpler, and the applicability is better.

[0071] In step S102, a first gesture feature vector is generated according to the first image vector and the weight of the first image vector.

[0072] After obtaining each first image vector and the weight of the first image vector, all the first image vectors are input into a first encoder for encoding processing, so as to convert all the first image vectors into the same coordinate system. Then, the weighted fusion processing is performed on all the first image vectors after the encoding processing and the weight of the corresponding first image vector, so as to generate an image vector with fixed dimension, and the image vector is determined as the first gesture feature vector. In this way, for the same meaning of the to-be-recognized gesture made by the user multiple times, the length of the first gesture feature vector obtained is fixed, and the gesture recognition process of the to-be-recognized gesture can be converted into the problem of searching and matching the first gesture feature vector with fixed length, so that the to-be-recognized gesture can be more simply and quickly recognized, the gesture recognition efficiency is higher, and the problem that a large number of matching templates need to be generated due to the randomness and variability of the dynamic gesture made by the user can be avoided, and the applicability is better.

[0073] From the above embodiment, in a possible implementation, before the first gesture feature vector is generated according to the first image vector and the weight of the first image vector, the first encoder needs to be obtained first. The implementation of obtaining the first encoder can include multiple modes, for example:

[0074] In a possible implementation, the first encoder can be obtained in the following manner: a plurality of third image vector sets are obtained; the preset encoder is trained using the plurality of third image vector sets to generate the first encoder. For example, the preset encoder can be an autoencoder (AE).

[0075] Optionally, when the attention model is trained using the first known image sequence set, each second image vector set is input into the deep learning model based on the attention mechanism, the attention weight of each image vector in the second image vector set is obtained, and then a set of image vectors with attention weight greater than an attention threshold in the second image vector set is determined as a third image vector set, so as to obtain a plurality of third image vector sets.

[0076] Optionally, a known image sequence in the second known image sequence set can be selected in sequence, each frame of image included in the known image sequence is subjected to image vectorization conversion to obtain an image vector of the frame of image, the image vectors of all images included in the known image sequence form a fourth image vector set, the fourth image vector set is input into the attention model, and an attention weight of each image vector in the fourth image vector set is calculated. A set of image vectors in the fourth image vector set whose attention weights are greater than an attention threshold value is determined as a third image vector set, so as to obtain a plurality of third image vector sets. It should be noted that the second known image sequence set can be the same as or different from the first known image sequence set.

[0077] Optionally, each frame of image included in the unknown image sequence can be subjected to image vectorization conversion to obtain an image vector of the frame of image, the image vectors of all images included in the unknown image sequence form a fifth image vector set, the fifth image vector set is input into the attention model, and an attention weight of each image vector in the fifth image vector set is calculated. A set of image vectors in the fifth image vector set whose attention weights are greater than an attention threshold value is determined as a third image vector set. In this way, a plurality of third image vector sets can be obtained according to a plurality of unknown image sequences.

[0078] The unknown image sequence is an image sequence generated according to a video image of an unknown dynamic gesture. The determination manner of the unknown image sequence can refer to the determination manner of the first image sequence in the foregoing embodiments, which will not be described here in detail. The unknown dynamic gesture is a dynamic gesture whose gesture category is unknown (i.e., the gesture category is not determined) and whose gesture meaning or gesture recognition result is unknown (i.e., the gesture meaning or gesture recognition result is not determined). The unknown dynamic gesture can be obtained by means of video shooting.

[0079] In a possible implementation manner, the trained first encoder can also be pre-set in the terminal device, so that the first encoder can be obtained by obtaining the pre-set first encoder. In this way, the first encoder can be quickly obtained, the process of obtaining the first encoder is simpler, and the applicability is better.

[0080] In step S103, a gesture recognition result of the to-be-recognized gesture is determined according to the first gesture feature vector and the first gesture feature vector set.

[0081] The first gesture feature vector set is a set of gesture feature vectors whose gesture recognition results or gesture meanings are known. In an embodiment of the present application, the gesture feature vector whose gesture recognition result or gesture meaning is known is defined as a known gesture feature vector. The known gesture feature vectors included in the first gesture feature vector set can correspond to at least one known gesture category, and a plurality of known gesture feature vectors can correspond to the same known gesture category. That is, the known gesture feature vectors included in the first gesture feature vector set belong to one or more known gesture categories, and each known gesture category can correspond to one known gesture feature vector or a plurality of known gesture feature vectors. According to the known gesture category corresponding to the known gesture feature vector, the known gesture recognition result or the known gesture meaning corresponding to the known gesture feature vector can be determined.

[0082] Before step S103 is performed, the first gesture feature vector set needs to be obtained. The first gesture feature vector set can be obtained in the following manner: a known image sequence is selected from the third known image sequence set one by one, each frame of image included in the known image sequence is subjected to image vectorization conversion to obtain an image vector of the frame of image, the image vectors of all images included in the known image sequence form a sixth image vector set, the sixth image vector set is input into the attention model, the attention weight of each image vector in the sixth image vector set is calculated, a set of image vectors in the sixth image vector set whose attention weights are greater than an attention threshold value is determined as a seventh image vector set, the seventh image vector set is input into the first encoder, all image vectors in the seventh image vector set are subjected to encoding processing, the image vectors after encoding processing are subjected to weighted fusion processing with the attention weights corresponding to the image vectors, a second gesture feature vector with a fixed dimension is generated, and a plurality of second gesture feature vectors form the first gesture feature vector set.

[0083] Since the second gesture feature vector is generated according to the known image sequence, the second gesture feature vector is a known gesture feature vector whose gesture recognition result is known, and the known gesture recognition result or the known gesture meaning corresponding to the second gesture feature vector is the known gesture recognition result or the known gesture meaning corresponding to the known dynamic gesture corresponding to the known image sequence.

[0084] It should be noted that the third known image sequence set can be the same as or different from the first known image sequence set or the second known image sequence set.

[0085] In a possible implementation, the first gesture feature vector set obtained according to the above embodiment can also be preconfigured in the terminal device, and when the first gesture feature vector set is needed, it can be directly obtained from the terminal device, so that the process of obtaining the first gesture feature vector set is simpler and faster, and the applicability is better.

[0086] In a possible implementation, after the first gesture feature vector set and the first gesture feature vector are obtained, the first gesture feature vector can be matched with at least one known gesture category corresponding to a known gesture feature vector (that is, a second gesture feature vector) included in the first gesture feature vector set, the similarity of the first gesture feature vector to each of the at least one known gesture category is calculated, and the known gesture category with the greatest similarity to the first gesture feature vector in the at least one known gesture category is determined as the first gesture category.

[0087] If the similarity of the first gesture feature vector to the first gesture category is greater than a preset similarity threshold, the known gesture recognition result or the known gesture meaning corresponding to the first gesture category is determined as the gesture recognition result or the gesture meaning of the gesture to be recognized. The preset similarity threshold can be set according to the needs of the actual application scenario.

[0088] Alternatively, if the similarity of the first gesture feature vector to the first gesture category is less than or equal to the preset similarity threshold, input information of a user is obtained, the input information of the user includes a second gesture category and a first gesture recognition result or a first gesture meaning, the first gesture recognition result or the first gesture meaning is determined as the gesture recognition result or the gesture meaning of the gesture to be recognized, and the second gesture category is determined as the gesture category corresponding to the first gesture feature vector. Optionally, in this case, the first gesture feature vector set can also be updated according to the first gesture feature vector, the second gesture category, and the first gesture recognition result or the first gesture meaning. Specifically, the first gesture feature vector can be added to the first gesture feature vector set as a known gesture feature vector, and a corresponding relationship among the first gesture feature vector, the second gesture category, the first gesture recognition result, and the first gesture meaning is established.

[0089] In a possible implementation, after the similarity of the first gesture feature vector to each of the at least one known gesture category corresponding to the known gesture feature vector included in the first gesture feature vector set is calculated, it can be determined whether there is a similarity greater than a preset similarity threshold, and if there is, the known gesture category corresponding to the greatest similarity in all similarities greater than the similarity threshold is determined as a third gesture category, and the known gesture recognition result or the known gesture meaning corresponding to the third gesture category is determined as the gesture recognition result or the gesture meaning of the gesture to be recognized.

[0090] Or, if there is no similarity greater than the preset similarity threshold, obtaining input information of the user, the input information of the user including a second gesture category and the first gesture recognition result or the first gesture meaning, determining the first gesture recognition result or the first gesture meaning as a gesture recognition result or a gesture meaning of the gesture to be recognized, and determining the second gesture category as a gesture category corresponding to the first gesture feature vector. Optionally, in this case, the first gesture feature vector set is also updated according to the first gesture feature vector, the second gesture category and the first gesture recognition result or the first gesture meaning. Specifically, the first gesture feature vector can be added to the first gesture feature vector set as a known gesture feature vector, and a corresponding relationship among the first gesture feature vector, the second gesture category, the first gesture recognition result or the first gesture meaning is established.

[0091] In the above embodiment, the first gesture feature vector is matched with at least one known gesture category corresponding to a known gesture feature vector (i.e. a second gesture feature vector) contained in the first gesture feature vector set, and the similarity between the first gesture feature vector and each of the at least one known gesture category is calculated, which can be implemented in the following manner: for each of the at least one known gesture category, the distance (for example, the distance can be a cosine distance) between the first gesture feature vector and each known gesture feature vector corresponding to the known gesture category is calculated, and then the mean value of all the distances corresponding to the known gesture category calculated is calculated, and the mean value calculated is taken as the similarity between the first gesture feature vector and the known gesture category.

[0092] In the gesture recognition method provided by the embodiment of the present application, first, a first image vector of a key frame in a first image sequence of a gesture to be recognized is extracted, and a weight of the first image vector used to represent the importance of the key frame in the first image sequence is obtained, then a first gesture feature vector is generated according to the first image vector and the weight of the first image vector, and finally, a gesture recognition result of the gesture to be recognized is determined according to the first gesture feature vector and a first gesture feature vector set containing gesture feature vectors with known gesture recognition results. In the scheme of the implementation manner, in the case that gestures made by different users for the same meaning are different, and in the case that gestures made by the same user for the same meaning are randomly changed, the information of important gesture features of the gesture to be recognized can be obtained through the extraction of the key frame, and the problem of noise introduction or gesture feature information loss caused by the fixed feature collection window can be avoided, so that the accuracy of gesture recognition can be improved. Secondly, by fusing the image vector of the key frame of the gesture to be recognized and the corresponding weight to generate the first gesture feature vector, the image vectors of the key frames corresponding to the gestures to be recognized made by the same user multiple times, changed and representing the same meaning can be all converted into the first gesture feature vector with a fixed length, so that the number of matching templates can be reduced, and the overhead of the terminal device can be reduced. In addition, the gesture recognition result of the gesture to be recognized can be determined through the matching of the first gesture feature vector and the first gesture feature vector set containing gesture feature vectors with known gesture recognition results, the gesture recognition process is converted into a retrieval process of the first gesture feature vector, the gesture recognition process is simpler, the gesture to be recognized can be recognized more quickly, and the efficiency of gesture recognition is improved.

[0093] Each method embodiment described herein can be an independent scheme or can be combined according to the inherent logic, and these schemes all fall within the protection scope of the present application.

[0094] It can be understood that the methods and operations implemented by the terminal device in each method embodiment described above can also be implemented by components (such as chips or circuits) that can be used for the terminal device.

[0095] The above describes the scheme provided by the embodiments of the present application mainly from the perspective of interaction between each network element. It can be understood that each network element, for example, the obtaining module and the processing module of the terminal device, comprises a hardware structure or a software module corresponding to each function or a combination of both, in order to implement the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0096] The embodiments of the present application can divide the functional modules of the terminal device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, there can be another division method. The following takes dividing each functional module according to each function as an example for description.

[0097] The above, in combination with Figure 1 The method provided by the embodiments of the present application is described in detail. The following, in combination with Figure 2 and Figure 3 The device provided by the embodiments of the present application is described in detail. It should be understood that the description of the device embodiments corresponds to the description of the method embodiments, and therefore, the content not described in detail can be referred to the above method embodiments. In order to be brief, it will not be described here.

[0098] Referring to Figure 2 , Figure 2 is a structural block diagram of one embodiment of the gesture recognition device provided by the present application. As Figure 2 indicated, the device 200 can comprise an obtaining module 201 and a processing module 202. The device 200 can be used to perform the actions performed by the terminal device in the above method embodiments.

[0099] For example, the obtaining module 201 can be used to obtain a first image vector and a weight of the first image vector; the first image vector is an image vector of a key frame in a first image sequence; the weight of the first image vector is a weight of the key frame in the first image sequence; and the first image sequence is an image sequence generated according to a video image of a gesture to be recognized.

[0100] The processing module 202 can be configured to generate a first gesture feature vector according to the first image vector and the weight of the first image vector; determine a gesture recognition result of the gesture to be recognized according to the first gesture feature vector and a first gesture feature vector set; and the first gesture feature vector set is a set of gesture feature vectors known to the gesture recognition result.

[0101] Optionally, the obtaining module 201 is specifically configured to: obtain a first image vector set; the first image vector set is a set of image vectors of images included in the first image sequence; calculate an attention weight of each image vector in the first image vector set according to an attention model; and determine an image vector and the attention weight of which are greater than an attention threshold as a first image vector and a weight of the first image vector.

[0102] Optionally, the processing module 202 is specifically configured to: perform encoding processing on the first image vector to obtain an encoded first image vector; and generate a first gesture feature vector according to the encoded first image vector and the weight of the first image vector.

[0103] Optionally, the first gesture feature vector set includes gesture feature vectors known to gesture recognition results, and the gesture feature vectors correspond to at least one known gesture category, one known gesture category corresponds to one known gesture recognition result; and the processing module 202 is specifically configured to: determine a first gesture category; the first gesture category is a known gesture category with the greatest similarity to the first gesture feature vector in the at least one known gesture category; and if the similarity between the first gesture feature vector and the first gesture category is greater than a preset similarity threshold, determine that a known gesture recognition result corresponding to the first gesture category is a gesture recognition result of the gesture to be recognized.

[0104] Optionally, the processing module 202 is further configured to: if the similarity between the first gesture feature vector and the first gesture category is less than or equal to a preset similarity threshold, obtain a first gesture recognition result input by a user; and determine that the first gesture recognition result is a gesture recognition result of the gesture to be recognized.

[0105] Optionally, the processing module 202 is further configured to: update the first gesture feature vector set according to the first gesture feature vector and the gesture recognition result of the gesture to be recognized.

[0106] Optionally, the attention model is preset.

[0107] Optionally, the first gesture feature vector set is preset.

[0108] That is, the device 200 can implement a gesture recognition method according to an embodiment of the present application. Figure 1The terminal device performs the steps or procedures in the method shown above. The apparatus 200 can include modules for performing the steps or procedures. Figure 1 The modules in the apparatus 200 perform the steps of the method shown above. The modules and other operations and / or functions described above are respectively for realizing the corresponding steps of the method shown above. Figure 1 The modules in the apparatus 200 perform the steps of the method shown above. The modules and other operations and / or functions described above are respectively for realizing the corresponding steps of the method shown above. Figure 1 The modules in the apparatus 200 perform the steps of the method shown above. The modules and other operations and / or functions described above are respectively for realizing the corresponding steps of the method shown above. Figure 1 The modules in the apparatus 200 perform the steps of the method shown above. The modules and other operations and / or functions described above are respectively for realizing the corresponding steps of the method shown above.

[0109] It should be understood that the specific process of each module performing the corresponding steps described above has been described in detail in the above method embodiments, and is omitted here for brevity.

[0110] In addition, the apparatus 200 can be a terminal device, which can perform the functions of the terminal device in the above method embodiments, or realize the steps or procedures performed by the terminal device in the above method embodiments.

[0111] The terminal device can include a processor and a transceiver. Optionally, the terminal device can further include a memory. The processor, the transceiver and the memory can communicate with each other through internal connection paths, and transfer control and / or data signals. The memory is used to store computer programs or instructions, and the processor is used to call and run the computer programs or instructions from the memory to control the transceiver to receive signals and / or send signals. Optionally, the terminal device can further include an antenna for transmitting uplink data or uplink control signaling output by the transceiver through wireless signals.

[0112] The processor can be combined with the memory to form a processing apparatus, and the processor is configured to execute the computer programs or instructions stored in the memory to realize the above functions. In specific implementation, the memory can be integrated in the processor, or independent of the processor. The processor can correspond to the processing module in the above description. Figure 2

[0113] The transceiver can also be referred to as a transceiving unit. The transceiver can include a receiver (or receiver circuit) and / or a transmitter (or transmitter circuit). The receiver is configured to receive signals, and the transmitter is configured to send signals.

[0114] It should be understood that the terminal device can realize the various processes related to the terminal device in the above method embodiments. The operations and / or functions of each module in the terminal device are respectively for realizing the corresponding procedures in the above method embodiments. For details, please refer to the description in the above method embodiments, and the detailed description is appropriately omitted here. ​

[0115] Optionally, the terminal device can further include a power supply for providing power supply for various devices or circuits in the terminal device.

[0116] In addition, in order to make the function of the terminal device more perfect, the terminal device can further include one or more of an input unit, a display unit, an audio circuit, a camera and a sensor, and the audio circuit can further include a speaker and a microphone.

[0117] The embodiments of the present application further provide a processing apparatus including a processor and an interface. The processor can be used to execute the method in the method embodiments.

[0118] It should be understood that the processing apparatus can be a chip. For example, refer to Figure 3 , Figure 3 The structural block diagram of one embodiment of the chip provided by the present application is shown. Figure 3 The chip shown can be a general-purpose processor or a special-purpose processor. The chip 300 includes a processor 301. The processor 301 can be used to support the device to execute the technical solutions shown in Figure 2 The device shown can execute the technical solutions shown in Figure 1 The device shown can execute the technical solutions shown in

[0119] Optionally, the chip 300 can further include a transceiver 302, which is used to accept the control of the processor 301 and is used to support the device to execute the technical solutions shown in Figure 2 The device shown can execute the technical solutions shown in Figure 1 The device shown can execute the technical solutions shown in Figure 3 Optionally, the chip 300 can further include a storage medium 303.

[0120] It should be noted that Figure 3The illustrated chip can be implemented using one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), application specific integrated chips (ASICs), system on chips (SoCs), central processor units (CPUs), network processors (NPs), digital signal processors (DSPs), micro controller units (MCUs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this application.

[0121] In the implementation process, each step of the above method can be completed by integrated logic circuit of hardware in the processor or instruction in the form of software. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution completion, or executed by hardware and software module combination in the processor. The software module can be located in the random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, register and other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0122] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with a signal processing capability. In the implementation process, the steps of the above method embodiments can be completed by an integrated logic circuit or an instruction in the form of software in the processor. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage, and the processor reads the information in the storage, and combines the hardware to complete the steps of the above method.

[0123] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct memory bus random access memory (DRAM). It should be noted that the memory of the system and method described herein is intended to include but not limited to these and any other suitable types of memory.

[0124] According to the method provided in the embodiments of the present application, the embodiments of the present application further provide a computer program product, which comprises a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer program or instructions cause the computer to perform the method of any one of the embodiments shown in the embodiments. Figure 1 According to the method provided in the embodiments of the present application, the embodiments of the present application further provide a computer program product, which comprises a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer program or instructions cause the computer to perform the method of any one of the embodiments shown in the embodiments.

[0125] According to the method provided in the embodiments of the present application, the embodiments of the present application further provide a computer storage medium, which stores a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer program or instructions cause the computer to perform the method of any one of the embodiments shown in the embodiments. Figure 1 According to the method provided in the embodiments of the present application, the embodiments of the present application further provide a computer program product, which comprises a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer program or instructions cause the computer to perform the method of any one of the embodiments shown in the embodiments.

[0126] In the above embodiments, the method can be implemented by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the method can be implemented in the form of a computer program product, entirely or partially. The computer program product comprises one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, the computer programs or instructions entirely or partially generate the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer programs or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer programs or instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, high-density digital video disc (DVD)) or a semiconductor medium (for example, solid state disc (SSD)) and the like.

[0127] As used in this description, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution and a component can be localized, partially localized, and / or distributed across two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as the Internet with other systems via the signal).

[0128] Those of skill in the art would understand that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or a combination of computer software and electronic hardware. The choice of hardware and software would depend on the particular application and general purpose nature of the technical solution. Those skilled in the art can use various methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present application.

[0129] Those of skill in the art would understand that, for the described convenience and brevity, the specific working processes of the above-described system, device and module can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0130] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, device or indirect coupling or communication connection of unit, which can be electrical, mechanical or other forms.

[0131] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0132] In addition, the functional modules in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit.

[0133] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage media that can store program codes.

[0134] The gesture recognition device, computer storage medium, computer program product, and chip provided by the embodiments of the present application are used to execute the method provided above, so the beneficial effects that can be achieved are referred to the beneficial effects of the method provided above, which will not be repeated here.

[0135] It should be understood that in each embodiment of the present application, the execution order of each step should be determined according to its function and inherent logic, and the size of the step serial number does not mean the execution order, and does not limit the implementation process of the embodiment.

[0136] Each part of the specification is described in a progressive manner, and the same or similar parts between each embodiment are referred to each other, and each embodiment mainly introduces the difference from other embodiments. In particular, the embodiments of the gesture recognition device, computer storage medium, computer program product, and chip are basically similar to the method embodiments, so the description is relatively simple, and the related parts are referred to the description in the method embodiments.

[0137] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such variations and modifications as fall within the scope of the application.

[0138] The embodiments of the application described above are not meant to limit the scope of the application.

Claims

1. A gesture recognition method, characterized by, include: Obtaining a first image vector and a weight of the first image vector; The first image vector is an image vector of a key frame in the first image sequence; The weight of the first image vector is the weight of the key frame in the first image sequence; the first image sequence is an image sequence generated based on a video image of a gesture to be recognized, the key frame is an image carrying hand information, a hand motion sequence is composed of multiple frames of images, the key frame is an image frame at the beginning of a motion, as well as when the motion changes and ends, and the hand motions corresponding to all key frames are connected together to form the gesture to be recognized. The weight of the first image vector is used to indicate the importance of the key frame in the first image sequence; performing encoding processing on the first image vector to obtain an encoded first image vector, and then performing weighted fusion processing on all the encoded first image vectors and the weights of the corresponding first image vectors to generate a first gesture feature vector, wherein the encoded first image vectors are located in the same coordinate system and the length of the first gesture feature vector is fixed; A gesture recognition result of the gesture to be recognized is determined according to the first gesture feature vector and a first gesture feature vector set; the first gesture feature vector set is a set of gesture feature vectors whose gesture recognition results are known.

2. The gesture recognition method of claim 1, wherein, The obtaining of the first image vector and the weight of the first image vector includes: Acquire a first image vector set; the first image vector set is a set consisting of image vectors of images included in the first image sequence; Calculating an attention weight for each image vector in the first set of image vectors according to the attention model; An image vector having an attention weight greater than an attention threshold and the attention weight being a first image vector and a weight of the first image vector are determined.

3. The gesture recognition method of claim 1 or 2, wherein, The gesture feature vectors whose gesture recognition results are known and included in the first gesture feature vector set correspond to at least one known gesture category, and one known gesture category corresponds to one known gesture recognition result; The determining the gesture recognition result of the gesture to be recognized according to the first gesture feature vector and the first gesture feature vector set includes: Determining a first gesture category; the first gesture category being the known gesture category having the greatest similarity to the first gesture feature vector among the at least one known gesture category; If the similarity between the first gesture feature vector and the first gesture category is greater than a preset similarity threshold, the known gesture recognition result corresponding to the first gesture category is determined as the gesture recognition result of the gesture to be recognized.

4. The gesture recognition method of claim 3, wherein, The method further comprises: If the similarity between the first gesture feature vector and the first gesture category is less than or equal to a preset similarity threshold, obtaining a first gesture recognition result input by the user; Determine the first gesture recognition result as the gesture recognition result of the gesture to be recognized.

5. The gesture recognition method of claim 4, wherein, The method further comprises: The first gesture feature vector set is updated according to the first gesture feature vector and the gesture recognition result of the gesture to be recognized.

6. A gesture recognition apparatus, characterized by include: an acquisition module, configured to acquire a first image vector and a weight of the first image vector; The first image vector is an image vector of a key frame in a first image sequence; The weight of the first image vector is a weight of the key frame in the first image sequence; the first image sequence is an image sequence generated according to a video image of a gesture to be recognized; the key frame is an image carrying hand information; one hand action sequence is composed of multiple image frames; the key frame is an image frame at the beginning of an action, at the occurrence of a change of the action, and at the end of the change; all hand actions corresponding to the key frames are connected together to form the gesture to be recognized; and the weight of the first image vector is used to indicate the importance of the key frame in the first image sequence. The processing module is configured to perform encoding processing on the first image vector to obtain an encoded first image vector, and perform weighted fusion processing on all the encoded first image vectors and the weights of the corresponding first image vectors to generate a first gesture feature vector, wherein the encoded first image vectors are located in the same coordinate system, and the length of the first gesture feature vector is fixed; the gesture recognition result of the gesture to be recognized is determined according to the first gesture feature vector and a first gesture feature vector set; and the first gesture feature vector set is a set composed of gesture feature vectors of known gesture recognition results.

7. The gesture recognition apparatus of claim 6, wherein, The acquisition module is specifically configured to: acquire a first image vector set; the first image vector set is a set composed of image vectors of images included in the first image sequence; calculate attention weights of each image vector in the first image vector set according to an attention model; determine an image vector and the attention weight of the image vector as a first image vector and a weight of the first image vector, if the attention weight of the image vector is greater than an attention threshold.

8. The gesture recognition apparatus according to claim 6 or 7, characterized in that, The first gesture feature vector set includes gesture feature vectors of known gesture recognition results, and the gesture feature vectors correspond to at least one known gesture category; one known gesture category corresponds to one known gesture recognition result. The processing module is specifically configured to: determine a first gesture category; the first gesture category is a known gesture category with the greatest similarity to the first gesture feature vector in the at least one known gesture category; if the similarity between the first gesture feature vector and the first gesture category is greater than a preset similarity threshold, determine that a known gesture recognition result corresponding to the first gesture category is a gesture recognition result of the gesture to be recognized.

9. The gesture recognition apparatus of claim 8, wherein, The processing module is further configured to: if the similarity between the first gesture feature vector and the first gesture category is less than or equal to the preset similarity threshold, acquire a first gesture recognition result input by a user; determine that the first gesture recognition result is a gesture recognition result of the gesture to be recognized.

10. The gesture recognition apparatus of claim 9, wherein, The processing module is further configured to: update the first gesture feature vector set according to the first gesture feature vector and the gesture recognition result of the gesture to be recognized.

11. An apparatus, comprising: The device includes a processor and a memory; The processor is configured to execute a computer program or instructions stored in the memory, and when the computer program or instructions are executed, the method in any one of claims 1 to 5 is executed.

12. An apparatus, comprising: The device includes a processor, a transceiver, and a memory; The transceiver is configured to receive or transmit signals; and the processor is configured to execute computer programs or instructions stored in the memory, and when the computer programs or instructions are executed, the apparatus implements the method of any one of claims 1-5.

13. A computer storage medium, characterized in that A computer program product comprising computer programs or instructions, which when executed by a computer, cause the method of any one of claims 1-5 to be performed.

14. A computer program product, characterised in that, A computer program product, which when executed on a computer, causes the computer to perform the method of any one of claims 1-5.

15. A chip, characterized by An apparatus comprising a processor coupled to a memory, the processor configured to execute computer programs or instructions stored in the memory, and when the computer programs or instructions are executed, the method of any one of claims 1-5 is performed.

Images