Gesture recognition methods, devices, equipment and storage media

Abstract

This application discloses a gesture recognition method, apparatus, device, and storage medium. The method involves extracting a hand region image from an acquired image to be recognized; determining a target descriptive text that matches the hand region image from several descriptive texts; using different descriptive texts to indicate different gesture types; and determining the gesture type indicated by the target descriptive text as the gesture recognition result of the image to be recognized. This application, by considering the image itself and introducing descriptive text to indicate the gesture type during gesture recognition, improves the accuracy of gesture recognition and thus enhances gesture interaction performance.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology, and more specifically, to a gesture recognition method, device, equipment, and storage medium. Background Technology

[0002] To enhance the diversity and convenience of human-computer interaction, gesture recognition-based human-computer interaction methods have emerged. When using electronic devices, users can control them by performing certain gestures or actions with their hands. During this process, the electronic device captures an image of the user's hand, identifies the gestures within the image to determine the interaction command, and then executes the corresponding operation.

[0003] However, current solutions rely solely on image-based gesture recognition, resulting in low accuracy and poor interactive performance. Summary of the Invention

[0004] In view of this, this application provides a gesture recognition method, apparatus, device, and storage medium to improve the accuracy of gesture recognition.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] A gesture recognition method, comprising:

[0007] Extract the hand region image from the acquired image to be identified;

[0008] Among several descriptive texts, a target descriptive text that matches the hand region image is determined; different descriptive texts are used to indicate different gesture types;

[0009] The gesture type indicated by the target description text is determined as the gesture recognition result of the image to be recognized.

[0010] Optionally, in the above method, determining the target descriptive text matching the hand region image from among several descriptive texts includes:

[0011] Calculate the similarity between the hand region image and each of the descriptive texts in the plurality of descriptive texts;

[0012] The descriptive text with the highest similarity to the hand region image is determined as the target descriptive text that matches the hand region image.

[0013] Optionally, in the above method, calculating the similarity between the hand region image and each of the descriptive texts in the plurality of descriptive texts includes:

[0014] Feature extraction is performed on the hand region image to obtain the hand appearance features;

[0015] Feature extraction is performed on each of the several descriptive texts to obtain the text features of each descriptive text;

[0016] For any descriptive text, the similarity between the hand region image and the descriptive text is calculated based at least on the hand appearance features and the text features of the descriptive text.

[0017] The above methods may optionally include:

[0018] Keypoint prediction is performed on the hand region image to determine the coordinate information of multiple key points of the hand in the hand region image;

[0019] Feature extraction is performed on the coordinate information of the multiple key points to obtain key point features;

[0020] The step of calculating the similarity between the hand region image and the descriptive text, based at least on the hand appearance features and the text features of any descriptive text, includes:

[0021] Based on the hand appearance features, the key point features, and the text features of any descriptive text, the similarity between the hand region image and any descriptive text is calculated.

[0022] Optionally, in the above method, calculating the similarity between the hand region image and the descriptive text based on the hand appearance features, the key point features, and the text features of any descriptive text includes:

[0023] The hand appearance features and the key point features are fused to obtain a first fused feature; the similarity between the first fused feature and the text features of any descriptive text is calculated as the similarity between the hand region image and any descriptive text.

[0024] or,

[0025] The key feature and the text feature of any descriptive text are fused to obtain a second fused feature; the similarity between the hand appearance feature and the second fused feature is calculated as the similarity between the hand region image and any descriptive text.

[0026] Optionally, in the above method, the feature extraction of the hand region image includes:

[0027] Multiple levels of feature extraction are performed on the hand region image; wherein,

[0028] The process of feature extraction at the target level across multiple levels includes:

[0029] The features of the hand region image output from the previous level of the target level and the features of the previous frame image of the hand region image are fused to obtain the third fused feature;

[0030] The third fusion feature is used to extract features at the target level to obtain the hand appearance features of the hand region image at the target level.

[0031] Optionally, in the above method, the process of extracting hand appearance features is implemented using a pre-trained appearance feature extraction module, and the process of extracting text features from each descriptive text is implemented using a pre-trained text feature extraction module; the appearance feature extraction module and the text feature extraction module are trained in the following manner:

[0032] The appearance feature extraction module and the text feature extraction module are self-supervised and trained using the first dataset to obtain a pre-trained appearance feature extraction module and a trained text feature extraction module; the first dataset includes a first type of gesture video and the corresponding descriptive text for the first type of gesture video, and each first type of gesture video has a gesture type;

[0033] The pre-trained appearance feature extraction module is fine-tuned in a supervised manner using the second dataset to obtain a trained appearance feature extraction module; the second dataset includes second-class gesture videos and multiple descriptive texts, each second-class gesture video is associated with a sample label representing the gesture type in the second-class gesture video; each second-class gesture video has a gesture type.

[0034] Optionally, in the above method, the process of extracting hand appearance features is implemented through a pre-trained appearance feature extraction module, the process of extracting text features of each descriptive text is implemented through a pre-trained text feature extraction module, and the process of extracting key point features is implemented through a key point feature extraction module; the appearance feature extraction module, the text feature extraction module, and the key point feature extraction module are trained in the following manner:

[0035] The appearance feature extraction module, the text feature extraction module, and the key point feature extraction module are self-supervised and trained using the first dataset to obtain a pre-trained appearance feature extraction module, a trained text feature extraction module, and a trained key point feature extraction module. The first dataset includes a first type of gesture video, coordinate information of multiple key points extracted from each frame of the first type of gesture video, and descriptive text corresponding to the first type of gesture video. Each first type of gesture video has a gesture type.

[0036] The pre-trained appearance feature extraction module is fine-tuned in a supervised manner using the second dataset to obtain a trained appearance feature extraction module. The second dataset includes a second type of gesture videos, coordinate information of multiple key points extracted from each frame of the second type of gesture videos, and multiple descriptive texts. Each second type of gesture video is associated with a sample label representing the gesture type in the second type of gesture video. Each second type of gesture video has a gesture type.

[0037] The above methods may optionally include:

[0038] Based on the gesture recognition results of the first T frames in the continuous T+1 frames, the target gesture recognition result corresponding to the T+1th frame in the continuous T+1 frames is determined.

[0039] Control commands are generated based on the target gesture recognition results.

[0040] The above methods are optional, wherein,

[0041] Each of the several descriptive texts is obtained by adding a word representing the gesture type to a text template;

[0042] in,

[0043] Different descriptive texts use the same text template;

[0044] or,

[0045] Different descriptive texts may use the same or different text templates.

[0046] A gesture recognition device, comprising:

[0047] The region image extraction module is used to extract the hand region image from the acquired image to be identified;

[0048] A text determination module is used to determine a target descriptive text that matches the hand region image from a plurality of descriptive texts; different descriptive texts are used to indicate different gesture types;

[0049] The recognition result determination module is used to determine the gesture type indicated by the target description text as the gesture recognition result of the image to be recognized.

[0050] A gesture recognition device, including a memory and a processor;

[0051] The memory is used to store programs;

[0052] The processor is configured to execute the program to implement the various steps of the gesture recognition method as described in any of the preceding claims.

[0053] A computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the various steps of the gesture recognition method as described in any of the preceding claims.

[0054] As can be seen from the above technical solutions, the gesture recognition method, apparatus, device, and storage medium provided in this application extract a hand region image from the acquired image to be recognized; determine a target descriptive text that matches the hand region image from several descriptive texts; different descriptive texts are used to indicate different gesture types; and the gesture type indicated by the target descriptive text is determined as the gesture recognition result of the image to be recognized. This application, when performing gesture recognition, introduces descriptive text to indicate the gesture type based on the image, thereby improving the accuracy of gesture recognition and thus improving gesture interaction performance. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating one implementation of the gesture recognition method disclosed in this application.

[0057] Figure 2 This is a flowchart illustrating an implementation of determining a target descriptive text that matches a hand region image from among several descriptive texts, as disclosed in an embodiment of this application.

[0058] Figure 3 This is a flowchart illustrating an implementation of calculating the similarity between a hand region image and various descriptive texts in several descriptive texts, as disclosed in an embodiment of this application.

[0059] Figure 4 This is an example diagram of key hand points disclosed in an embodiment of this application;

[0060] Figure 5 This is another implementation flowchart of calculating the similarity between a hand region image and any descriptive text based on hand appearance features, key point features, and text features of any descriptive text disclosed in the embodiments of this application.

[0061] Figure 6 This is a flowchart illustrating an implementation of calculating the similarity between a hand region image and any descriptive text based on hand appearance features, key point features, and text features of any descriptive text, as disclosed in an embodiment of this application.

[0062] Figure 7 This is a flowchart illustrating an implementation of feature extraction at multiple target levels as disclosed in an embodiment of this application.

[0063] Figure 8 This is a flowchart illustrating another implementation of the gesture recognition method disclosed in the embodiments of this application;

[0064] Figure 9 An example diagram illustrating the extraction of hand appearance features by the appearance feature extraction module disclosed in this application embodiment;

[0065] Figure 10 A schematic diagram illustrating how the multi-clue hint feature extraction module disclosed in this application obtains a multi-clue hint feature;

[0066] Figure 11 This is an example diagram of a self-supervised training of the appearance feature extraction module, text feature extraction module and key point feature extraction module using a first dataset, as disclosed in an embodiment of this application.

[0067] Figure 12 This is an example diagram illustrating supervised fine-tuning of the appearance feature extraction module using a second dataset, as disclosed in an embodiment of this application.

[0068] Figure 13 This is a schematic diagram of the structure of the gesture recognition device disclosed in an embodiment of this application;

[0069] Figure 14 This is a hardware structure block diagram of the gesture recognition device disclosed in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] This application is made in order to improve the accuracy of gesture recognition.

[0072] like Figure 1 The diagram shown is a flowchart of one implementation of the gesture recognition method provided in this application, which may include:

[0073] Step S101: Extract the hand region image from the acquired image to be identified.

[0074] The gesture recognition method provided in this application embodiment can be used in electronic devices. When the electronic device activates the gesture recognition function, the image acquisition device of the electronic device can acquire images in real time. When the user or the user's hand is within the field of view of the image acquisition device, the image acquisition device can acquire an image containing the user's hand.

[0075] The image to be identified can be any frame in the video stream captured by the image acquisition device.

[0076] The image to be identified can be a color image (RGB image), a near-infrared image, or a grayscale image, etc.

[0077] As an example, the image acquisition device can be an adaptive lighting image acquisition device, which can acquire ambient light brightness. When the brightness is greater than a brightness threshold, it uses a visible light imaging mode to acquire images, and when the brightness is less than or equal to the brightness threshold, it uses a near-infrared imaging mode to acquire images.

[0078] For the image to be identified, a rectangular bounding box of the hand region can be determined in the image. This bounding box is usually represented by the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2). The hand region image is extracted from the image to be identified based on the rectangular bounding box.

[0079] As an example, a pre-trained hand detection model can be used to determine the rectangular bounding box of the hand region in the image to be identified.

[0080] As an example, a skin-detection-based hand recognition algorithm can be used to determine the rectangular bounding box of the hand region in the image to be recognized.

[0081] As an example, a pixel-based hand recognition algorithm can be used to determine the rectangular bounding box of the hand region in the image to be recognized.

[0082] As an example, a hand recognition algorithm based on HSV (Hue, Saturation, Value) can be used to determine the rectangular bounding box of the hand region in the image to be recognized.

[0083] Step S102: Identify the target descriptive text that matches the hand region image from among several descriptive texts. Different descriptive texts are used to indicate different gesture types.

[0084] This application pre-sets a descriptive text library, which stores several descriptive texts, with different texts indicating different gesture types.

[0085] Optionally, the descriptive text can be in the form of any of the following sentences: This is a ____ gesture, This is a ____ video clip, This action is a ____ gesture, etc. For any of the above sentence types, fill in the blank with a gesture type word to obtain the descriptive text.

[0086] For example, taking the gesture type word "heart gesture" as an example, the descriptive text could be: This is a heart gesture, or, This is a video clip of a heart gesture, or, This action is a heart gesture, etc.

[0087] The gesture types in this application may include, but are not limited to, the following: thumbs up, heart, OK, V, waving to the left, waving to the right, waving upwards, waving downwards, etc.

[0088] The gesture types mentioned above can be further divided into static gestures and dynamic gestures. Static gestures refer to gestures that can be completed without hand movement, such as thumbs up, heart gestures, OK, and V gestures. Dynamic gestures, on the other hand, are gestures that require hand or finger movement to complete, such as swaying fingers left and right or waving in various directions.

[0089] Step S103: Determine the gesture type indicated by the target description text as the gesture recognition result of the image to be recognized.

[0090] The gesture recognition method provided in this application extracts a hand region image from an acquired image to be recognized; determines a target descriptive text that matches the hand region image from several descriptive texts; different descriptive texts are used to indicate different gesture types; and the gesture type indicated by the target descriptive text is determined as the gesture recognition result of the image to be recognized. This application, by considering the image itself and introducing descriptive text to indicate the gesture type during gesture recognition, improves the accuracy of gesture recognition, thereby enhancing gesture interaction performance.

[0091] Based on this application, users can perform gesture interaction with electronic devices within a range of 5 meters from the image acquisition device. In other words, this application can achieve both near-field control and far-field control of electronic devices, thus increasing the distance range for gesture control of electronic devices.

[0092] In addition, new descriptive text can be added to the descriptive text in this application at any time. The new descriptive text indicates a new gesture type and has scalability for recognizing new gesture types.

[0093] In an optional embodiment, each of the aforementioned descriptive texts is obtained by adding a word representing the gesture type to a text template.

[0094] Different descriptive texts use the same text template. That is, all descriptive texts use the same text template. For example, if all descriptive texts use the text template "This is a ____ gesture", then all the above descriptive texts will be in the sentence structure "This is a ____ gesture".

[0095] or,

[0096] Different descriptive texts may use the same or different text templates. That is, the aforementioned descriptive texts use at least two different text templates. For example, some descriptive texts use the text template "This is a ____ gesture," while others use the text template "This is a ____ video clip." In this case, the aforementioned descriptive texts use two different text templates: "This is a ____ gesture" and "This is a ____ video clip."

[0097] In an optional embodiment, a flowchart illustrating one method for determining the target descriptive text matching the hand region image from among several descriptive texts is shown below. Figure 2 As shown, it may include:

[0098] Step S201: Calculate the similarity between the hand region image and each descriptive text in several descriptive texts.

[0099] Optionally, feature extraction can be performed on the hand region image to obtain hand appearance features.

[0100] Feature extraction is performed on each descriptive text in a number of descriptive texts to obtain the text features of each descriptive text.

[0101] For any descriptive text, the similarity between the hand region image and the descriptive text is calculated based on the aforementioned hand appearance features and the text features of the descriptive text. As an example, the distance between the aforementioned hand appearance features and the text features of the descriptive text can be calculated to characterize the similarity between the hand region image and the descriptive text. The distance between the aforementioned hand appearance features and the text features of the descriptive text can be, but is not limited to, any of the following: Euclidean distance, cosine distance, Hamming distance, correlation coefficient, etc.

[0102] Step S202: The descriptive text with the highest similarity to the hand region image is determined as the target descriptive text that matches the hand region image.

[0103] In the above embodiments, the similarity between the hand region image and the descriptive text is calculated based solely on the hand's appearance features and the textual features of the descriptive text. In an optional embodiment, the similarity between the hand region image and the descriptive text can be calculated by combining the coordinate information of key hand points.

[0104] like Figure 3 The diagram shown is a flowchart illustrating an implementation of calculating the similarity between a hand region image and various descriptive texts in a plurality of descriptive texts, according to an embodiment of this application. The flowchart includes:

[0105] Step S301: Extract features from the hand region image to obtain hand appearance features.

[0106] Step S302: Perform keypoint prediction on the hand region image to determine the coordinate information of multiple key points of the hand in the hand region image.

[0107] A top-down regression scheme can be used to predict key points in hand region images.

[0108] like Figure 4 The image shown is an example diagram of key hand points provided in an embodiment of this application. This example includes a total of 21 key points, namely: wrist key point, thumb wrist-palm key point, thumb finger-palm key point, thumb interphalangeal key point, thumb fingertip key point, index finger-palm key point, proximal interphalangeal key point of index finger, distal interphalangeal key point of index finger, index fingertip key point, middle finger-palm key point, proximal interphalangeal key point of middle finger, distal interphalangeal key point of middle finger, middle fingertip key point, ring finger-palm key point, proximal interphalangeal key point of ring finger, distal interphalangeal key point of ring finger, ring fingertip key point, little finger-palm key point, proximal interphalangeal key point of little finger, distal interphalangeal key point of little finger, and little fingertip key point.

[0109] Step S303: Extract features from the coordinate information of the above-mentioned key points to obtain key point features.

[0110] Step S304: Extract features from each of the several descriptive texts to obtain the text features of each descriptive text.

[0111] In this application, steps S301, S302, and S304 can be executed simultaneously or sequentially. When executed sequentially, step S301 can be executed first, followed by steps S302 and S304; or step S302 can be executed first, followed by steps S301 and S304; or step S304 can be executed first, followed by steps S301 and S302, etc. This application does not specify the execution order of the three steps.

[0112] Step S305: Based on the hand appearance features, key point features, and text features of any descriptive text, calculate the similarity between the hand region image and any descriptive text.

[0113] This application performs hand detection and hand keypoint prediction based on hand region images, which can obtain refined hand appearance and structural information, improving the robustness of the gesture recognition system under different viewing angles, lighting and posture scenarios; on the other hand, by using customized descriptive text to indicate gesture types, the system can flexibly support the recognition of new gesture types, increasing the scalability of gesture recognition and greatly improving the user's interactive experience.

[0114] In summary, for any given descriptive text, this application calculates the similarity between the hand region image of the image to be identified and the descriptive text based at least on the hand appearance features of the image to be identified and the text features of the descriptive text.

[0115] like Figure 5 The diagram shown is a flowchart illustrating an implementation of this application's method for calculating the similarity between a hand region image and any descriptive text based on hand appearance features, key point features, and text features of any descriptive text. The method may include:

[0116] Step S501: Fuse the hand appearance features and key point features to obtain the first fused feature.

[0117] Optionally, the hand appearance features and key point features can be added together to obtain the first fused feature. Alternatively, the hand appearance features and key point features can be weighted and summed to obtain the first fused feature. Or, the mean of the hand appearance features and key point features can be calculated to obtain the first fused feature.

[0118] Step S502: Calculate the similarity between the first fused feature and the text features of any descriptive text, and use it as the similarity between the hand region image and any descriptive text.

[0119] The similarity between the first fusion feature and the text feature of any descriptive text can be determined by the distance between the first fusion feature and the text feature of any descriptive text. This distance can include, but is not limited to, any of the following: Euclidean distance, cosine distance, Hamming distance, correlation coefficient, etc.

[0120] like Figure 6 The diagram shown is another implementation flowchart of calculating the similarity between a hand region image and any descriptive text based on hand appearance features, key point features, and text features of any descriptive text, according to an embodiment of this application. It may include:

[0121] Step S601: Fuse the key point features and the text features of any descriptive text to obtain the second fused feature.

[0122] Optionally, the key feature can be added to the text features of any descriptive text to obtain the second fused feature. Alternatively, the key feature can be weighted and summed with the text features of any descriptive text to obtain the second fused feature. Or, the key feature can be calculated as the sum of the key feature and the text features of any descriptive text to obtain the second fused feature.

[0123] This application refers to the second fusion feature as the multi-clue cue feature.

[0124] Step S602: Calculate the similarity between the hand appearance features and the second fused features, as the similarity between the hand region image and any descriptive text.

[0125] The similarity between hand appearance features and second fusion features can be determined by the distance between the hand appearance features and the second fusion features. This distance can include, but is not limited to, any of the following: Euclidean distance, cosine distance, Hamming distance, correlation coefficient, etc.

[0126] In the above embodiments, gesture recognition is performed by integrating information from three dimensions: hand appearance, key points, and text, thereby further improving the accuracy of gesture recognition.

[0127] In an optional embodiment, one way to implement the above-described feature extraction of the hand region image can be:

[0128] This application performs feature extraction at multiple levels on a hand region image; a flowchart illustrating one implementation method for feature extraction at multiple target levels is shown below. Figure 7 As shown, it may include:

[0129] Step S701: Fuse the features of the hand region image output from the previous level of the target layer with the features of the hand region image of the previous frame to obtain the third fused feature. The third fused feature includes both the single-frame features of the hand region image of the image to be identified and the temporal features between the image to be identified and the previous frame image.

[0130] Optionally, the features of the hand region image can be multi-channel features. As an example, the features of the target channel in the hand region image can be replaced with the features of the target channel in the hand region image of the previous frame of the image to be recognized. By replacing the feature data of the corresponding channels, a temporal shift between different features is achieved, enhancing the temporal sequence of hand appearance features and further improving the accuracy of gesture recognition.

[0131] The target channel can be at least one channel in the features of the hand region image. If the target channel is at least two channels in the features of the hand region image, these at least two channels can be consecutive or non-consecutive. As an example, the target channel can be one-quarter of the channels in the features of the hand region image, and this one-quarter of the channels can be multiple consecutive channels in the features of the hand region image.

[0132] Optionally, when there are multiple target levels, these multiple target levels are evenly distributed among the aforementioned multiple levels.

[0133] Step S702: Extract features at the target level from the third fusion feature to obtain the hand appearance features of the hand region image at the target level.

[0134] In terms of appearance feature extraction, this application uses video stream as input to jointly extract image features of a single frame and temporal features between frames, so as to simultaneously support the recognition of static gestures and dynamic gestures.

[0135] In an optional embodiment, the process of extracting hand appearance features can be implemented using a pre-trained appearance feature extraction module, and the process of extracting text features from each descriptive text can be implemented using a pre-trained text feature extraction module. That is,

[0136] The pre-trained appearance feature extraction module extracts features from the hand region of the image to be recognized, thus obtaining the appearance features of the hand in the image to be recognized.

[0137] By using a pre-trained text feature extraction module, features are extracted from each of the several descriptive texts to obtain the text features of each descriptive text.

[0138] The aforementioned appearance feature extraction module and text feature extraction module can be trained in the following way:

[0139] The appearance feature extraction module and the text feature extraction module are trained in a self-supervised manner using the first dataset to obtain a pre-trained appearance feature extraction module and a trained text feature extraction module.

[0140] The first dataset includes videos of a first category of gestures, along with corresponding descriptive text for each video. Each video of a first category of gestures contains one gesture type. Each video of a first category of gestures corresponds to one descriptive text.

[0141] As an example, the gesture types in the first category of gesture videos in the first dataset may or may not include gesture types used in specific application scenarios. The first category of gesture videos can be gesture videos collected from the Internet.

[0142] The pre-trained appearance feature extraction module was fine-tuned in a supervised manner using the second dataset to obtain a well-trained appearance feature extraction module.

[0143] The second dataset includes second-class gesture videos and multiple descriptive texts. Each second-class gesture video is associated with a sample label representing the gesture type in the second-class gesture video; each second-class gesture video has a gesture type.

[0144] As an example, the gesture types in the second type of gesture video can be gesture types defined according to specific application scenarios, that is, the gesture types in the second type of gesture video are gesture types used in specific application scenarios.

[0145] In other words, in this application, the text feature extraction module is obtained through self-supervised training using the first dataset, while the appearance feature extraction module is obtained through two stages of training, in which the first stage is self-supervised training using the first dataset and the second stage is supervised training using the second dataset.

[0146] In an optional embodiment, one implementation of the above-described method for self-supervised training of the appearance feature extraction module and the text feature extraction module using the first dataset can be as follows:

[0147] The first gesture video is input into the appearance feature extraction module to obtain the hand appearance features of each frame in the first gesture video. The first gesture video can be a complete first-type gesture video, or a gesture video with a lower frame rate obtained by sampling the first-type gesture video.

[0148] The hand appearance features of the first gesture video are obtained based on the hand appearance features of each frame in the first gesture video.

[0149] The hand appearance features of each frame can be summed to obtain the hand appearance features of the first gesture video. Alternatively,

[0150] The mean value of the hand appearance features in each frame of the image can be calculated to obtain the hand appearance features of the first gesture video.

[0151] Input the descriptive text corresponding to the first gesture video into the text feature extraction module to obtain the text features of the descriptive text corresponding to the first gesture video.

[0152] With the goal of making the hand appearance features of the first gesture video approximate the text features of the corresponding descriptive text of the first gesture video, the parameters of the appearance feature extraction module and the text feature extraction module are updated to obtain the pre-trained appearance feature extraction module and the trained text feature extraction module.

[0153] Optionally, the similarity between the hand appearance features of the first gesture video and the text features of the corresponding descriptive text of the first gesture video can be calculated. With the goal of increasing the similarity between the hand appearance features of the first gesture video and the text features of the corresponding descriptive text of the first gesture video, the parameters of the appearance feature extraction module and the text feature extraction module can be updated to obtain a pre-trained appearance feature extraction module and a trained text feature extraction module.

[0154] As an example, for the n first gesture videos input to the appearance feature extraction module and the n descriptive texts corresponding to the n first gesture videos input to the text feature extraction module, the cosine similarity of the hand appearance features of each of the n first gesture videos and the text features of each of the n descriptive texts is calculated to obtain n×n cosine similarities. The video-text pairs consisting of the first gesture video and the corresponding descriptive text are used as positive sample pairs, and the video-text pairs consisting of the gesture video and other descriptive texts (i.e., non-corresponding descriptive texts) are used as negative sample pairs. The InfoNCE loss is calculated using the n×n cosine similarities. The parameters of the appearance feature extraction module and the text feature extraction module are updated based on the InfoNCE loss to obtain the pre-trained appearance feature extraction module and the trained text feature extraction module.

[0155] In an optional embodiment, the process of supervising the pre-trained appearance feature extraction module using the second dataset includes:

[0156] The second gesture video is input into a pre-trained appearance feature extraction module to obtain the hand appearance features of each frame in the second gesture video. The second gesture video can be a complete second-type gesture video, or it can be a gesture video with a lower frame rate obtained by sampling a second-type gesture video.

[0157] The hand appearance features of the second gesture video are obtained based on the hand appearance features of each frame in the second gesture video.

[0158] The hand appearance features of each frame can be summed to obtain the hand appearance features of the second gesture video. Alternatively,

[0159] The mean value of the hand appearance features in each frame of the image can be calculated to obtain the hand appearance features of the second gesture video.

[0160] Input multiple descriptive texts into the text feature extraction module to obtain the text features of each descriptive text in the multiple descriptive texts.

[0161] Based on the hand appearance features of the second gesture video and the text features of each of the multiple descriptive texts, the similarity between the second gesture video and each of the multiple descriptive texts is calculated.

[0162] The gesture types indicated by the multiple descriptive texts here are the gesture types used in specific application scenarios.

[0163] The gesture type indicated by the descriptive text that has the highest similarity to the second gesture video is determined as the gesture recognition result of the second gesture video.

[0164] With the goal of ensuring that the gesture type of the second gesture video, as represented by the similarity between the second gesture video and each of the various descriptive texts, is close to the sample label, the parameters of the pre-trained appearance feature extraction module are updated to obtain the trained appearance feature extraction module.

[0165] Optionally, the similarity between the second gesture video and each of the descriptive texts in multiple descriptive texts and the cross-entropy loss of the sample labels can be calculated. Based on the cross-entropy loss, the parameters of the pre-trained appearance feature extraction module can be updated to obtain the trained appearance feature extraction module.

[0166] While the pre-trained appearance feature extraction module is being fine-tuned in a supervised manner, the parameters of the text feature extraction module are frozen.

[0167] In an optional embodiment, the process of extracting hand appearance features is implemented using a pre-trained appearance feature extraction module, the process of extracting text features from each descriptive text is implemented using a pre-trained text feature extraction module, and the process of extracting key point features is implemented using a key point feature extraction module. That is,

[0168] The pre-trained appearance feature extraction module extracts features from the hand region of the image to be recognized, thus obtaining the appearance features of the hand in the image to be recognized.

[0169] By using a pre-trained text feature extraction module, features are extracted from each of the several descriptive texts to obtain the text features of each descriptive text.

[0170] The key point features are obtained by extracting the coordinate information of multiple key points on the hand using a pre-trained key point feature extraction module.

[0171] The aforementioned appearance feature extraction module, text feature extraction module, and key point feature extraction module were trained in the following manner:

[0172] The appearance feature extraction module, text feature extraction module, and key point feature extraction module were self-supervised using the first dataset to obtain a pre-trained appearance feature extraction module, a trained text feature extraction module, and a trained key point feature extraction module.

[0173] The first dataset includes a first type of gesture videos, coordinate information of multiple key points extracted from each frame of the first type of gesture videos, and descriptive text corresponding to each first type of gesture video. Each first type of gesture video contains one gesture type. Each first type of gesture video corresponds to one descriptive text.

[0174] As an example, the gesture types in the first category of gesture videos in the first dataset may or may not include gesture types used in specific application scenarios. The first category of gesture videos can be gesture videos collected from the Internet.

[0175] The pre-trained appearance feature extraction module was fine-tuned in a supervised manner using the second dataset to obtain a well-trained appearance feature extraction module.

[0176] The second dataset includes second-class gesture videos, coordinate information of multiple key points extracted from each frame of the second-class gesture videos, and multiple descriptive texts. Each second-class gesture video is associated with a sample label representing the gesture type in the second-class gesture video; each second-class gesture video has a gesture type.

[0177] As an example, the gesture types in the second type of gesture videos in the second dataset can be defined according to specific application scenarios, that is, the gesture types in the second type of gesture videos are the gesture types used in specific application scenarios.

[0178] In other words, in this application, the text feature extraction module and the key point feature extraction module are obtained through self-supervised training using the first dataset, while the appearance feature extraction module is obtained through two stages of training, in which the first stage is self-supervised training using the first dataset and the second stage is supervised training using the second dataset.

[0179] In an optional embodiment, one implementation of the above-described method for self-supervised training of the appearance feature extraction module, text feature extraction module, and key point feature extraction module using the first dataset can be as follows:

[0180] The first gesture video is input into the appearance feature extraction module to obtain the hand appearance features of each frame in the first gesture video. The first gesture video can be a complete first-type gesture video, or a gesture video with a lower frame rate obtained by sampling the first-type gesture video.

[0181] The hand appearance features of the first gesture video are obtained based on the hand appearance features of each frame in the first gesture video.

[0182] The hand appearance features of each frame can be summed to obtain the hand appearance features of the first gesture video. Alternatively,

[0183] The mean value of the hand appearance features in each frame of the image can be calculated to obtain the hand appearance features of the first gesture video.

[0184] The coordinate information of multiple key points in each frame of the first gesture video is input into the key point feature extraction module to obtain the key point features of each frame of the first gesture video.

[0185] The key point features of the first gesture video are obtained based on the key point features of each frame of the first gesture video.

[0186] The keypoint features of each frame can be summed to obtain the keypoint features of the first gesture video. Alternatively,

[0187] The mean value of the key point features of each frame image can be calculated to obtain the key point features of the first gesture video.

[0188] Input the descriptive text corresponding to the first gesture video into the text feature extraction module to obtain the text features of the descriptive text corresponding to the first gesture video.

[0189] With the goal of making the hand appearance features of the first gesture video approximate the text features of the corresponding descriptive text of the first gesture video, and making the key point features of the first gesture video approximate the text features of the corresponding descriptive text of the first gesture video, the parameters of the appearance feature extraction module, the text feature extraction module, and the key point feature extraction module are updated to obtain the pre-trained appearance feature extraction module, the trained text feature extraction module, and the trained key point feature extraction module.

[0190] Optionally, a first similarity can be calculated between the hand appearance features of the first gesture video and the text features of the corresponding descriptive text of the first gesture video, and a second similarity can be calculated between the key point features of the first gesture video and the text features of the corresponding descriptive text of the first gesture video. With the goal of increasing the first similarity and the second similarity, the parameters of the appearance feature extraction module, the text feature extraction module and the key point feature extraction module are updated to obtain a pre-trained appearance feature extraction module, a trained text feature extraction module and a trained key point feature extraction module.

[0191] As an example, the cosine similarity of the hand appearance features of each of the n gesture videos and the text features of each of the n descriptive texts can be calculated to obtain n×n first cosine similarities. The video-text pairs consisting of the gesture video and its corresponding descriptive text are used as positive sample pairs, and the video-text pairs consisting of the gesture video and other descriptive texts (i.e., non-corresponding descriptive texts) are used as negative sample pairs. The first InfoNCE loss is calculated using the n×n first cosine similarities. The cosine similarity of the keypoint features of each of the n gesture videos and the text features of each of the n descriptive texts can be calculated to obtain n×n second cosine similarities. The video-text pairs consisting of the gesture video and its corresponding descriptive text are used as positive sample pairs, and the video-text pairs consisting of the gesture video and other descriptive texts are used as negative sample pairs. The second InfoNCE loss is calculated using the n×n second cosine similarities. The parameters of the appearance feature extraction module, text feature extraction module, and key point feature extraction module are updated based on the first InfoNCE loss and the second InfoNCE loss to obtain the pre-trained appearance feature extraction module, the trained text feature extraction module, and the trained key point feature extraction module.

[0192] In an optional embodiment, one implementation of the above-described supervised fine-tuning of the pre-trained appearance feature extraction module using the second dataset can be:

[0193] The second gesture video is input into a pre-trained appearance feature extraction module to obtain the hand appearance features of each frame in the second gesture video. The second gesture video can be a complete second-type gesture video, or it can be a gesture video with a lower frame rate obtained by sampling a second-type gesture video.

[0194] The hand appearance features of the second gesture video are obtained based on the hand appearance features of each frame in the second gesture video.

[0195] The hand appearance features of each frame can be summed to obtain the hand appearance features of the second gesture video. Alternatively,

[0196] The mean value of the hand appearance features in each frame of the image can be calculated to obtain the hand appearance features of the second gesture video.

[0197] Input multiple descriptive texts into the text feature extraction module to obtain the text features of each descriptive text in the multiple descriptive texts.

[0198] The coordinate information of multiple key points in each frame of the second gesture video is input into the key point feature extraction module to obtain the key point features of each frame of the second gesture video.

[0199] The key point features of the second gesture video are obtained based on the key point features of each frame of the second gesture video.

[0200] The keypoint features of each frame can be summed to obtain the keypoint features of the second gesture video. Alternatively,

[0201] The mean value of the key point features of each frame image can be calculated to obtain the key point features of the second gesture video.

[0202] The text features of each descriptive text are added to the key point features of the second gesture video to obtain the multi-clue prompt features corresponding to each descriptive text.

[0203] Based on the hand appearance features of the second gesture video and the multi-cue cue features corresponding to each descriptive text in the multiple descriptive texts, the similarity between the second gesture video and each descriptive text in the multiple descriptive texts is calculated.

[0204] The gesture types indicated by the multiple descriptive texts here are the gesture types used in specific application scenarios.

[0205] The gesture type indicated by the descriptive text that has the highest similarity to the second gesture video is determined as the gesture recognition result of the second gesture video.

[0206] With the goal of ensuring that the gesture type of the second gesture video, as represented by the similarity between the second gesture video and each of the various descriptive texts, is close to the sample label, the parameters of the pre-trained appearance feature extraction module are updated to obtain the trained appearance feature extraction module.

[0207] Optionally, the similarity between the second gesture video and each of the descriptive texts in multiple descriptive texts and the cross-entropy loss of the sample labels can be calculated. Based on the cross-entropy loss, the parameters of the pre-trained appearance feature extraction module can be updated to obtain the trained appearance feature extraction module.

[0208] When performing supervised fine-tuning on the pre-trained appearance feature extraction module, the parameters of the text feature extraction module and the key point feature extraction module are frozen and unchanged.

[0209] This application introduces descriptive text indicating gesture types as prompts. On the one hand, it can flexibly support different gesture types by replacing the descriptive text. On the other hand, it can leverage massive open-source text data to enhance the generalization recognition ability of gesture recognition-related models (such as appearance feature extraction modules, text feature extraction modules, key point feature extraction modules, etc.) for new gesture types, alleviating the strong dependence of gesture recognition-related models on video data and manual annotation. Moreover, the descriptive text is manually designed, allowing this application to introduce prior knowledge and achieve refined descriptions of different gestures, reducing the training difficulty of fine-grained gesture recognition models.

[0210] In addition, by introducing key hand information as a cue, this application can explicitly model the structured information of the hand, thereby enhancing the model's overall ability to distinguish gestures with similar appearances.

[0211] In an optional embodiment, after acquiring at least T+1 frames of images and obtaining the gesture recognition results of the T+1 frames,

[0212] Based on the gesture recognition results of the previous T frames in a series of T+1 frames, the target gesture recognition result corresponding to the T+1th frame in a series of T+1 frames can be determined.

[0213] To ensure that the overall gesture recognition results remain stable over time, this application adds a time-series sliding window storage module. Its storage length T can be configured manually, and it can store the gesture recognition results of T consecutive frames of images. The time-series sliding window storage module is a time-series queue with a fixed length, which is dynamically maintained using the first-in-first-out principle.

[0214] When the gesture recognition results of the T frames stored in the time-series sliding window storage module are obtained as the gesture recognition results of the previous T frames, the target gesture recognition result of the T+1 frame can be obtained by voting on the gesture recognition results of the previous T frames. Specifically, the gesture type that appears most frequently in the gesture recognition results of the previous T frames can be determined as the target gesture recognition result of the T+1 frame, and this target gesture recognition result can be used for interactive control.

[0215] In other words, starting from frame T+1, each frame has two gesture recognition results. One result is calculated based on the descriptive text, and the other is obtained by voting on the gesture recognition results of the previous T frames. The calculated gesture recognition result is used to participate in the voting to obtain the target gesture recognition result. The target gesture recognition result obtained by voting on the recognition results of the previous T frames is used for gesture interaction control.

[0216] Control commands are generated based on the target gesture recognition result corresponding to the T+1 frame image. For example, waving to the left triggers the "next page" function. That is, if the target gesture recognition result is a leftward wave, the generated control command can be an instruction to turn to the next page.

[0217] Compared with the memory networks commonly used in existing gesture recognition systems, the time-series sliding window storage module proposed in this application has a smaller memory footprint and can make gesture interaction control more stable.

[0218] like Figure 8 The diagram shown is another implementation flowchart of the gesture recognition method provided in this application, which may include:

[0219] Step S801: Log in to the gesture recognition system.

[0220] Users can trigger the electronic device's login gesture recognition system by performing preset operations (such as clicking) on ​​the target interaction interface provided by the electronic device.

[0221] Alternatively, users can perform a preset operation (such as clicking) on ​​the target interaction interface provided by the electronic device to trigger the electronic device to display a login information input interface. After entering the username, password, and other information that needs to be verified on the login information input interface, the user logs into the gesture recognition system.

[0222] Step S802: The image acquisition device of the electronic device captures scene images in real time.

[0223] The image acquisition device can operate in visible light mode or near-infrared mode.

[0224] Step S803: Hand detection and key point detection, including: the processor of the electronic device performs hand region detection on each frame of the acquired image (denoted as the i-th frame image). If a hand region is detected, the hand region image is cropped from the acquired image, and key point detection is performed on the hand region image to obtain the coordinate information of 21 key points.

[0225] The coordinate information of the 21 key points can be recorded in the following form: {(x0, y0)…(x 20 y 20 )}.

[0226] Step S804: The processor's appearance feature extraction module extracts features from the hand region image to obtain hand appearance features.

[0227] like Figure 9 The image shown is an example of a hand appearance feature extraction module provided in this application for extracting hand appearance features. In this example, the appearance feature extraction module includes multiple convolutional modules, each containing multiple convolutional layers. This example adds a temporal offset module to at least some of the convolutional modules. For each convolutional module with an added temporal offset module, the added temporal offset module can be located before the middle convolutional layer of the convolutional module. The convolutional layer with the added temporal offset module is the aforementioned target layer. The temporal offset module is used to fuse the features of the i-th frame image output from the previous convolutional layer with the features of the (i-1)-th frame image to obtain a third fused feature corresponding to the i-th frame image. The third fused feature is then input into the convolutional layer corresponding to the temporal offset module to obtain the hand appearance features of the i-th frame image at the convolutional layer corresponding to the temporal offset module.

[0228] Step S805: Multi-clue hint feature extraction: The key point feature extraction module in the processor's multi-clue hint feature extraction module extracts features from the coordinate information of 21 key points to obtain key point features; the text feature extraction module in the multi-clue hint feature extraction module extracts features from K descriptive texts to obtain text features of K descriptive texts; the fusion module in the multi-clue hint feature extraction module adds the key point features to each text feature to obtain the multi-clue hint features corresponding to each descriptive text.

[0229] like Figure 10 The diagram shown illustrates how the multi-clue hint feature extraction module provided in this application obtains a multi-clue hint feature. Figure 10 In the text, "This is an OK gesture" is one of the K descriptive texts. Figure 10 The multi-cue cue features obtained correspond to the descriptive text "This is an OK gesture". Figure 10 By replacing the descriptive text in the list with other descriptive texts from among the K descriptive texts, we can obtain the multi-clue hint features corresponding to the other descriptive texts.

[0230] Figure 10 Both the keypoint features and text features in the text are vectors.

[0231] Step S806: Gesture recognition, including: for each frame image, the processor's gesture recognition module calculates the similarity between the i-th frame image and each descriptive text based on the hand appearance features of the i-th frame image and the multi-clue prompt features corresponding to each descriptive text; the gesture type indicated by the descriptive text with the highest similarity to the i-th frame image is determined as the gesture recognition result corresponding to the i-th frame image.

[0232] The gesture recognition module can calculate the distance between the hand appearance features of the i-th frame image and the multi-cue cue features corresponding to any descriptive text, so as to characterize the similarity between the i-th frame image and any descriptive text.

[0233] Step S807: Does the gesture type trigger a preset function? If yes, proceed to step S808; otherwise, return to step S803.

[0234] The target gesture recognition result can be determined by using the gesture recognition results of the previous T frames of the i-th frame. If the target gesture recognition result is a preset gesture type, the preset function corresponding to the target gesture recognition result is triggered; otherwise, no preset function corresponding to any preset gesture type is triggered.

[0235] Step S808: Trigger the electronic device to execute a preset function, and return to step S803.

[0236] like Figure 11The image shown is an example of self-supervised training of the appearance feature extraction module, text feature extraction module, and key point feature extraction module using a first dataset, as provided in an embodiment of this application. In this example, the first dataset is constructed from gesture videos collected from the internet. Each gesture video contains a gesture type, and a descriptive text indicating the gesture type in the video is annotated. Subsequently, the starting frame number of the corresponding gesture in the gesture video segment is manually annotated. Finally, for all video frames, a pre-trained hand detection and hand key point detection model is used to obtain the hand bounding box and key point coordinate information for each frame. The hand region image is cropped based on the hand bounding box to obtain the hand video segment.

[0237] In the joint training of the appearance feature extraction module, text feature extraction module, and key point feature extraction module, a batch optimization method is used for training. Each training batch samples n hand video clips and their corresponding descriptive text and key point location information from the first dataset. For each hand video clip, after obtaining the starting frame number of the video gesture, based on the starting frame number, t frames are randomly sampled from the subsequent frames (sampling improves the model's adaptability to videos with different frame rates) and fed into the appearance feature extraction module. The hand appearance features of each frame in the t frames are extracted, and the hand appearance features of the t frames are summed to obtain the result. The hand appearance features V of each hand video clip are denoted as V1, V2, ..., Vn. The descriptive text corresponding to each hand video clip is fed into the text feature extraction module to obtain the text features T of the descriptive text corresponding to each hand video clip, denoted as T1, T2, ..., Tn. The keypoint coordinate information corresponding to t frames of images sampled from the same hand video clip is input into the keypoint feature extraction module to obtain the keypoint features of each frame in the t frames. The keypoint features of the t frames are added together to obtain the keypoint features K corresponding to the same hand video clip. The keypoint features corresponding to n hand video clips are denoted as K1, K2, ..., Kn. After obtaining the three types of features V, text features T, and keypoint features K, each type of feature is first normalized. For example, since each type of feature is a vector, for each vector, the magnitude of the vector can be calculated, and the elements of the vector can be divided by the magnitude to obtain the normalized result of the vector (i.e., the normalized feature). Using normalized features, the cosine similarity matrix C1 between hand appearance features and text features, and the cosine similarity matrix C2 between keypoint features and text features are calculated through vector dot product. Samples on the diagonal of the similarity matrix are designated as positive pairs, and others as negative pairs. A first InfoNCE loss and a second InfoNCE loss are calculated, and the parameters of the appearance feature extraction module, text feature extraction module, and keypoint feature extraction module are updated based on these losses. During self-supervised training, the hand appearance features output by the appearance feature extraction module and the keypoint features output by the keypoint feature extraction module are aligned with their corresponding text features, enhancing the semantic attributes of the hand appearance features and keypoint features, and improving their generalization ability.

[0238] like Figure 12 The diagram shown is an example of supervised fine-tuning of the appearance feature extraction module using a second dataset, as provided in an embodiment of this application. In this example, the parameters of the appearance feature extraction module are fine-tuned according to the specific gesture types required by the scene to achieve higher recognition performance for the required gestures.

[0239] In this example, for each hand video segment, the multi-cue cue feature extraction module receives multiple descriptive texts, each indicating a different gesture type. Figure 12 The example shown only displays four descriptive texts. In practical applications, more descriptive texts can be used. In this example, the different descriptive texts use the sentence structure "a __ gesture". For any gesture video clip, the gesture video clip is input into the appearance feature extraction module to obtain the hand appearance features of each frame of the gesture video clip. The hand appearance features of each frame of the gesture video clip are summed to obtain the hand appearance feature V of the gesture video clip, denoted as V1. For any frame of the hand video clip, after obtaining the key point features of the hand key point coordinates of the frame image and the text features of the four descriptive texts, the multi-cue cue feature extraction module sums the key point features of each frame of the hand video clip to obtain the key point features of the hand video clip. Then, the text features of each descriptive text are summed with the key point features of the hand video clip to obtain the multi-cue cue features M corresponding to each descriptive text, denoted as M1, M2, M3, and M4 respectively. Normalize M and V respectively, and use the normalized M and V to calculate the cosine similarity between any hand video segment and each descriptive text. Use cross-entropy loss to maximize the similarity between the cosine similarity and the real gesture type label. Based on the similarity, fine-tune the features of the appearance feature extraction module. During the fine-tuning process, the parameters of the multi-cue cue feature extraction module are frozen and unchanged.

[0240] During the fine-tuning phase, effective gestures used in specific application scenarios are labeled with data at a lower cost to improve the recognition performance of effective gestures.

[0241] In summary, this application transforms gesture recognition from a closed-set classification problem into an open-set retrieval problem by manually designing descriptive text. This modeling approach improves the system's ability to recognize gestures without pre-defined categories and enhances its scalability, while reducing the data collection and annotation costs required when adding new gesture categories.

[0242] Corresponding to the method embodiments, this application also provides a gesture recognition device. A schematic diagram of the structure of the gesture recognition device provided in this application embodiment is shown below. Figure 13 As shown, it may include:

[0243] The module comprises a region image extraction module 1301, a text determination module 1302, and a recognition result determination module 1303; wherein,

[0244] The region image extraction module 1301 is used to extract the hand region image from the acquired image to be identified;

[0245] The text determination module 1302 is used to determine a target descriptive text that matches the hand region image from a plurality of descriptive texts; different descriptive texts are used to indicate different gesture types;

[0246] The recognition result determination module 1303 is used to determine the gesture type indicated by the target description text as the gesture recognition result of the image to be recognized.

[0247] The gesture recognition device provided in this application improves the accuracy of gesture recognition by introducing descriptive text indicating the gesture type in addition to considering the image during gesture recognition. This, in turn, enhances the performance of gesture interaction.

[0248] In an optional embodiment, the text determination module 1302 is used to:

[0249] Calculate the similarity between the hand region image and each of the descriptive texts in the plurality of descriptive texts;

[0250] The descriptive text with the highest similarity to the hand region image is determined as the target descriptive text that matches the hand region image.

[0251] In an optional embodiment, when the text determination module 1302 calculates the similarity between the hand region image and each of the descriptive texts in the plurality of descriptive texts, it is used to:

[0252] Feature extraction is performed on the hand region image to obtain the hand appearance features;

[0253] Feature extraction is performed on each of the several descriptive texts to obtain the text features of each descriptive text;

[0254] For any descriptive text, the similarity between the hand region image and the descriptive text is calculated based at least on the hand appearance features and the text features of the descriptive text.

[0255] In an optional embodiment, the text determination module 1302 is further configured to:

[0256] Key point location prediction is performed on the hand region image to determine the coordinate information of multiple key points of the hand in the hand region image;

[0257] Feature extraction is performed on the coordinate information of the multiple key points to obtain key point features;

[0258] The step of calculating the similarity between the hand region image and the descriptive text, based at least on the hand appearance features and the text features of any descriptive text, includes:

[0259] Based on the hand appearance features, the key point features, and the text features of any descriptive text, the similarity between the hand region image and any descriptive text is calculated.

[0260] In an optional embodiment, when the text determination module 1302 calculates the similarity between the hand region image and the descriptive text based on the hand appearance features, the key point features, and the text features of any descriptive text, it is used to:

[0261] The hand appearance features and the key point features are fused to obtain a first fused feature; the similarity between the first fused feature and the text features of any descriptive text is calculated as the similarity between the hand region image and any descriptive text.

[0262] or,

[0263] The key feature and the text feature of any descriptive text are fused to obtain a second fused feature; the similarity between the hand appearance feature and the second fused feature is calculated as the similarity between the hand region image and any descriptive text.

[0264] In an optional embodiment, when the text determination module 1302 extracts features from the hand region image, it is used to:

[0265] Multiple levels of feature extraction are performed on the hand region image; wherein,

[0266] The process of feature extraction at the target level across multiple levels includes:

[0267] The features of the hand region image output from the previous level of the target level and the features of the previous frame image of the hand region image are fused to obtain the third fused feature;

[0268] The third fusion feature is used to extract features at the target level to obtain the hand appearance features of the hand region image at the target level.

[0269] In an optional embodiment, the text determination module 1302 extracts hand appearance features using a pre-trained appearance feature extraction module and extracts text features for each descriptive text using a pre-trained text feature extraction module; the appearance feature extraction module and the text feature extraction module are trained in the following manner:

[0270] The appearance feature extraction module and the text feature extraction module are self-supervised and trained using the first dataset to obtain a pre-trained appearance feature extraction module and a trained text feature extraction module; the first dataset includes a first type of gesture video and the corresponding descriptive text for the first type of gesture video, and each first type of gesture video has a gesture type;

[0271] The pre-trained appearance feature extraction module is fine-tuned in a supervised manner using the second dataset to obtain a trained appearance feature extraction module; the second dataset includes second-class gesture videos and multiple descriptive texts, each second-class gesture video is associated with a sample label representing the gesture type in the second-class gesture video; each second-class gesture video has a gesture type.

[0272] In an optional embodiment, the text determination module 1302 extracts hand appearance features through a pre-trained appearance feature extraction module, extracts text features of each descriptive text through a pre-trained text feature extraction module, and extracts key point features through a key point feature extraction module; the appearance feature extraction module, the text feature extraction module, and the key point feature extraction module are trained in the following manner:

[0273] The appearance feature extraction module, the text feature extraction module, and the key point feature extraction module are self-supervised and trained using the first dataset to obtain a pre-trained appearance feature extraction module, a trained text feature extraction module, and a trained key point feature extraction module. The first dataset includes a first type of gesture video, coordinate information of multiple key points extracted from each frame of the first type of gesture video, and descriptive text corresponding to the first type of gesture video. Each first type of gesture video has a gesture type.

[0274] The pre-trained appearance feature extraction module is fine-tuned in a supervised manner using the second dataset to obtain a trained appearance feature extraction module. The second dataset includes a second type of gesture videos, coordinate information of multiple key points extracted from each frame of the second type of gesture videos, and multiple descriptive texts. Each second type of gesture video is associated with a sample label representing the gesture type in the second type of gesture video. Each second type of gesture video has a gesture type.

[0275] In an optional embodiment, the device further includes:

[0276] The control module is used to determine the target gesture recognition result corresponding to the T+1th frame image in the continuous T+1 frame images based on the gesture recognition results of the previous T frames in the continuous T+1 frame images; and to trigger the generation of control commands based on the target gesture recognition result.

[0277] In an alternative embodiment, wherein,

[0278] Each of the several descriptive texts is obtained by adding a word representing the gesture type to a text template;

[0279] in,

[0280] Different descriptive texts use the same text template;

[0281] or,

[0282] Different descriptive texts may use the same or different text templates.

[0283] The gesture recognition device provided in this application embodiment can be applied to gesture recognition devices, such as PC terminals, cloud platforms, servers, and server clusters. Optionally, Figure 14 A hardware block diagram of the gesture recognition device is shown, with reference to... Figure 14 The hardware structure of a gesture recognition device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0284] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0285] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0286] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0287] The memory stores a program, which the processor can call. The program is used for:

[0288] Extract the hand region image from the acquired image to be identified;

[0289] Among several descriptive texts, a target descriptive text that matches the hand region image is determined; different descriptive texts are used to indicate different gesture types;

[0290] The gesture type indicated by the target description text is determined as the gesture recognition result of the image to be recognized.

[0291] Optionally, the refined and extended functions of the program can be found in the description above.

[0292] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0293] Extract the hand region image from the acquired image to be identified;

[0294] Among several descriptive texts, a target descriptive text that matches the hand region image is determined; different descriptive texts are used to indicate different gesture types;

[0295] The gesture type indicated by the target description text is determined as the gesture recognition result of the image to be recognized.

[0296] Optionally, the refined and extended functions of the program can be found in the description above.

[0297] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0298] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0299] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0300] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0301] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0302] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0303] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0304] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition method, characterized in that, include: Extract the hand region image from the acquired image to be identified; Feature extraction is performed on the hand region image to obtain the hand appearance features; Key point prediction is performed on the hand region image to determine the coordinate information of multiple key points of the hand in the hand region image; feature extraction is performed on the coordinate information of the multiple key points to obtain key point features; Feature extraction is performed on each descriptive text in a set of descriptive texts to obtain the text features of each descriptive text; different descriptive texts are used to indicate different gesture types; Based on the hand appearance features, the key point features, and the text features of any descriptive text, the similarity between the hand region image and any descriptive text is calculated; the descriptive text with the highest similarity to the hand region image is determined as the target descriptive text that matches the hand region image. The gesture type indicated by the target description text is determined as the gesture recognition result of the image to be recognized.

2. The method according to claim 1, characterized in that, The step of calculating the similarity between the hand region image and the descriptive text based on the hand appearance features, the key point features, and the text features of any descriptive text includes: The hand appearance features and the key point features are fused to obtain a first fused feature; the similarity between the first fused feature and the text features of any descriptive text is calculated as the similarity between the hand region image and any descriptive text. or, The key feature and the text feature of any descriptive text are fused to obtain a second fused feature; the similarity between the hand appearance feature and the second fused feature is calculated as the similarity between the hand region image and any descriptive text.

3. The method according to claim 1, characterized in that, The feature extraction of the hand region image includes: Multiple levels of feature extraction are performed on the hand region image; wherein, The process of feature extraction at the target level across the multiple levels includes: The features of the hand region image output from the previous level of the target level and the features of the previous frame image of the hand region image are fused to obtain the third fused feature; The third fusion feature is used to extract features at the target level to obtain the hand appearance features of the hand region image at the target level.

4. The method according to claim 1, characterized in that, The process of extracting hand appearance features is implemented through a pre-trained appearance feature extraction module, and the process of extracting text features from each descriptive text is implemented through a pre-trained text feature extraction module; the appearance feature extraction module and the text feature extraction module are trained in the following manner: The appearance feature extraction module and the text feature extraction module are self-supervised and trained using the first dataset to obtain a pre-trained appearance feature extraction module and a trained text feature extraction module; the first dataset includes a first type of gesture video and the corresponding descriptive text for the first type of gesture video, and each first type of gesture video has a gesture type; The pre-trained appearance feature extraction module is fine-tuned in a supervised manner using the second dataset to obtain a trained appearance feature extraction module. The second dataset includes second-class gesture videos and multiple descriptive texts. Each second-class gesture video is associated with a sample label representing the gesture type in the second-class gesture video; each second-class gesture video has a gesture type.

5. The method according to claim 1, characterized in that, The process of extracting hand appearance features is implemented through a pre-trained appearance feature extraction module; the process of extracting text features from each descriptive text is implemented through a pre-trained text feature extraction module; and the process of extracting key point features is implemented through a key point feature extraction module. The appearance feature extraction module, the text feature extraction module, and the key point feature extraction module are trained in the following manner: The appearance feature extraction module, the text feature extraction module, and the key point feature extraction module are self-supervised and trained using the first dataset to obtain a pre-trained appearance feature extraction module, a trained text feature extraction module, and a trained key point feature extraction module. The first dataset includes a first type of gesture video, coordinate information of multiple key points extracted from each frame of the first type of gesture video, and descriptive text corresponding to the first type of gesture video. Each first type of gesture video has a gesture type. The pre-trained appearance feature extraction module is fine-tuned in a supervised manner using the second dataset to obtain a trained appearance feature extraction module. The second dataset includes a second type of gesture videos, coordinate information of multiple key points extracted from each frame of the second type of gesture videos, and multiple descriptive texts. Each second type of gesture video is associated with a sample label that represents the gesture type in the second type of gesture video; each second type of gesture video has a gesture type.

6. The method according to claim 1, characterized in that, Also includes: Based on the gesture recognition results of the first T frames in the continuous T+1 frames, the target gesture recognition result corresponding to the T+1th frame in the continuous T+1 frames is determined. Control commands are generated based on the target gesture recognition results.

7. The method according to claim 1, characterized in that, in, Each of the several descriptive texts is obtained by adding a word representing the gesture type to a text template; in, Different descriptive texts use the same text template; or, Different descriptive texts may use the same or different text templates.

8. A gesture recognition device, characterized in that, include: The region image extraction module is used to extract the hand region image from the acquired image to be identified; The text determination module performs feature extraction on the hand region image to obtain hand appearance features; performs keypoint prediction on the hand region image to determine the coordinate information of multiple key points of the hand in the hand region image; extracts features from the coordinate information of the multiple key points to obtain key point features; extracts features from each of the several descriptive texts to obtain text features of each descriptive text; different descriptive texts are used to indicate different gesture types; based on the hand appearance features, the key point features, and the text features of any descriptive text, calculates the similarity between the hand region image and any descriptive text; and determines the descriptive text with the highest similarity to the hand region image as the target descriptive text that matches the hand region image. The recognition result determination module is used to determine the gesture type indicated by the target description text as the gesture recognition result of the image to be recognized.

9. A gesture recognition device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the gesture recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the gesture recognition method as described in any one of claims 1-7.

Images