Gesture recognition method and device, electronic device, and storage medium

Abstract

The application provides a gesture recognition method and device, electronic equipment and storage medium, the method comprises the following steps: inputting at least one image into an encoding module of a gesture recognition model, and outputting an encoding feature corresponding to each image; each image comprises at least one object; inputting the encoding feature corresponding to each image and at least one query feature into a decoding module corresponding to the image in the gesture recognition model, and outputting a decoding feature; the query feature corresponds to the decoding module; the object and the query feature correspond to each other, and the decoding feature comprises a gesture feature of the corresponding object which is queried from the encoding feature through the query feature; inputting the decoding feature into a recognition module corresponding to the image in the gesture recognition model, and determining a gesture category of each object in the image based on the gesture feature; and determining a gesture of each object based on the gesture category of each object in each image. Based on this, the gesture of multiple objects can be recognized simultaneously through one model, and the efficiency of gesture recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a gesture recognition method and device, an electronic device, and a storage medium. BACKGROUND

[0002] The gesture recognition method can recognize gestures of target objects in images to obtain gesture recognition results, which can be applied to gesture unlocking or gesture control and other gesture application scenarios.

[0003] The existing gesture recognition method based on a deep learning network model mainly obtains gesture recognition results by sequentially executing two steps. The first step is a target detection step, which detects human detection box information of target objects in each image according to an input image sequence, and associates each target object in time sequence to obtain a "spatiotemporal region" corresponding to the target object in combination with spatial position information. The second step is a gesture recognition step, which performs gesture recognition on each target object one by one according to the "spatiotemporal region" of each target object obtained in the first step to obtain gesture recognition results of each target object.

[0004] The network model used in the gesture recognition method described above is a detection model corresponding to the target detection step and a recognition model corresponding to the gesture recognition step. The two models are independent models obtained by independent training, and each time gesture recognition is applied, the two steps of "detection first and then recognition" are required. When the image sequence to be recognized contains multiple target objects, the "spatiotemporal region" of each target object in each image needs to be detected first, and then the "spatiotemporal region" of each target object is input into the recognition model to recognize the gestures of each target object one by one. This implementation method that requires two steps to complete recognition greatly increases the time consumption of the whole process and reduces the recognition efficiency. SUMMARY

[0005] The present application provides a gesture recognition method, device, electronic device, and storage medium to solve the problem of low gesture recognition efficiency in the prior art and achieve the purpose of improving gesture recognition efficiency.

[0006] The present application provides a gesture recognition method, which comprises:

[0007] At least one image is input into an encoding module of a gesture recognition model to obtain encoding features corresponding to each image output by the encoding module; each of the images includes at least one object.

[0008] The encoding features corresponding to each image are input into a decoding module corresponding to the image in the gesture recognition model, to obtain decoding features output by the decoding module; the at least one query feature corresponds to the decoding module; the objects and the query features correspond one by one, and the decoding features include gesture features of corresponding objects queried from the encoding features by the query features;

[0009] The decoding features are input into a recognition module corresponding to the image in the gesture recognition model, a gesture category of each object in the image is determined based on the gesture features, and the gesture category is output by the recognition module;

[0010] Based on the gesture category of each object in each image, the gesture of each object is determined.

[0011] According to the gesture recognition method provided by the application, the number of images is at least two frames, and the gesture recognition model includes at least two decoding modules; the method further comprises:

[0012] The decoding features corresponding to each image are input into a decoding module corresponding to the image in the gesture recognition model, to obtain decoding features output by the decoding module; the at least one query feature corresponds to the decoding module; the objects and the query features correspond one by one, and the decoding features include gesture features of corresponding objects queried from the encoding features by the query features;

[0013] According to the gesture recognition method provided by the application, the number of images is at least two frames, and the gesture recognition model includes at least two decoding modules; the method further comprises:

[0014] The method further comprises:

[0015] The decoding features are input into a position detection module corresponding to the image in the gesture recognition model, position information of each object in the image is determined based on the position features, and the position information is output by the position detection module;

[0016] Based on the position information of each object in at least two frames of images, the trajectory of each object is determined.

[0017] According to the gesture recognition method provided by the application, the decoding features further include category features of positions of corresponding objects queried from the encoding features by the query features;

[0018] The method further comprises:

[0019] The decoding features are input into a category recognition module corresponding to the image in the gesture recognition model, a category of the position of each object in the image is determined based on the category features, and the category is output by the category recognition module, the category including foreground or background.

[0020] According to the gesture recognition method provided by the application, the gesture of each object in each image is determined based on the gesture category of each object in each image, which comprises the following steps:

[0021] Based on the category of the position of each object in each image, foreground objects are screened from each of the images;

[0022] Based on the gesture category of each foreground object in each of the images, the gesture of each foreground object is determined.

[0023] According to the gesture recognition method provided by the application, the gesture recognition model is obtained based on the following method:

[0024] A sample image sequence is obtained, and the sample image sequence comprises at least one frame of sample image collected based on time sequence;

[0025] Each of the at least one frame of sample image is input into an initial encoding module of an initial gesture recognition model, so as to obtain sample encoding features corresponding to each of the sample image output by the initial encoding module; each of the sample image comprises at least one sample object;

[0026] For the sample encoding features corresponding to each of the sample image, the sample encoding features and at least one sample query feature are input into an initial decoding module corresponding to the sample image in the initial gesture recognition model, so as to obtain sample decoding features output by the initial decoding module; the at least one sample query feature corresponds to the initial decoding module; the sample object and the sample query feature correspond to each other, and the sample decoding features comprise sample gesture features of the corresponding sample object queried from the sample encoding features through the sample query feature;

[0027] Each of the sample decoding features is input into an initial recognition module corresponding to the sample image in the initial gesture recognition model, based on the sample gesture features, a predicted gesture category of each sample object in the sample image is determined, and the predicted gesture category is output through the initial recognition module;

[0028] Based on the predicted gesture category and label information, model parameters of the initial gesture recognition model are updated, so as to obtain the gesture recognition model, and the label information is used to represent the gesture category in the sample image.

[0029] According to the gesture recognition method provided by the application, the method further comprises:

[0030] Based on the predicted gesture category and the label information, the sample query feature is iteratively updated, and finally obtained sample query feature is determined as the query feature.

[0031] The application further provides a gesture recognition device, which comprises:

[0032] An encoding unit is configured to input at least one frame of image into an encoding module of a gesture recognition model to obtain an encoding feature corresponding to each image output by the encoding module, wherein each image comprises at least one object;

[0033] A decoding unit is configured to input the encoding feature corresponding to each image and at least one query feature into a decoding module corresponding to the image in the gesture recognition model, to obtain a decoding feature output by the decoding module, wherein the at least one query feature corresponds to the decoding module, the object and the query feature correspond to each other, and the decoding feature comprises a gesture feature of the corresponding object queried from the encoding feature by the query feature;

[0034] A recognition unit is configured to input the decoding feature into a recognition module corresponding to the image in the gesture recognition model, to determine a gesture category of each object in the image based on the gesture feature, and to output the gesture category by the recognition module.

[0035] A determination unit is configured to determine a gesture of each object based on the gesture category of each object in each image.

[0036] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the gesture recognition method according to any one of the above when executing the program.

[0037] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the gesture recognition method according to any one of the above.

[0038] The application further provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the gesture recognition method according to any one of the above.

[0039] This invention provides a gesture recognition method, apparatus, electronic device, and storage medium. The method inputs at least one image frame into the encoding module of a gesture recognition model to obtain coded features corresponding to each image output by the encoding module. Each image includes at least one object, allowing feature extraction for each object in the image. For the coded features corresponding to each image, the coded features and at least one query feature are input into the decoding module corresponding to the image in the gesture recognition model to obtain decoded features output by the decoding module. At least one query feature corresponds to the decoding module; there is a one-to-one correspondence between the object and the query feature. The decoded feature includes the gesture features of the corresponding object retrieved from the coded features through the query feature. Thus, one-to-one gesture feature extraction can be performed on each object in the image using the one-to-one correspondence of the query features, achieving the purpose of multi-path parallel gesture feature information extraction through query features. The decoded features are input into the recognition module corresponding to the image in the gesture recognition model. Based on the gesture features, the gesture category of each object in the image is determined, and the recognition module outputs the gesture category. In this way, based on the gesture features of each object and the corresponding recognition module, the gesture category of each object in each image can be determined separately. Furthermore, based on the gesture category of each object in each image, the gesture of each object can be determined. Therefore, the gesture recognition model can be used to perform gesture recognition on each object in an image sequence, achieving the goal of recognizing gestures of multiple objects simultaneously using a single model. This reduces the implementation stages of gesture recognition, shortens the time consumption of multi-object gesture recognition, and improves the efficiency of gesture recognition. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the gesture recognition method provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic block diagram of the gesture recognition model provided in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of the gesture recognition device provided in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0046] It should be noted that the serial numbers of the described objects in the present application, such as "first", "second", etc., are only used to distinguish the described objects, and do not have any order or technical meaning.

[0047] The existing gesture recognition method based on a neural network model needs to go through two stages of "detection first and then recognition", and different models are used in each stage to obtain the output results required in each stage. Therefore, the existing method has the problem of low recognition efficiency. In particular, in the scene of simultaneously recognizing gestures of multiple people, the problem of low recognition efficiency of the existing method is particularly prominent. When there are multiple objects to be recognized in an image sequence, that is, there are multiple associated "spatiotemporal regions", for the recognition model in the recognition stage, the "spatiotemporal region" of each object needs to be judged one by one, which increases the time required for recognition, so that the method does not have real-time performance in the scene of recognizing gestures of multiple people, affecting the use of real interactive scenes.

[0048] In view of the above problems, the gesture recognition method provided in the embodiments of the present application is based on a deep learning end-to-end gesture recognition model. Query features are applied in the gesture recognition model, so that the model can detect, match and associate each object in the image in an adaptive manner, and at the same time complete the recognition of the gestures of the objects. The detection function and the recognition function of the model can be jointly optimized, the recognition process can be changed from two stages to one stage, the stage of the recognition process is reduced, complex object matching rules are avoided, gestures of each object in the input image sequence can be recognized based on one model, and the gesture recognition result of all objects in the image is output. The problem of low inference efficiency of the model in the multiple-person scene is solved, the recognition efficiency of gesture recognition is greatly improved, and the timeliness requirement of gesture recognition application in the multiple-person scene is met. The gesture recognition method provided in the embodiments of the present application will be described below in combination with Figure 1 and Figure 2 The gesture recognition method provided in the embodiments of the present application will be described below in combination with

[0049] Figure 1is a flowchart of a gesture recognition method provided by an embodiment of the present application. The embodiment of the present application can be applied to various types of gesture recognition scenarios, for example, can be applied to a motion sensing game scenario of multi-person gesture recognition, etc. The execution subject of the method can be an electronic device such as a smart phone, a smart television, a tablet computer, a virtual reality terminal, a computer, or a specially designed gesture recognition device, or can be a gesture recognition apparatus provided in the electronic device, which can be implemented by software, hardware, or a combination of both. As shown in Figure 1 The gesture recognition method includes steps 110 to 140.

[0050] In step 110, at least one image is input into an encoding module of a gesture recognition model to obtain encoding features corresponding to each image output by the encoding module; each image includes at least one object.

[0051] Specifically, the at least one image can be a single image, or one or more images in an image sequence. The image sequence includes at least one image, and each image has a time sequence, for example, the image sequence can be composed of multiple images in a video, and the multiple images can be extracted from a video by dense sampling or equal-interval sampling to form an image sequence. The images in the image sequence can be various types of images, including but not limited to red-green-blue (RGB) color mode images, infrared images, or depth images, etc. Each image in the image sequence includes at least one object, for example, the image includes several people. When the image sequence is input into the gesture recognition model, the gestures formed by the hand movements of each person in the image can be recognized.

[0052] The gesture recognition model can be a neural network model for recognizing gestures of at least one object in an image. The gesture recognition model includes an encoding module, which can be a module for extracting features of each image in an image sequence. Each image included in the image sequence is input into the encoding module of the gesture recognition model, the encoding module can extract image information in each image to obtain encoding features representing the image information of the corresponding image, and can output the encoding features corresponding to each image. The encoding module can be a functional module composed of various types of network structure layers based on an attention mechanism, including but not limited to attention network structures such as a Transformer network structure and its variants.

[0053] For example, the image sequence includes 5 consecutive images, namely image 1 to image 5, each of which includes object A, object B, and object C. When the image sequence is input into the encoding module of the gesture recognition model, the 5 images can be encoded to extract encoding features corresponding to each image, which can represent the feature information of object A, object B, and object C in image 1 to image 5, respectively.

[0054] In step 120, for each image corresponding encoding feature, the encoding feature and at least one query feature are input into the decoding module corresponding to the image in the gesture recognition model to obtain the decoding feature output by the decoding module. The at least one query feature corresponds to the decoding module. The object and the query feature correspond one-to-one, and the decoding feature includes the gesture feature of the corresponding object queried from the encoding feature by the query feature.

[0055] Specifically, the gesture recognition model can include a decoding module, which can be used to decode each encoding feature and the query feature corresponding to each encoding feature to obtain a decoding feature. The decoding module can be a functional module composed of various types of network structure layers based on attention mechanism, including but not limited to cross-attention network structures such as Transformer network structure and its variants. The decoding module corresponds to each image in the image sequence, which can be understood as each image being decoded by its corresponding decoding module.

[0056] The query feature can be used for association of different object spatio-temporal relationships in the image and extraction of gesture information. The query feature can be, for example, a query vector. The setting method of the query feature includes but is not limited to a fixed parameter query vector and a learnable query vector, etc. The encoding feature and the query feature corresponding to the object in the image are input into the decoding module, and the attention mechanism of the decoding module is used for feature extraction and enhancement, so that the decoding feature corresponding to each image can be obtained. The decoding feature includes the gesture feature of the corresponding object queried from the encoding feature by the query feature, and the gesture feature can be a feature used to represent the gesture information of the corresponding object.

[0057] For example, image 1 to image 5 correspond to respective decoding modules, each decoding module corresponds to three query features, and the encoding features corresponding to image 1 to image 5 and the query features corresponding to object A, object B and object C one-to-one are input into each decoding module for decoding. Each decoding module corresponding to image 1 to image 5 can output the decoding feature of the corresponding image, and each decoding feature includes the gesture feature representing the gesture information of object A, object B and object C.

[0058] In step 130, the decoding feature is input into the recognition module corresponding to the image in the gesture recognition model, the gesture category of each object in the image is determined based on the gesture feature, and the gesture category is output by the recognition module.

[0059] Specifically, the gesture recognition model can include a recognition module corresponding to each image, and the recognition module can be a classifier capable of classifying the gestures of the objects. For example, the classifier can include but is not limited to a fully connected layer, a convolutional layer, and a multilayer perceptron, etc.

[0060] The recognition module can be used to classify the gesture features. For each image in the image sequence, the decoded features are input into the recognition module corresponding to the image, and each recognition module can determine and classify the gesture features representing each object in the corresponding image to obtain the gesture category corresponding to each object, and output the gesture category of each object.

[0061] For example, the decoded features of the images 1 to 5 are input into the recognition modules corresponding to the images 1 to 5 respectively to determine and classify the gesture features. The recognition module corresponding to the image 1 outputs the gesture category of object A, the gesture category of object B and the gesture category of object C in the image 1; the recognition module corresponding to the image 2 outputs the gesture category of object A, the gesture category of object B and the gesture category of object C in the image 2; and so on, and the recognition module corresponding to the image 5 outputs the gesture category of object A, the gesture category of object B and the gesture category of object C in the image 5.

[0062] In step 140, the gesture of each object is determined based on the gesture category of each object in each image.

[0063] Specifically, based on the gesture category of each object in each image output by the recognition module, the gesture category of each object in each frame of image can be determined. For each object, the gesture category of the object in each frame of image can be analyzed and determined in time sequence according to the order of the images in the image sequence, and thus the gesture of each object can be obtained.

[0064] For example, the five gesture categories of object A in the images 1 to 5 can be analyzed and determined in time sequence based on the time order of the image sequence, and thus the gesture of object A can be obtained, and the gestures of object B and object C can also be obtained. The gesture category of an object can be analyzed and determined in time sequence, for example, the gesture categories of the object can be sequentially concatenated to obtain a combined gesture category, and the gesture of the object can be determined based on the corresponding relationship between the combined gesture category and the gesture, wherein the corresponding relationship between the combined gesture category and the gesture can be a preset mapping relationship for determining the gesture.

[0065] The gesture recognition method provided by the embodiment of the present application inputs at least one frame of image into an encoding module of a gesture recognition model, obtains the encoding features corresponding to each image output by the encoding module, and each image includes at least one object, so that the feature information of each object in the image can be extracted. The encoding features corresponding to each image and at least one query feature are input into the decoding module corresponding to the image in the gesture recognition model, and the decoding features output by the decoding module are obtained; the at least one query feature corresponds to the decoding module; the object and the query feature correspond to each other, and the decoding feature includes the gesture feature of the corresponding object in the encoding feature obtained by querying the query feature, so that the one-to-one gesture feature extraction of each object in the image can be realized through the one-to-one query feature, and the purpose of multi-path parallel gesture feature information extraction through the query feature can be achieved. The decoding feature is input into the recognition module corresponding to the image in the gesture recognition model, the gesture category of each object in the image is determined based on the gesture feature, and the gesture category is output through the recognition module, so that the gesture category of each object in each image can be determined respectively based on the gesture feature of each object and the recognition module corresponding to the image, and the gesture of each object can be determined based on the gesture category of each object in each image. Based on this, the gesture of each object in the image sequence can be recognized through the gesture recognition model, the purpose of recognizing the gesture of multiple objects at the same time based on one model can be achieved, the implementation stage of gesture recognition is reduced, the time consumption of multi-object gesture recognition is shortened, and the efficiency of gesture recognition is improved.

[0066] For example, when the objects in at least two frames of images are subjected to gesture recognition, in order to respectively establish the corresponding association of each object in each image at different time points, the information learned by the query feature corresponding to the previous frame of image can be transmitted to the query feature corresponding to the next frame of image through a query feature information transmission mechanism, so that each query feature can be self-learned to more accurately obtain the information of each object in the image through the query information, and the accuracy of recognition is improved.

[0067] In an embodiment, the number of images is at least two frames, and the gesture recognition model includes at least two decoding modules; the method further includes: splicing the decoding features output by the decoding module corresponding to the previous frame of image and at least one initial query feature to obtain at least one query feature of the decoding module corresponding to the current frame of image.

[0068] Specifically, the gesture recognition model includes at least two decoding modules, and each decoding module has an initial query feature. The initial query feature can be an initially set query feature, for example, the same initial query vector can be set for each decoding module. The decoding feature and the at least one initial query feature are spliced, for example, splicing can be performed through feature fusion.

[0069] For example, for a first frame image in the image sequence, the encoding features of the image and the initial query features of each object in the image are input into the corresponding decoding module to obtain the decoding features output by the decoding module. The decoding features and the initial query features of each object in the second frame image are spliced to obtain the query features of each object in the second frame image. The first frame image and the second frame image are sorted based on the time sequence of the images in the image sequence, so the first frame image is the previous frame image and the second frame image is the current frame image.

[0070] In this embodiment, the decoding features output by the decoding module corresponding to the previous frame image and the at least one initial query feature are spliced to obtain at least one query feature of the decoding module corresponding to the current frame image. Based on this, the query features of each object in each image in the at least two images except the first frame image can be obtained, and the feature information learned by each query feature of the previous frame image can be transmitted to each query feature of the next frame image to achieve the purpose of information transmission, so that each object in each image can establish correct association and avoid incorrect association between objects. Therefore, when classification is performed based on the decoding features, a gesture category with high accuracy can be obtained, and the accuracy of determining the gesture of each object can be improved.

[0071] In actual applications, the relative positions of the objects in the image sequence may differ in each image, so the position information of each object can be detected and the trajectory of each object can be determined.

[0072] In an embodiment, the number of images is at least two frames, and the decoding features further include position features of the corresponding objects queried from the encoding features by the query features; the method further includes: inputting the decoding features into the position detection module of the gesture recognition model corresponding to the image, determining the position information of each object in the image based on the position features, and outputting the position information through the position detection module; determining the trajectory of each object based on the position information of each object in the at least two images.

[0073] Specifically, the at least two images can be images in the image sequence, and the gesture recognition model can include a position detection module, which can be used to determine the position information of the objects in the decoding features. The position detection module corresponds to each image in the image sequence, which means that the position information of each image is determined by its corresponding position detection module.

[0074] The query features can extract the position information of the corresponding objects from the encoding features to obtain the position features of the corresponding objects. The decoding features corresponding to the image are input into the position detection module of the gesture recognition model corresponding to the image, and the position information of each object in the image can be determined based on the position features, and the position information of each object can be output.

[0075] Exemplarily, the position information of the object can be represented by a position detection box, and the output position information is represented as N x 4, where N represents the Nth image in the image sequence, 4 represents the center point and width and height information (x, y, w, h) of the position detection box, x represents the horizontal coordinate of the center point, y represents the vertical coordinate of the center point, w represents the width value of the center point, and h represents the height value of the center point.

[0076] Exemplarily, the trajectory of each object is determined based on the position information of each object in each image. For example, the position information of the object is analyzed in time sequence according to the time sequence of each image in the image sequence, and the position change process of the object can be obtained, and the trajectory of the object can be determined through the position change process. The trajectory of each object can be determined based on the position information of each object in each image.

[0077] In this embodiment, the decoding feature is input into the position detection module corresponding to the image in the gesture recognition model. The position information of each object in the image can be determined based on the position feature, and the trajectory of each object can be determined based on the position information of each object in each image. Therefore, the trajectory of each object can be determined through the decoding feature, and the tracking detection of each object can be realized.

[0078] In actual application, it is necessary to determine whether the object detected at the position is a valid object through the category of the position where the object is located. When the category is determined incorrectly, the accuracy of the gesture recognition result will be affected. For example, an object in the background category is determined based on each image in the image sequence, and a gesture recognition result is obtained for the object. However, the object in the background is not a valid object, and the recognition of the object is invalid recognition. The recognition of the object will occupy computing power, and invalid results will be obtained, which will reduce the overall recognition efficiency of the recognition process.

[0079] In an embodiment, the decoding feature further includes a category feature of the position where the corresponding object is located, which is queried from the encoding feature through the query feature; and the method further includes: inputting the decoding feature into a category recognition module corresponding to the image in the gesture recognition model, determining the category of the position where each object in the image is located based on the category feature, and outputting the category through the category recognition module, the category including foreground or background.

[0080] Specifically, the gesture recognition model can include a category recognition module corresponding to each image, and the category recognition module can be used to determine the category of the position where the object is located in the decoding feature. The category recognition module corresponds to each image, that is, each image determines the category of the position by the category recognition module corresponding thereto.

[0081] Exemplarily, the category recognition module can be a classifier capable of classifying the category of the position where the object is located. For example, the classifier can include, but is not limited to, a fully connected layer, a convolutional layer, a multi-layer perception, and the like.

[0082] The query feature can extract the category information of the position where the corresponding object is located from the encoded feature to obtain the category feature of the position where the corresponding object is located. When the decoded feature corresponding to the image is input into the category recognition module corresponding to the image in the gesture recognition model, the category of the position where each object in the image is located can be determined based on the category feature of the position where the corresponding object is located, and the category of the position where each object is located can be output. Wherein, the category can include foreground or background, for example, the object at the foreground category position can be set as a valid object, and the object at the background category position can be set as an invalid object.

[0083] Further, based on the gesture category of each object in each image, the gesture of each object can be determined, specifically, based on the category of the position where each object in each image is located, the foreground object can be screened from each image; based on the gesture category of each foreground object in each image, the gesture of each foreground object can be determined.

[0084] Specifically, the foreground object can be understood as a valid object that needs to output the gesture recognition result. After determining the category of the position where each object in the image is located, each object corresponding to the category of the foreground is determined by judging each category, which is the foreground object. After determining that the object is a foreground object, the gesture of each foreground object is determined based on the gesture category of each foreground object.

[0085] Exemplarily, the output category of the position where the object is located can be represented as N x 1, wherein N represents the Nth frame of image in the image sequence, 1 represents the category, for example, 1 can represent the foreground object or the background. Optionally, when the category is a foreground object, the result output is valid; when the category is a background, the output result will be directly filtered out. Optionally, the gesture category output can be represented as N x 1, wherein 1 represents the gesture category, for example, 1 can represent no gesture or one of the plurality of preset gesture categories. Accordingly, when the category of the position where the object corresponding to the gesture category is located is a background category, the output gesture result can be directly filtered out.

[0086] In this embodiment, the decoded features are input into the gesture recognition model and the category recognition module corresponding to the image, the category of the position of each object in the image can be determined based on the category features, and the category can be output through the category recognition module. Further, the foreground objects are determined according to the category, and the gestures of the foreground objects are determined based on the gesture categories of the foreground objects in each image when determining the gestures of each object. Based on this, by determining the category of the position of the object, the effectiveness of the object can be judged, and then the invalid objects can be filtered during gesture recognition, avoiding wasting computing resources and avoiding outputting invalid recognition results, and improving the recognition efficiency of recognition.

[0087] In actual application, the existing gesture recognition method is based on a detection model and a recognition model trained independently to implement two-stage tasks respectively. The detection model and the recognition model are obtained by a non-joint optimization training method. When training the recognition model, the input of the recognition model is a training sample with high accuracy, but when applying the recognition model, the recognition model takes the output of the detection model as the input. At this time, the accuracy of the output of the detection model will reduce the accuracy of the output of the recognition model. Therefore, the inconsistency of the input information of the recognition model will result in a low accuracy of the obtained recognition result.

[0088] Meanwhile, after obtaining the position information of all objects at different time sequences through the detection model, the corresponding objects need to be associated in time sequence to ensure that the same object is contained in the same "spatiotemporal region". Therefore, the detection model needs to set a complex matching strategy for object association. When the number of objects is small, the key strategy is relatively easy to set, but when dealing with a multi-object scene, as the number of objects increases, the complexity of the scene also increases, and a complex and tedious matching strategy is needed to associate each object, which makes the algorithm implementation of the detection model complex and reduces the efficiency of object detection.

[0089] In an embodiment, the gesture recognition model can be obtained after training the initial gesture recognition model. During training, the image features are extracted based on the cross-attention network, and the position information of each object in the image sequence is associated and positioned and the gesture intention is judged by introducing a learnable query vector. Compared with the two-stage model training framework of detecting first and then recognizing, the training method provided in this embodiment can output the predicted gesture categories of all sample objects after inputting the sample image sequence, without post-processing, truly realizing the recognition of a single end-to-end model, so that the gesture recognition model obtained after training can efficiently recognize the gestures of each object in a multi-person scene. The gesture recognition model can be trained based on the following method.

[0090] The sample image sequence is obtained, and at least one frame of sample image based on time sequence acquisition is included in the sample image sequence; the at least one frame of sample image is respectively input into an initial encoding module of the initial gesture recognition model to obtain sample encoding features corresponding to each sample image output by the initial encoding module; each sample image includes at least one sample object; for the sample encoding features corresponding to each sample image, the sample encoding features and at least one sample query feature are input into an initial decoding module corresponding to the sample image in the initial gesture recognition model to obtain sample decoding features output by the initial decoding module; the at least one sample query feature corresponds to the initial decoding module; the sample object and the sample query feature correspond one by one, and the sample decoding feature includes a sample gesture feature of a corresponding sample object queried from the sample encoding feature through the sample query feature; each sample decoding feature is input into an initial recognition module corresponding to the sample image in the initial gesture recognition model, a predicted gesture category of each sample object in the sample image is determined based on the sample gesture feature, and the predicted gesture category is output through the initial recognition module; based on the predicted gesture category and label information, model parameters of the initial gesture recognition model are updated to obtain a gesture recognition model, and the label information is used to represent a gesture category in the sample image.

[0091] Specifically, when obtaining the sample image sequence, the sample image sequence can be collected by a video image collection device such as an RGB camera, an infrared camera, a depth camera, and the like. The sampling mode includes but is not limited to dense sampling, equal interval sampling, and sampling the image sequence in chronological order to obtain N frames of sample images and corresponding label information. For example, a data reader takes the N frames of sample images and corresponding label information obtained by sampling as a group of samples, and inputs the sample image sequence into the initial gesture recognition model for training.

[0092] The multiple frames of sample images are respectively input into the initial encoding module of the initial gesture recognition model, and sample encoding features corresponding to each sample image output by the initial encoding module can be obtained. Each sample image includes at least one sample object, for the sample encoding features corresponding to each sample image, the sample encoding features and at least one sample query feature are input into an initial decoding module corresponding to the sample image in the initial gesture recognition model to obtain sample decoding features output by the initial decoding module; the at least one sample query feature corresponds to the initial decoding module; the sample object and the sample query feature correspond one by one, and the sample decoding feature includes a sample gesture feature of a corresponding sample object queried from the sample encoding feature through the sample query feature. Each sample decoding feature is input into an initial recognition module corresponding to the sample image in the initial gesture recognition model, a predicted gesture category of each sample object in the sample image is determined based on the sample gesture feature, and the predicted gesture category is output through the initial recognition module.

[0093] Further, based on the predicted gesture category and the label information, the sample query feature is iteratively updated, and the finally obtained sample query feature is determined as the query feature.

[0094] Specifically, based on the predicted gesture category and the label information representing the gesture category in the sample image, the loss of the initial recognition module is determined, the model parameters are adjusted based on the value of the loss, and the model parameters of the initial gesture recognition model are updated, that is, one training process is completed. Through multiple training iterations, the model parameters are iteratively updated and continuously optimized. When the loss convergence effect of the model meets the application requirements, the gesture recognition model can be obtained. The finally obtained sample query feature after training can be determined as the query feature.

[0095] In the embodiment, a training method of an end-to-end gesture recognition model is provided. Based on the initial gesture recognition model, the gesture recognition model can be obtained, and then the gestures of the objects can be recognized through the gesture recognition model. When the model is applied, the image sequence is input, and the spatial position information and the gesture category information of all objects in the image sequence can be directly output after inference of each module. The whole process does not need other intermediate steps and post-processing. On the one hand, the training method improves the recognition effect of the model, and on the other hand, it significantly improves the recognition efficiency of gesture recognition in a multi-person scene, and further improves the practicability of gesture interaction in human-computer interaction.

[0096] The following is an example of inputting an image sequence composed of images collected from t time to t+N time into the gesture recognition model, and the modules are described in detail. Figure 2 is a schematic block diagram of the gesture recognition model provided by the embodiment of the present application, as shown in Figure 2 The gesture recognition model includes an encoding module, a decoding module, a position detection module, a category recognition module and a recognition module. A set of learnable query vectors are also provided for human body detection frame time sequence feature association and corresponding gesture category classification.

[0097] In the present scheme, at least one frame of image is input frame by frame according to the time sequence dimension. The dimension of the image is CxHxW, wherein C represents the channel number of the image, for example, C=3, that is, R, G and B three channels are included; H and W represent the height and width of the image respectively.

[0098] Exemplarily, the gesture recognition model further comprises a backbone feature extraction module. The backbone feature extraction module can be used for downsampling and spatial feature information extraction of input image features. The backbone feature extraction module extracts features of each image, and inputs the extracted image features into an encoding module to obtain corresponding encoding features of each object. The backbone feature extraction module can be a backbone feature extractor, including but not limited to a 2D convolutional neural network (2D-CNN), a 3D convolutional neural network (3D-CNN), a convolutional neural network and a long short-term memory network (CNN-LSMT), a temporal and spatial feature extraction network such as a temporal shift module network (TSM), and the like. The 2D-CNN can be, for example, a residual neural network (ResNet), a MobileNet, a ShuffleNet, or the like.

[0099] Exemplarily, the encoding module can further encode the features extracted by the backbone feature extraction module. The encoding module can include but is not limited to an attention network such as a Transformer and its variants. Meanwhile, in order to model the temporal information at different time points, the input of the encoding module further comprises a position encoder, and the position encoding method adopted by the position encoder includes but is not limited to fixed absolute position encoding, learnable absolute position encoding, relative position encoding, and the like. For example, the position encoding can be performed according to the index of the image in the image sequence, and the corresponding feature information is spliced as the input of the encoding module, and the corresponding encoding features are output.

[0100] Exemplarily, the decoding module can be a module for decoding the encoding features output by the encoding module, and outputting decoding features for object position positioning, detection, class recognition, and gesture recognition. The decoding module can include but is not limited to a cross-attention network such as a Transformer and its variants. The input of the decoding module includes, in addition to the encoding features corresponding to the image, a query feature corresponding to each object in the image, which is used for association of different object spatio-temporal relationships and extraction of gesture information. The setting mode of the query feature includes but is not limited to a fixed parameter query vector or a learnable query vector.

[0101] The query vector can be a learnable vector, and its initialization methods include but are not limited to random initialization, zero-mean initialization, and an explicit initialization strategy combined with object position information label guiding, etc. For example, the query vector can be set to random initialization. The encoded features, query features and position codes are input into the decoding module together, and the attention mechanism network is used for feature extraction and enhancement to obtain decoding features for position detection, category recognition and gesture category recognition.

[0102] For example, the position detection module, the category recognition module and the recognition module can all detect the spatial information position of the object appearing in the image, locate and recognize the category, and classify the gesture category of the object. Each module can include but is not limited to a classifier such as a fully connected layer, a convolutional layer and a multilayer perceptron. It should be noted that the spatial position, category and gesture category of each output object are one-to-one corresponding to the query features. Since the spatial position, category and gesture category of each output object are one-to-one corresponding to the query features, the gesture recognition model does not need other intermediate processing in the inference process, and can realize end-to-end multi-person gesture recognition of single image input and multiple result output.

[0103] The gesture recognition model provided by the embodiment of the present application unifies the detection model and the recognition model by using one model, so that it can realize joint optimization and solve the inconsistency problem of training and testing. Secondly, by introducing the query feature, the model can adaptively match and associate the objects in time sequence through learning, avoiding complex matching rules. At the same time, through the inference mode of single image input and multi-object gesture recognition output, the gesture recognition results of each object in the image sequence are output, solving the problem of low recognition efficiency in a multi-person scene and improving the recognition efficiency and the applicability of the model.

[0104] The gesture recognition device provided by the embodiment of the present application is described below. The gesture recognition device described below can be correspondingly referred to the gesture recognition method described above.

[0105] Figure 3 FIG. 1 is a structural schematic diagram of the gesture recognition device provided by the embodiment of the present application. As shown in FIG. 1, the gesture recognition device 300 includes: Figure 3

[0106] The encoding unit 310 is configured to input at least one image into the encoding module of the gesture recognition model to obtain the encoded features corresponding to each image output by the encoding module; and each image includes at least one object.

[0107] ​The decoding unit 320 is configured to input the encoded feature and at least one query feature into a decoding module corresponding to the image in the gesture recognition model for each image, to obtain decoded features output by the decoding module, the at least one query feature corresponds to the decoding module, the query feature and the object are in one-to-one correspondence, and the decoded feature includes gesture features of the corresponding object queried from the encoded feature by the query feature.

[0108] The recognition unit 330 is configured to input the decoded feature into a recognition module corresponding to the image in the gesture recognition model, to determine gesture categories of each object in the image based on the gesture feature, and to output the gesture categories by the recognition module.

[0109] The determination unit 340 is configured to determine the gesture of each object based on the gesture categories of each object in each image.

[0110] In an example embodiment, the number of images is at least two frames, and the gesture recognition model includes at least two decoding modules; the gesture recognition device 300 further includes a splicing unit configured to splice the decoded features output by the decoding module corresponding to the previous frame of image and at least one initial query feature, to obtain at least one query feature of the decoding module corresponding to the current frame of image.

[0111] In an example embodiment, the number of images is at least two frames, and the decoded feature further includes position features of the corresponding object queried from the encoded feature by the query feature; the gesture recognition device 300 further includes a position detection unit configured to input the decoded feature into a position detection module corresponding to the image in the gesture recognition model, to determine position information of each object in the image based on the position feature, and to output the position information by the position detection module; and to determine the trajectory of each object based on the position information of each object in the at least two frames of images.

[0112] In an example embodiment, the decoded feature further includes category features of the position where the corresponding object is located, which are queried from the encoded feature by the query feature; the gesture recognition device 300 further includes a position category recognition unit configured to input the decoded feature into a category recognition module corresponding to the image in the gesture recognition model, to determine the category of the position where each object is located in the image based on the category feature, and to output the category by the category recognition module, the category including foreground or background.

[0113] In an example embodiment, the determination unit 340 is specifically configured to: filter foreground objects from each image based on the category of the position where each object is located in each image; and determine the gesture of each foreground object based on the gesture category of each foreground object in each image.

[0114] In an example embodiment, the gesture recognition model is trained based on the following manner: a sample image sequence is obtained, the sample image sequence includes at least one frame of sample image collected based on time sequence; the at least one frame of sample image is input into an initial encoding module of an initial gesture recognition model respectively, to obtain sample encoding features corresponding to each sample image output by the initial encoding module; each sample image includes at least one sample object; for the sample encoding features corresponding to each sample image, the sample encoding features and at least one sample query feature are input into an initial decoding module corresponding to the sample image in the initial gesture recognition model, to obtain sample decoding features output by the initial decoding module; the at least one sample query feature corresponds to the initial decoding module; the sample object and the sample query feature are one-to-one corresponding, and the sample decoding features include sample gesture features of the corresponding sample object queried from the sample encoding features through the sample query feature; each sample decoding feature is input into an initial recognition module corresponding to the sample image in the initial gesture recognition model, to determine a predicted gesture category of each sample object in the sample image based on the sample gesture features, and output the predicted gesture category through the initial recognition module; based on the predicted gesture category and label information, model parameters of the initial gesture recognition model are updated to obtain the gesture recognition model, and the label information is used to represent the gesture category in the sample image.

[0115] In an example embodiment, the gesture recognition device 300 further includes an iterative updating unit configured to iteratively update the sample query feature based on the predicted gesture category and the label information, and determine the finally obtained sample query feature as the query feature.

[0116] The device of the embodiment can be used to execute the method of any one of the gesture recognition method-side embodiments, and the specific implementation process and technical effects are similar to those of the gesture recognition method-side embodiments. For details, refer to the detailed description in the gesture recognition method-side embodiments, which will not be described here.

[0117] Figure 4 is a structural schematic diagram of an electronic device provided by the embodiment of the present application, like Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke a logic instruction in the memory 430 to execute a gesture recognition method, which includes: inputting at least one frame of image into an encoding module of a gesture recognition model to obtain an encoding feature corresponding to each image output by the encoding module; each image includes at least one object; for the encoding feature corresponding to each image, inputting the encoding feature and at least one query feature into a decoding module corresponding to the image in the gesture recognition model to obtain a decoding feature output by the decoding module; the at least one query feature corresponds to the decoding module; the object and the query feature are one-to-one corresponding, and the decoding feature includes a gesture feature of the corresponding object queried from the encoding feature through the query feature; inputting the decoding feature into a recognition module corresponding to the image in the gesture recognition model, determining a gesture category of each object in the image based on the gesture feature, and outputting the gesture category through the recognition module; and determining a gesture of each object based on the gesture category of each object in each image.

[0118] In addition, the logic instruction in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0119] In another aspect, the embodiment of the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the gesture recognition method provided by the above method. The method comprises: inputting at least one frame of image into an encoding module of a gesture recognition model to obtain an encoding feature corresponding to each image output by the encoding module; each image comprises at least one object; for the encoding feature corresponding to each image, inputting the encoding feature and at least one query feature into a decoding module corresponding to the image in the gesture recognition model to obtain a decoding feature output by the decoding module; the at least one query feature corresponds to the decoding module; the object and the query feature correspond to each other, and the decoding feature comprises a gesture feature of the corresponding object queried from the encoding feature by the query feature; inputting the decoding feature into a recognition module corresponding to the image in the gesture recognition model, determining a gesture category of each object in the image based on the gesture feature, and outputting the gesture category through the recognition module; and determining a gesture of each object based on the gesture category of each object in each image.

[0120] In another aspect, the embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program can be stored on a non-transitory computer readable storage medium. When the computer program is executed by a processor, the computer can execute the gesture recognition method provided by the above method. The method comprises: inputting at least one frame of image into an encoding module of a gesture recognition model to obtain an encoding feature corresponding to each image output by the encoding module; each image comprises at least one object; for the encoding feature corresponding to each image, inputting the encoding feature and at least one query feature into a decoding module corresponding to the image in the gesture recognition model to obtain a decoding feature output by the decoding module; the at least one query feature corresponds to the decoding module; the object and the query feature correspond to each other, and the decoding feature comprises a gesture feature of the corresponding object queried from the encoding feature by the query feature; inputting the decoding feature into a recognition module corresponding to the image in the gesture recognition model, determining a gesture category of each object in the image based on the gesture feature, and outputting the gesture category through the recognition module; and determining a gesture of each object based on the gesture category of each object in each image.

[0121] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0122] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A gesture recognition method, characterized in that, include: At least one frame of image is input into the encoding module of the gesture recognition model to obtain the encoding features corresponding to each image output by the encoding module; each image includes at least one object; For each image, the encoded features and at least one query feature are input into the decoding module corresponding to the image in the gesture recognition model to obtain the decoded features output by the decoding module. The at least one query feature corresponds to the decoding module; the object and the query feature correspond one-to-one, and the decoding feature includes the gesture feature of the corresponding object queried from the encoding feature through the query feature; The decoded features are input into the recognition module corresponding to the image in the gesture recognition model. Based on the gesture features, the gesture category of each object in the image is determined, and the gesture category is output through the recognition module. Based on the gesture category of each object in each image, determine the gesture of each object; The number of images is at least two frames, and the gesture recognition model includes at least two decoding modules; the method further includes: The decoding features output by the decoding module corresponding to the previous frame image are concatenated with at least one initial query feature to obtain at least one query feature of the decoding module corresponding to the current frame image.

2. The gesture recognition method according to claim 1, characterized in that, The number of images is at least two frames, and the decoding feature also includes the location feature of the corresponding object queried from the encoding feature through the query feature; The method further includes: The decoded features are input into the position detection module corresponding to the image in the gesture recognition model. The position information of each object in the image is determined based on the position features, and the position information is output through the position detection module. The trajectory of each object is determined based on the positional information of each object in at least two frames of images.

3. The gesture recognition method according to claim 1, characterized in that, The decoding feature also includes a category feature indicating the location of the corresponding object, which is retrieved from the encoding feature using the query feature; The method further includes: The decoded features are input into the category recognition module corresponding to the image in the gesture recognition model. Based on the category features, the category of each object in the image is determined, and the category is output through the category recognition module. The category includes foreground or background.

4. The gesture recognition method according to claim 3, characterized in that, Determining the gestures of each object based on the gesture category of each object in each image includes: Based on the category of the location of each object in each image, foreground objects are selected from each of the images; The gesture of each foreground object is determined based on the gesture category of each foreground object in each of the images.

5. The gesture recognition method according to any one of claims 1-4, characterized in that, The gesture recognition model was trained in the following manner: Acquire a sample image sequence, wherein the sample image sequence includes at least one sample image acquired based on time sequence; At least one sample image is input into the initial encoding module of the initial gesture recognition model to obtain the sample encoding features corresponding to each sample image output by the initial encoding module. Each of the sample images includes at least one sample object; For each sample image, the sample encoding features and at least one sample query feature are input into the initial decoding module corresponding to the sample image in the initial gesture recognition model to obtain the sample decoding features output by the initial decoding module. The at least one sample query feature corresponds to the initial decoding module; the sample object and the sample query feature correspond one-to-one, and the sample decoding feature includes the sample gesture feature of the corresponding sample object queried from the sample encoding feature through the sample query feature; The sample decoding features are input into the initial recognition module corresponding to the sample image in the initial gesture recognition model. Based on the sample gesture features, the predicted gesture category of each sample object in the sample image is determined, and the predicted gesture category is output through the initial recognition module. Based on the predicted gesture category and label information, the model parameters of the initial gesture recognition model are updated to obtain the gesture recognition model. The label information is used to characterize the gesture category in the sample image.

6. The gesture recognition method according to claim 5, characterized in that, The method further includes: Based on the predicted gesture category and the label information, the sample query features are iteratively updated, and the final sample query features are determined as the query features.

7. A gesture recognition device, characterized in that, include: An encoding unit is used to input at least one frame of image into the encoding module of the gesture recognition model to obtain the encoding features corresponding to each image output by the encoding module; each image includes at least one object; The decoding unit is used to input the encoded features and at least one query feature into the decoding module corresponding to the image in the gesture recognition model for each image, so as to obtain the decoded features output by the decoding module. The at least one query feature corresponds to the decoding module; the object and the query feature correspond one-to-one, and the decoding feature includes the gesture feature of the corresponding object queried from the encoding feature through the query feature; The recognition unit is used to input the decoded features into the recognition module corresponding to the image in the gesture recognition model, determine the gesture category of each object in the image based on the gesture features, and output the gesture category through the recognition module. The determining unit is used to determine the gesture of each object based on the gesture category of each object in each image; The number of images is at least two frames, and the gesture recognition model includes at least two decoding modules; the gesture recognition device also includes a splicing unit, which is used to splice the decoding features output by the decoding module corresponding to the previous frame image and at least one initial query feature to obtain at least one query feature of the decoding module corresponding to the current frame image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the gesture recognition method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the gesture recognition method as described in any one of claims 1 to 6.

Images