Gesture recognition method, gesture interaction method, and related devices, apparatuses, and media

Abstract

The application discloses a gesture recognition method, a gesture interaction method, and related devices, equipment and media, wherein the gesture recognition method comprises: extracting target image features of a target image in a shooting video, and extracting reference image features of a reference image before the target image in the shooting video; performing prediction based on the target image features and the reference image features to obtain predicted images of a plurality of preset categories; and performing analysis based on each predicted image to obtain a gesture recognition result of the target image; wherein the gesture recognition result at least includes: whether a gesture exists in the target image, and in the case that a gesture exists in the target image, a gesture category of each gesture. The above scheme can ensure that it is applicable to multi-person gesture interaction, further improve the accuracy of gesture recognition, as far as possible, reduce the calculation load of gesture recognition, and improve the applicability in a long-distance use scenario.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a gesture recognition method, a gesture interaction method, and related devices, equipment and media. BACKGROUND

[0002] Gesture interaction is an efficient and convenient interaction method, which has been gradually applied to smart home, intelligent driving and other scenarios. As a front-end technology of gesture interaction, the accuracy of gesture recognition is one of the necessary prerequisites to ensure the effect of gesture interaction.

[0003] At present, existing gesture recognition either relies on sensor devices such as photosensitive elements and depth sensors, which is easy to cause misrecognition (such as misrecognition caused by walking, light and other factors), or recognizes gestures through hand key point detection, and once the distance between the person and the camera is far, the accuracy of hand key point detection will drop straight, resulting in a large error in gesture recognition, or detects the whole person first and then recognizes gestures for each person individually, and the calculation load increases linearly with the increase in the number of people, so it is usually only suitable for gesture interaction with a small number of people. Therefore, how to further improve the accuracy of gesture recognition while ensuring its applicability to multi-person gesture interaction, and as far as possible to reduce the calculation load of gesture recognition, and to improve the applicability in long-distance use scenarios has become a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is to provide a gesture recognition method, a gesture interaction method, and related devices, equipment and media, which can further improve the accuracy of gesture recognition while ensuring its applicability to multi-person gesture interaction, and as far as possible to reduce the calculation load of gesture recognition, and to improve the applicability in long-distance use scenarios.

[0005] To solve the above technical problem, the first aspect of the present application provides a gesture recognition method, comprising: extracting a target image feature of a target image in a photographed video, and extracting a reference image feature of a reference image located before the target image in the photographed video; performing prediction based on the target image feature and the reference image feature to obtain a prediction image of a plurality of preset categories; wherein the plurality of preset categories include no gesture and at least one gesture category, and a pixel value of a first pixel point in the prediction image represents a possibility that a second pixel point corresponding to the first pixel point in the target image belongs to a target category, and the target category is a preset category to which the prediction image belongs; analyzing based on each prediction image to obtain a gesture recognition result of the target image; wherein the gesture recognition result at least includes: whether there is a gesture in the target image, and in the case that there is a gesture in the target image, the gesture category of each gesture.

[0006] To solve the above technical problems, the second aspect of the present application provides a gesture interaction method, comprising: obtaining a gesture recognition result of a target image in a shooting video; wherein the gesture recognition result is obtained based on the gesture recognition method in the first aspect, and the gesture recognition result comprises: a gesture category of each gesture in the target image; and responding to each gesture respectively based on the gesture category of each gesture.

[0007] To solve the above technical problems, the third aspect of the present application provides a gesture recognition device, comprising: an extraction module, a prediction module and an analysis module, the extraction module is configured to extract a target image feature of a target image in a shooting video, and extract a reference image feature of a reference image located before the target image in the shooting video; the prediction module is configured to perform prediction based on the target image feature and the reference image feature to obtain a prediction image of a plurality of preset categories; wherein the plurality of preset categories comprise no gesture and at least one gesture category, a pixel value of a first pixel point in the prediction image represents a possibility that a second pixel point corresponding to the first pixel point in the target image belongs to a target category, and the target category is a preset category to which the prediction image belongs; and the analysis module is configured to analyze based on each prediction image to obtain a gesture recognition result of the target image; wherein the gesture recognition result at least comprises: whether there is a gesture in the target image, and in the case that there is a gesture in the target image, a gesture category of each gesture.

[0008] To solve the above technical problems, the fourth aspect of the present application provides a gesture interaction device, comprising: a result obtaining module and a gesture response module, the result obtaining module is configured to obtain a gesture recognition result of a target image in a shooting video; wherein the gesture recognition result is obtained based on the gesture recognition device in the third aspect, and the gesture recognition result comprises: a gesture category of each gesture in the target image; and the gesture response module is configured to respond to each gesture respectively based on the gesture category of each gesture.

[0009] To solve the above technical problems, the fifth aspect of the present application provides an electronic device, comprising a camera, a memory and a processor, the camera and the memory are respectively coupled to the processor, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the gesture recognition method in the first aspect, or implement the gesture interaction method in the second aspect.

[0010] To solve the above technical problems, the sixth aspect of the present application provides a computer readable storage medium, which stores program instructions capable of being executed by a processor, and the program instructions are used to implement the gesture recognition method in the first aspect, or implement the gesture interaction method in the second aspect.

[0011] The scheme extracts target image features of a target image in a photographed video, extracts reference image features of a reference image before the target image in the photographed video, and performs prediction based on the target image features and the reference image features to obtain predicted images of a plurality of preset categories, wherein the plurality of preset categories include a no-gesture category and at least one gesture category, and a pixel value of a first pixel point in the predicted image represents a possibility that a second pixel point corresponding to the first pixel point in the target image belongs to a target category, and the target category is a preset category to which the predicted image belongs. On this basis, gesture recognition results of the target image are obtained based on the predicted images, and the gesture recognition results at least include whether there is a gesture in the target image, and in the case where there is a gesture in the target image, a gesture category of each gesture. On the one hand, since the possibility that the target image pixel point respectively belongs to the plurality of preset categories is directly predicted, i.e., there is no need to divide it into two-stage tasks of person detection and gesture recognition, the influence of the number of persons on the calculation load can be reduced as much as possible, and the applicability to multi-person gesture interaction can be improved. On the other hand, since the gesture recognition process does not depend on a sensor, the influence of factors such as walking and light on gesture recognition can be reduced as much as possible, and the accuracy of gesture recognition can be improved. On the other hand, since the possibility that the target image pixel point respectively belongs to the plurality of preset categories is directly predicted in the gesture recognition process, and does not depend on hand key point detection, even if the person is far away from the camera, the accuracy of gesture recognition can be ensured as much as possible. Therefore, the accuracy of gesture recognition can be further improved while ensuring applicability to multi-person gesture interaction, and the calculation load of gesture recognition can be reduced as much as possible, and the applicability in a long-distance use scenario can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a flowchart of an embodiment of the gesture recognition method of the present application;

[0013] Figure 2 is a process diagram of an embodiment of the gesture recognition method of the present application;

[0014] Figure 3 is a process diagram of an embodiment of post-processing;

[0015] Figure 4 is a flowchart of an embodiment of the gesture interaction method of the present application;

[0016] Figure 5 is a diagram of an embodiment of back projection;

[0017] Figure 6 is a process diagram of an embodiment of the gesture interaction method of the present application;

[0018] Figure 7 is a framework diagram of an embodiment of the gesture recognition device of the present application;

[0019] Figure 8 is a frame diagram of an embodiment of the gesture interaction apparatus of the present application;

[0020] Figure 9 is a frame diagram of an embodiment of the electronic device of the present application;

[0021] Figure 10 is a frame diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0022] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0023] In the following description, specific details are set forth in order to provide a thorough understanding of the present application. However, persons having ordinary skill in the art will appreciate that the present application can be practiced without the specific details.

[0024] The terms "system" and "network" are often used interchangeably herein. The term "and / or", merely describes an associated relationship, which means that there can be three relationships, for example, A and / or B, which means that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" herein means two or more than two.

[0025] Please refer to Figure 1 , Figure 1 is a flow diagram of an embodiment of the gesture recognition method of the present application. Specifically, it can include the following steps:

[0026] Step S11: extracting target image features of target images in the photographed video, and extracting reference image features of reference images before the target images in the photographed video.

[0027] In one implementation scenario, the photographed video can be obtained by controlling the camera in response to a gesture interaction request. The gesture interaction request can include but is not limited to being triggered by the following ways: key, voice, etc., which are not limited herein. Exemplarily, taking a conference scenario as an example, the conference place can be arranged with a camera to realize gesture interaction through the camera, and mobile terminals such as smart phones, tablet computers, etc. can control the camera, such as the mobile terminal can run an application, and the application can be provided with an opening button and a closing button of gesture interaction, then the mobile terminal can send a command to start gesture interaction to the camera in response to the opening button being triggered, and the camera can start recording the photographed video in response to the command. Other scenarios can be similarly deduced, which will not be exemplified one by one herein.

[0028] In one implementation scenario, the target image can be any video frame image in the captured video, which is not limited herein. For example, the target image can be the first video frame image in the captured video, or can also be the second video frame image in the captured video. For ease of description, the target image can be the i-th video frame image in the captured video, and i is an integer greater than 0. Of course, considering that in the process of gesture interaction, the first N frames can still be in the gesture preparation stage, i.e., the first N frames have not yet truly started to perform the gesture, for example, for the gesture "wave", to implement the gesture, it is usually necessary to include two stages of lifting the hand and waving the hand, and therefore, to avoid misrecognition, the target image can also be set as any video frame image in the captured video starting from the N+1-th frame. It should be noted that the specific value of N can be set in combination with the frame rate of the camera, such as can be set as a preset multiple of the frame rate. Specifically, considering that the gesture usually needs a preset time length for preparation, the preset time length can be taken as the preset multiple. Exemplarily, it is statistically known that the gesture preparation usually needs 1 second, and the frame rate of the camera is 15 fps (frame per second), and therefore, N can be set as 15. Other cases can be deduced in a similar manner, which will not be exemplified one by one herein.

[0029] In another implementation scenario, each video frame image in the captured video can also be selected in turn as the target image, and the steps in the embodiments of the present disclosure can be performed to perform gesture recognition on each video frame image in the captured video to obtain gesture recognition results of each video frame image in the captured video. Of course, as described above, considering that the gesture usually needs a preset time length for preparation, each video frame image in the captured video starting from the N+1-th frame can be selected in turn as the target image, and the steps in the embodiments of the present disclosure can be performed to perform gesture recognition on each video frame image in the captured video starting from the N+1-th frame to obtain gesture recognition results of each video frame image in the captured video starting from the N+1-th frame. It should be noted that the specific value of N can be referred to the foregoing related description, which will not be repeated herein.

[0030] In one implementation scenario, taking the i-th (i is an integer greater than 0) video frame image in the captured video as the target image as an example, the reference image can include the first video frame image to the i-1-th video frame image in the captured video. Of course, as described above, considering that the gesture usually needs a preset time length for preparation, taking the i-th (i is not less than N+1) video frame image in the captured video as the target image as an example, the reference video can include the N+1-th video frame image to the i-1-th video frame image in the captured video.

[0031] In an implementation scenario, in order to improve the efficiency of gesture recognition, a gesture recognition model can be pre-trained, and the gesture recognition model can include a feature extraction network, which can include but is not limited to a convolutional neural network, etc., and the network structure of the feature extraction network is not limited herein. In addition, the training process of the gesture recognition model can refer to the related description below, which is not described herein. On this basis, the feature extraction network in the gesture recognition model can be used to extract features of a target image to obtain target image features, and the feature extraction network in the gesture recognition model can be used to extract features of each reference image to obtain reference image features of each reference image. In addition, the target image features can include a plurality of sub-features arranged in a channel order, and the reference image features include a plurality of sub-features arranged in a channel order. It should be noted that the target image features and the reference image features can have the same number of channels, and for ease of description, the number of channels can be denoted as C1. In addition, the resolution of the sub-features can be denoted as W1*H1, and the size of the target image features can be represented as W1*H1*C1, and the size of the reference image features can also be denoted as W1*H1*C1.

[0032] Step S12: predicting based on the target image features and the reference image features to obtain a plurality of prediction images of preset categories.

[0033] In the embodiments of the present disclosure, the plurality of preset categories include no gesture and at least one gesture category, and the pixel value of the first pixel point in the prediction image represents the possibility that the second pixel point corresponding to the position of the first pixel point in the target image belongs to the target category, and the target category is the preset category to which the prediction image belongs. It should be noted that the above-mentioned "possibility" can be represented by a probability value, and the higher the probability value, the greater the possibility that the second pixel point corresponding to the position of the first pixel point in the target image belongs to the target category, and vice versa. The lower the probability value, the smaller the possibility that the second pixel point corresponding to the position of the first pixel point in the target image belongs to the target category. In addition, the resolution of each prediction image is the same as that of the target image, that is, for a target image with a resolution of W*H, a prediction image with a size of W*H*O can be predicted, where W*H is the resolution of each prediction image, and O is the number of prediction images, that is, the total number of preset categories, which includes "no gesture" and at least one gesture category.

[0034] In an embodiment scenario, if the second pixel point belongs to the preset category "no gesture", it means that the second pixel point is irrelevant to any gesture, and if the second pixel point belongs to the preset category "gesture category", it means that the second pixel point is relevant to the gesture category. In addition, the at least one gesture category can include but is not limited to waving, liking, heart, praying, number, OK, and fist, without limitation. For example, the target image can include a person and a background, and for the second pixel point located on the person and outside the hand, or the second pixel point located in the background, since both of the two types of second pixel points are irrelevant to any gesture, the possibility of the two types of second pixel points belonging to the preset category "no gesture" should be as high as possible, and the possibility of the two types of second pixel points belonging to the preset category "gesture category" should be as low as possible. In addition, for the second pixel point located in the hand, if the hand presents a certain gesture category, the possibility of the second pixel point belonging to the gesture category should be as high as possible, and the possibility of the second pixel point belonging to other gesture categories or no gesture should be as low as possible, and vice versa. If the hand does not present any gesture category, the possibility of the second pixel point belonging to the preset category "gesture category" should be as low as possible, and the possibility of the second pixel point belonging to the preset category "no gesture" should be as high as possible. Other cases can be similarly deduced, which will not be repeated here.

[0035] In an embodiment scenario, please refer to Figure 2 , Figure 2 is a process schematic diagram of an embodiment of the gesture recognition method of the present application. As described above, in order to improve the efficiency of gesture recognition, the gesture recognition model can be trained in advance, and on this basis, the video can be recognized based on the gesture recognition model, and the prediction image corresponding to the target image in the shooting video can be obtained. As shown in Figure 2 , the prediction image of the plurality of preset categories can include but is not limited to the prediction image of the preset category "OK", the prediction image of the preset category "fist", and the prediction image of the preset category "no gesture", without limitation. In addition, as shown in Figure 2 , the first pixel point highlighted in each prediction image of each preset category indicates that the second pixel point corresponding to the position of the first pixel point in the target image belongs to the preset category. For example, the more obvious the first pixel point is highlighted, the higher the possibility that the second pixel point corresponding to the position of the first pixel point in the target image belongs to the preset category, and vice versa. The less obvious the first pixel point is highlighted, the lower the possibility that the second pixel point corresponding to the position of the first pixel point in the target image belongs to the preset category.

[0036] In one implementation scenario, as described above, the target image feature includes a plurality of sub-features arranged in a channel order, and the reference image feature includes a plurality of sub-features arranged in a channel order. In order to construct time sequence feature information in the gesture recognition process to improve the accuracy of gesture recognition, the sub-feature at the target sequence position in the target image feature can be replaced by the sub-feature at the target sequence position in the first image feature to update the target image feature, and the first image feature is the reference image feature of the reference image adjacent to the target image. On this basis, the prediction image of a plurality of preset categories can be obtained based on the latest updated target image feature. The above-mentioned manner replaces the sub-feature at the target sequence position in the target image feature with the sub-feature at the target sequence position in the first image feature, thereby realizing the shift in the time sequence dimension of the target image feature, and then the time sequence feature information can be constructed in the updated target image feature, and on this basis, the prediction can be combined with the image feature information and the time sequence feature information, which helps to improve the accuracy of gesture recognition.

[0037] In a specific implementation scenario, the target sequence position can be set according to actual application needs. For example, in the case of needing to construct as rich time sequence feature information as possible, the target sequence position can be as much as possible, such as the target sequence position can be set to the first fifty percent; or in the case of needing to appropriately reduce the calculation load of constructing time sequence feature information, the target sequence position can be as little as possible, such as the target sequence position can be set to the first ten percent; or if a balance needs to be achieved between the richness and the calculation load of constructing time sequence feature information, the target sequence position can be moderate, such as the target sequence position can be set to the first twenty-five percent. Other cases can be similarly extended, which will not be repeated here. Of course, the target sequence position can not be limited to being set to the first X percent, but can also be set to any channel or a combination of discontinuous channels, which is not limited here.

[0038] In a specific implementation scenario, please continue to refer to the time sequence shift module shown in Figure 2 As shown in Figure 2 The time sequence shift module is used to realize the operation of replacing the sub-feature. Each row represents the image feature of different images in the video, such as the first row can represent the image feature of the image at t1 time in the video, the second row can represent the image feature of the image at t2 time in the video, and so on. In addition, each column in the image feature corresponding to each row can represent a sub-feature arranged in a channel order. It should be noted that Figure 2For the convenience of description, the image features are represented in two-dimensional form, and in fact, as mentioned above, the image features can be represented in three-dimensional form of length * width * channel number. Further, for the image features at t2, the first two sub-features are replaced by the first two sub-features in the image features at t1; similarly, for the image features at t3, the first two sub-features are replaced by the first two sub-features in the image features at t2; similarly, for the image features at t4, the first two sub-features are replaced by the first two sub-features in the image features at t3. Other cases can be similarly deduced, and here, no further examples are given.

[0039] In a specific implementation scenario, as mentioned above, in order to improve the efficiency of gesture recognition, the gesture recognition model can be pre-trained, and the gesture recognition model can include a prediction network, which can include but is not limited to: a fully connected layer, etc., and the network structure of the prediction network is not limited here. On this basis, the latest updated target image features can be input into the prediction network to obtain the predicted images of a plurality of preset categories.

[0040] In another implementation scenario, different from the foregoing manner, in order to further improve the depth of the timing feature information, before predicting based on the latest updated target image feature to obtain a prediction image of a preset category, it can be further judged whether the total number of times of feature replacement performed by the target image feature is less than a preset value, so that in response to the total number of times being less than the preset value, each latest acquired reference image feature can be taken as a second image feature, and a sub-feature located at the target sequence position in the second image feature can be replaced by a sub-feature located at the target sequence position in a third image feature to update the second image feature, and the third image feature is a latest acquired reference image feature of a previous reference image of the reference image to which the second image feature belongs. Then, the feature extraction can be performed based on the latest updated target image feature to update the target image feature, and the feature extraction can be performed based on the latest updated reference image feature to update the reference image feature, and the steps of replacing the sub-feature located at the target sequence position in the target image feature by the sub-feature located at the target sequence position in the first image feature to update the target image feature and the subsequent steps can be re-executed. In the above manner, when the total number of times of feature replacement performed by the target image feature is less than the preset value, the timing feature information is constructed for each reference image feature with reference to the target image feature, so that each reference image feature contains part of the feature information of the previous reference image, and on this basis, the deeper image features of the latest updated target image feature and the reference image feature are extracted, and when the total number of times of feature replacement performed by the target image feature is still less than the preset value, the timing feature information is constructed for the latest updated target image feature and the latest updated reference image feature, respectively, so that for each image, as many historical images as possible can be referred to for constructing the timing feature information, which helps to improve the accuracy of subsequent gesture recognition.

[0041] In a specific implementation scenario, the target image feature performs feature replacement, specifically, the aforementioned "replacing the sub-feature located at the target sequence position in the target image feature by the sub-feature located at the target sequence position in the first image feature". It should be noted that even if the target sequence position covers multiple channels, the sub-feature replacement of the multiple channels of the target image feature in one replacement process is still counted as one feature replacement operation. In addition, the preset value can be set according to actual needs. For example, in the case of needing to construct as rich timing feature information as possible, the preset value can be set to be appropriately large, such as 20, or in the case of needing to appropriately reduce the calculation load of constructing the timing feature information, the preset value can be set to be appropriately small, such as the preset value can be set to 5; or if a balance between the richness and the calculation load of constructing the timing feature information is needed, the preset value can be set to be moderate, such as 15. Other cases can be similarly deduced, which will not be listed one by one here.

[0042] In a specific implementation scenario, the video is captured by the camera, and considering that the effective occurrence duration of the gesture is relatively stable, any one of the average value, the maximum value, the median value, etc. of the effective occurrence duration of various gestures can be counted as the reference duration, and then the reference duration and the frame rate of the camera are combined to determine the preset value. Specifically, the product of the reference duration and the frame rate can be used as the preset value. For example, the frame rate of the camera is 25 fps, and the reference duration is 1 second, and the preset value can be set to 25; or the frame rate of the camera is 15 fps, and the reference duration is 1 second, and the preset value can be set to 15. Other cases can be similarly deduced, and will not be listed one by one here. In the above manner, the video is captured by the camera, and the preset value is determined based on the frame rate of the camera, so that the timing feature information sufficient for subsequent prediction can be ensured as much as possible, which helps to further improve the accuracy of gesture recognition.

[0043] In a specific implementation scenario, in response to the total number of times being not less than the preset value, the step of predicting the target image feature based on the latest updated target image feature to obtain the predicted image of the preset category can be performed. That is, in the case where the total number of times is not less than the preset value, the loop process can be ended, and the subsequent prediction process can be continued.

[0044] In a specific implementation scenario, taking the preset numerical value of 2 as an example, for the three video frame images ti, ti+1, ti+2 in the photographed video, the image features Fi, Fi+1, Fi+2 can be respectively extracted in the first round, on this basis, the sub-feature located at the target sequence in the image feature Fi of the video frame image ti can be replaced with the sub-feature located at the target sequence in the image feature Fi-1 of the video frame image ti-1 (if the video frame image ti-1 does not exist, it is equivalent to the sub-feature located at the target sequence in the image feature Fi of the video frame image ti is eliminated from the image feature Fi of the video frame image ti), to obtain the latest updated image feature Gi, for the image feature Fi+1 of the video frame image ti+1 and the image feature Fi+2 of the video frame image ti+2, the same can be done to obtain the latest updated image features Gi+1 and Gi+2 respectively, the specific process is not repeated here, and the total number is 1 at this time. On this basis, the latest updated image features Gi, Gi+1 and Gi+2 are continuously extracted to obtain the image features Hi, Hi+1 and Hi+2, and the sub-feature located at the target sequence in the image feature Hi of the video frame image ti is continuously replaced with the sub-feature located at the target sequence in the image feature Hi-1 of the video frame image ti-1 (if the video frame image ti-1 does not exist, it is equivalent to the sub-feature located at the target sequence in the image feature Hi of the video frame image ti is eliminated from the image feature Hi of the video frame image ti), to obtain the latest updated image feature Ii, for the image feature Hi+1 of the video frame image ti+1 and the image feature Hi+2 of the video frame image ti+2, the same can be done to obtain the latest updated image features Ii+1 and Ii+2 respectively, the specific process is not repeated here, and the total number is 2 at this time. Since the total number does not meet the preset number, the loop process can be ended, and the subsequent prediction process can be continued. At this time, if the video frame image ti is the target image, the latest updated image feature Ii can be used for prediction, or if the video frame image ti+1 is the target image, the latest updated image feature Ii+1 can be used for prediction; or if the video frame image ti+2 is the target image, the latest updated image feature Ii+2 can be used for prediction. Other cases can be done in the same way, which will not be repeated here.

[0045] In a specific implementation scenario, as described above, in order to improve the efficiency of gesture recognition, a gesture recognition model can be trained, and as Figure 2As shown, the gesture recognition model can include a preset number of sub-networks connected in sequence, and a prediction network connected with the last sub-network, and the sub-networks include a feature extraction network and a time offset module; wherein the prediction network is used to predict a prediction image of a plurality of preset categories based on the latest updated target image feature, the feature extraction network is used to perform feature extraction, and the time offset module is used to perform feature replacement. In the above manner, the prediction image is predicted by the gesture recognition model, and the gesture prediction model includes a preset number of sub-networks connected in sequence, and a prediction network connected with the last sub-network, and the sub-networks include a feature extraction network and a time offset module, the prediction network is used to predict a prediction image of a plurality of preset categories based on the latest updated target image feature, the feature extraction network is used to perform feature extraction, and the time offset module is used to perform feature replacement, so that for each image, as many historical images as possible can be referred to to construct time sequence feature information, which helps to improve the accuracy of subsequent gesture recognition.

[0046] Step S13: Based on each prediction image, the gesture recognition result of the target image is obtained.

[0047] In the embodiments of the present disclosure, the gesture recognition result at least includes: whether there is a gesture in the target image, and in the case that there is a gesture in the target image, the gesture category of each gesture.

[0048] In one implementation scenario, each prediction image can be post-processed based on each prediction image to obtain a post-processed image corresponding to each prediction image, and the post-processing can include: denoising, closing operation and watershed operation. On this basis, the gesture recognition result of the target image can be obtained based on the post-processed image corresponding to each prediction image. In the above manner, after obtaining a plurality of prediction images of a plurality of preset categories, a post-processed image is obtained through a series of post-processing operations, and on this basis, the gesture recognition result can be obtained, which can help to reduce the complexity of gesture recognition.

[0049] In one specific implementation scenario, a low-pass filter can be designed in advance, and on this basis, the low-pass filter can be used to denoise each prediction image to filter out first pixel points (i.e. noise points) in the prediction image whose pixel values are lower than a preset threshold. Illustratively, the pixel value of the first pixel point in the prediction image represents the probability value that the second pixel point in the target image corresponding to the position of the first pixel point belongs to the target category, and after denoising the prediction image by the low-pass filter, the interference can be reduced as much as possible, which helps to improve the accuracy of gesture recognition. In addition, please refer to Figure 3 , Figure 3 is a process schematic diagram of one embodiment of post-processing. As shown in the prediction image Figure 3 Figure 2 ​The first pixel point in the prediction image predicted by the gesture recognition model (i.e., the prediction image containing preset categories such as "OK", "fist", "no gesture", etc.) and not highlighted in the prediction image can be considered as the probability value of the second pixel point corresponding to the first pixel point in the target image belonging to the target category being 0. The above "filtering" can specifically include: setting the first pixel point with a pixel value lower than a preset threshold in the prediction image to 0.

[0050] In one specific implementation scenario, the closing operation includes sequentially performing dilation processing and erosion processing on the denoised prediction image, which can fill relatively narrow discontinuities or gullies, eliminate small holes, and fill in interrupted contour lines. For specific processes of the dilation processing and the erosion processing, refer to the technical details of the closing operation, which will not be described here.

[0051] In one specific implementation scenario, the watershed operation is used to connect pixel points that are close in spatial position and have similar gray values to form a closed contour. For specific processes of the watershed operation, refer to the technical details of the watershed operation, which will not be described here.

[0052] In one specific implementation scenario, after obtaining the post-processing image corresponding to each prediction image, contour detection can be performed on the post-processing image corresponding to each prediction image to obtain a contour detection result corresponding to each prediction image, and the contour detection result at least includes whether there is a contour in the post-processing image. On this basis, the gesture recognition result of the target image can be obtained based on the contour detection result corresponding to the prediction image and the preset category to which the prediction image belongs. For details, refer to Figure 3 For example, Figure 3As shown, after the watershed operation, the contour detection can be performed on the post-processing image corresponding to each prediction image. If a contour is detected, the contour can be marked with the minimum circumscribed rectangle, so as to obtain the contour detection result. On this basis, if the contour detection result corresponding to the prediction image includes an existing contour, and the preset category to which the prediction image belongs is "no gesture", it can be determined that the contour is not a gesture, and can be a background, a region other than a hand of a person, or the like. Similarly, if the contour detection result corresponding to the prediction image includes a non-existing contour, and the preset category to which the prediction image belongs is "no gesture", it can be considered that there is no background or region other than the hand of the person in the target image, and based on this, the prediction image of other preset categories can be further viewed. Further, if the preset category to which the prediction image belongs is a gesture category (for example, "OK"), and the contour detection result corresponding to the prediction image includes an existing contour, it can be determined that the gesture recognition result of the target image includes: there is a gesture in the target image, and the gesture category of the gesture is "OK". Conversely, if the preset category to which the prediction image belongs is a gesture category (for example, "OK"), and the contour detection result corresponding to the prediction image includes a non-existing contour, it can be determined that the gesture recognition result of the target image includes: there is no gesture of the gesture category "OK" in the target image. Further, in order to distinguish each gesture in the target image, in the case that the preset category to which the prediction image belongs is a gesture category, and the contour detection result corresponding to the prediction image includes an existing contour, the gesture position of the gesture in the target image can be further determined based on the detected contour. For example, the preset position (for example, the corner point, the midpoint, or the like) of the minimum circumscribed rectangle of the contour can be taken as the gesture position of the gesture corresponding to the contour in the target image. The above method performs contour detection on the post-processing image corresponding to each prediction image, obtains the contour detection result corresponding to each prediction image, and the contour detection result at least includes: whether there is a contour in the post-processing image. On this basis, the gesture recognition result of the target image is obtained based on the contour detection result corresponding to the prediction image and the preset category to which the prediction image belongs. Therefore, after obtaining the post-processing image, only contour detection and other analysis need to be performed, so as to obtain the gesture recognition result, which helps to reduce the complexity of gesture recognition as much as possible.

[0053] In one implementation scenario, different from the aforementioned manner of obtaining the gesture recognition result based on a series of post-processing analysis, in order to further reduce the complexity of gesture recognition, each prediction image can also be subjected to denoising processing (for details, refer to the foregoing related description). On this basis, each denoised prediction image can be subjected to binarization processing. Illustratively, a first pixel point in the prediction image with a pixel value lower than a threshold value can have the pixel value reset to a first value (such as 0), and a first pixel point in the prediction image with a pixel value not lower than the threshold value can have the pixel value reset to a second value (such as 1). Further, the prediction image subjected to binarization processing can be subjected to connected component analysis, so that the gesture recognition result of the target image can be obtained based on each prediction image subjected to connected component analysis. Specifically, each prediction image subjected to connected component analysis can be subjected to connected component detection to obtain a connected component detection result of each prediction image subjected to connected component analysis, and the connected component detection result can at least include whether there is a connected component in the prediction image subjected to connected component analysis, and the gesture recognition result of the target image can be obtained based on the connected component detection result of the prediction image subjected to connected component analysis and the preset category to which the prediction image belongs. For example, if the connected component detection result of the prediction image subjected to connected component analysis includes that there is a connected component, and the preset category to which the prediction image belongs is a “gesture category” (such as “OK”), it can be determined that there is a gesture in the target image, and the gesture category of the gesture is “OK”, otherwise, if the connected component detection result of the prediction image subjected to connected component analysis includes that there is no connected component, and the preset category to which the prediction image belongs is a “gesture category” (such as “OK”), it can be determined that there is no gesture of the gesture category (such as “OK”) in the target image. Similarly, if the connected component detection result of the prediction image subjected to connected component analysis includes that there is a connected component, and the preset category to which the prediction image belongs is “no gesture”, it can be considered that there is a background, a region other than a hand of a person, etc. in the target image, and other cases can refer to the foregoing related description about “obtaining the gesture recognition result of the target image based on the contour detection result corresponding to the prediction image and the preset category to which the prediction image belongs”, which will not be described here. Further, in order to distinguish each gesture in the target image, if the connected component detection result of the prediction image subjected to connected component analysis includes that there is a connected component, the preset position (such as a corner point, a midpoint, etc.) of the minimum bounding rectangle of the connected component can be further taken as the gesture position of the gesture in the target image.

[0054] The above scheme extracts target image features of a target image in the photographed video, extracts reference image features of a reference image before the target image in the photographed video, and performs prediction based on the target image features and the reference image features to obtain predicted images of a plurality of preset categories, wherein the plurality of preset categories include a no-gesture category and at least one gesture category, and a pixel value of a first pixel point in the predicted image represents a possibility that a second pixel point corresponding to the first pixel point in the target image belongs to a target category, and the target category is a preset category to which the predicted image belongs. On this basis, gesture recognition results of the target image are obtained based on the predicted images, and the gesture recognition results at least include whether there is a gesture in the target image, and in the case where there is a gesture in the target image, a gesture category of each gesture. On the one hand, since the possibility that the target image pixel point respectively belongs to the plurality of preset categories is directly predicted, i.e., there is no need to further divide into two-stage tasks of person detection and gesture recognition, the influence of the number of persons on the calculation load can be reduced as much as possible, and the applicability to multi-person gesture interaction can be improved. On the other hand, since the gesture recognition process does not depend on a sensor, the influence of factors such as walking and light on gesture recognition can be reduced as much as possible, and the accuracy of gesture recognition can be improved. On the other hand, since the possibility that the target image pixel point respectively belongs to the plurality of preset categories is directly predicted in the gesture recognition process, and does not depend on hand key point detection, even if the person is far away from the camera, the accuracy of gesture recognition can be ensured as much as possible. Therefore, the accuracy of gesture recognition can be further improved while ensuring applicability to multi-person gesture interaction, and the calculation load of gesture recognition can be reduced as much as possible, and the applicability in a long-distance use scenario can be improved.

[0055] Please refer to Figure 4 , Figure 4 is a flowchart of an embodiment of the gesture interaction method of the present application. Specifically, it can include the following steps:

[0056] Step S41: obtaining gesture recognition results of a target image in a photographed video.

[0057] In the embodiment of the present disclosure, the gesture recognition results can be obtained based on the steps in any of the above gesture recognition method embodiments. In addition, the gesture recognition results can include gesture categories of each gesture in the target image, which can be referred to the related description in the foregoing gesture recognition method embodiments, and will not be described here.

[0058] Step S42: responding to each gesture respectively based on the gesture category of each gesture.

[0059] In one embodiment scenario, the interaction mapping relationship about the gesture category-response mode can be pre-set, and on this basis, the response mode of each gesture can be determined according to the gesture category of each gesture in the gesture recognition result, and each gesture can be responded according to the response mode of each gesture. Exemplarily, the interaction mapping relationship can include but is not limited to: gesture category "OK" - response mode "snap", gesture category "fist" - response mode "hibernate", etc., which is not limited herein.

[0060] In one embodiment scenario, the gesture recognition result can further include the spatial position of each gesture. It should be noted that the spatial position of the gesture can specifically include the position coordinates of the gesture in the three-dimensional space. In addition, the spatial position of each gesture can be calculated based on the gesture position of each gesture in the target image. For specific acquisition process of the gesture position, please refer to the related description in the foregoing disclosed embodiments, which will not be described here. On this basis, in order to further improve the reliability of multi-person interaction, each gesture can be responded based on the gesture category and the spatial position of each gesture. Compared with responding to the gesture based on only the gesture category, the above-mentioned manner of responding to each gesture based on the gesture category and the spatial position of each gesture can further improve the reliability of gesture interaction.

[0061] In one specific embodiment scenario, based on the determination of the gesture position of each gesture in the target image, the spatial position of the gesture can be obtained by back projection based on the internal parameters and the external parameters of the camera and the gesture position. Please refer to Figure 5 , Figure 5 for back projection an embodiment of the schematic diagram. As shown in Figure 5 , in order to facilitate the description, the gesture position of the gesture in the target image can be recorded as (u, v), and the spatial position of the gesture can be recorded as (X, Y, Z), and the following relationship should be met between them:

[0062]

[0063] In the above formula (1), [R|T] represents the external parameters of the camera, wherein R represents the rotation matrix of the world coordinate system to the camera coordinate system, and T represents the translation matrix of the world coordinate system to the camera coordinate system, and are the internal parameters of the camera, wherein f represents the focal length of the camera, d x and d y respectively represent the length unit corresponding to one pixel in x and y directions in the image shot by the camera, γ represents the distortion factor of the camera, u0 and v0 respectively represent the pixel number of the upper left corner point to the center in the u direction and the v direction of the image shot by the camera, Z crepresents the depth in the camera coordinate system (i.e. the distance from the camera to the interactive person). Further, the above formula (1) can be rewritten as:

[0064]

[0065] The matrix M is further split as:

[0066]

[0067] Thus, the spatial position can be calculated as:

[0068]

[0069] In one specific implementation scenario, an interactive mapping relationship among gesture category, spatial position and response mode can be pre-set, based on which the response mode for each gesture can be determined according to the spatial position of the gesture category in the gesture recognition result, and each gesture can be responded according to the response mode of each gesture. Exemplarily, the interactive mapping relationship can include: gesture category "OK" - spatial position "leftmost" - response mode "start snapshot", gesture category "OK" - spatial position "rightmost" - response mode "end snapshot", gesture category "fist" - spatial position "leftmost" - response mode "start dormancy", gesture category "fist" - spatial position "rightmost" - response mode "end dormancy", etc., which are not limited herein. It should be noted that the above "leftmost", "rightmost", etc. related descriptions can be determined according to the three-dimensional coordinates represented by the spatial position.

[0070] In one implementation scenario, please refer to Figure 6 , Figure 6 is the process schematic diagram of one embodiment of the gesture interaction method of the present application. As Figure 6As shown, for the application scenario of long-distance multi-person, the camera collects and captures the video, and the gesture recognition model is used to recognize the captured image in the video, so as to obtain the predicted image of the target image in the captured video in several preset categories, and the predicted image in several categories is post-processed (such as denoising, closing operation, watershed operation, etc.), and then it is detected whether the gesture category is triggered, that is, for the predicted image in the preset category "gesture category", whether the contour is detected after the post-processing, if the contour is detected, the "gesture category" can be triggered, otherwise, it can be considered that the "gesture category" does not exist in the target image, if all the predicted images in the preset category do not trigger the gesture category, the shooting can be continued, and the gesture recognition of the subsequent video frame image can be continued. Further, if the gesture category is triggered, the spatial position of the gesture can be calculated based on the gesture position of the gesture in the target image and the camera parameters (such as internal parameters, external parameters, etc.) of the camera. On this basis, the spatial positions of the gestures and the gesture categories in the target image can be combined to respond to each gesture respectively.

[0071] The above scheme obtains the gesture recognition result of the target image in the captured video, and the gesture recognition result is obtained based on any one of the gesture recognition method embodiments. The gesture recognition result includes the gesture category of each gesture in the target image. On this basis, each gesture is responded based on the gesture category of each gesture, so that the accuracy of gesture recognition can be further improved while ensuring that it is suitable for multi-person gesture interaction, and the computational load of gesture recognition can be reduced as much as possible, and the applicability in long-distance use scenarios can be improved.

[0072] Please refer to Figure 7 , Figure 7 is a frame diagram of an embodiment of the gesture recognition device 70 of the present application. The gesture recognition device 70 includes an extraction module 71, a prediction module 72 and an analysis module 73. The extraction module 71 is used to extract the target image features of the target image in the captured video, and extract the reference image features of the reference image before the target image in the captured video. The prediction module 72 is used to predict based on the target image features and the reference image features to obtain the predicted image in several preset categories. The several preset categories include no gesture and at least one gesture category. The pixel value of the first pixel point in the predicted image represents the possibility that the second pixel point corresponding to the first pixel point position in the target image belongs to the target category, and the target category is the preset category to which the predicted image belongs. The analysis module 73 is used to analyze based on each predicted image to obtain the gesture recognition result of the target image. The gesture recognition result at least includes whether there is a gesture in the target image, and in the case that there is a gesture in the target image, the gesture category of each gesture.

[0073] The above scheme, on the one hand, directly predicts the possibility that the target image pixel points belong to a plurality of preset categories, i.e., without the need for two-stage tasks of person detection and gesture recognition, thereby being able to reduce the influence of the number of persons on the computing load as much as possible, and further being able to improve the applicability to the multi-person gesture interaction situation; on the other hand, the gesture recognition process does not depend on the sensor device, and the influence of factors such as walking and light on gesture recognition can be reduced as much as possible, thereby being able to improve the accuracy of gesture recognition; on the other hand, the gesture recognition process directly predicts the possibility that the target image pixel points belong to a plurality of preset categories, without relying on hand key point detection, so that even if the person is far away from the camera, the accuracy of gesture recognition can be ensured as much as possible. Therefore, the accuracy of gesture recognition can be further improved while ensuring applicability to multi-person gesture interaction, and the computing load of gesture recognition can be reduced as much as possible, and the applicability in long-distance use scenarios can be improved.

[0074] In some disclosed embodiments, the target image feature includes a plurality of sub-features arranged in a channel order, and the reference image feature includes a plurality of sub-features arranged in a channel order; the prediction module 72 includes a first offset sub-module for replacing the sub-feature located at the target sequence position in the target image feature with the sub-feature located at the target sequence position in the first image feature to update the target image feature; wherein the first image feature is the reference image feature of the reference image adjacent to the target image; the prediction module 72 includes an image prediction sub-module for performing prediction based on the latest updated target image feature to obtain a plurality of preset categories of predicted images.

[0075] In some disclosed embodiments, the prediction module 72 further includes a number of judgment sub-module for judging whether the total number of times of performing feature replacement on the target image feature is less than a preset value; the prediction module 72 includes a second offset sub-module for, in response to the total number of times being less than the preset value, taking each of the latest obtained reference image features as a second image feature, and replacing the sub-feature located at the target sequence position in the second image feature with the sub-feature located at the target sequence position in the third image feature to update the second image feature; wherein the third image feature is the latest obtained reference image feature of a previous reference image of the reference image to which the second image feature belongs; the prediction module 72 further includes a feature extraction sub-module for performing feature extraction based on the latest updated target image feature to update the target image feature, and performing feature extraction based on the latest updated reference image feature to update the reference image feature; the prediction module 72 further includes a loop execution sub-module for re-executing the steps of replacing the sub-feature located at the target sequence position in the target image feature with the sub-feature located at the target sequence position in the first image feature to update the target image feature and the subsequent steps in combination with the first offset sub-module, the number of judgment sub-module, the second offset sub-module, and the feature extraction sub-module.

[0076] In some disclosed embodiments, the predicted image is predicted by a gesture prediction model, which includes a preset number of sub-networks connected sequentially, and a prediction network connected to the last sub-network, and the sub-network includes a feature extraction network and a timing offset module; wherein the prediction network is used to predict predicted images of several preset categories based on the latest updated target image features, the feature extraction network is used to perform feature extraction, and the timing offset module is used to perform feature replacement.

[0077] In some disclosed embodiments, the captured video is acquired by a camera, and the preset value is determined based on the frame rate of the camera.

[0078] In some disclosed embodiments, the analysis module 73 includes a post-processing submodule for performing post-processing based on each predicted image to obtain a post-processed image corresponding to each predicted image; wherein the post-processing includes: denoising, closing operation and watershed operation; the analysis module 73 includes an image analysis submodule for obtaining a gesture recognition result of the target image based on the post-processed image corresponding to each predicted image.

[0079] In some disclosed embodiments, the image analysis submodule includes a contour detection unit for performing contour detection on the post-processed images corresponding to each predicted image to obtain a contour detection result corresponding to each predicted image; wherein the contour detection result includes at least: whether there is a contour in the post-processed image; the image analysis submodule includes a result analysis unit for obtaining a gesture recognition result of the target image based on the contour detection result corresponding to the predicted image and the preset category to which the predicted image belongs.

[0080] See also Figure 8 , Figure 8 Schematic diagram of a framework of an embodiment of a gesture interaction device 80 of the present application. Gesture interaction device 80 includes a result acquisition module 81 and a gesture response module 82. Result acquisition module 81 is configured to acquire gesture recognition results for a target image in a captured video. The gesture recognition results are obtained based on the gesture recognition device described in the third aspect above and include gesture categories for each gesture in the target image. Gesture response module 82 is configured to respond to each gesture based on its category.

[0081] The above scheme obtains gesture recognition results for a target image in a captured video. These results, obtained based on any of the above-described gesture recognition device embodiments, include the gesture categories of each gesture in the target image. Based on these categories, each gesture is then responded to separately. This approach further improves gesture recognition accuracy while ensuring suitability for multi-person gesture interaction, minimizes the computational load of gesture recognition, and enhances applicability in remote use scenarios.

[0082] Please refer to Figure 9 , Figure 9 is a schematic diagram of a framework of an embodiment of the electronic device 90. The electronic device 90 includes a camera 91, a memory 92, and a processor 93, the camera 91 and the memory 92 are respectively coupled to the processor 93, the memory 92 stores program instructions, and the processor 93 is configured to execute the program instructions to implement the steps in any of the above gesture recognition method embodiments. Specifically, the electronic device 90 can include but is not limited to a surveillance camera, a video conference camera, a smart speaker, and the like, which are not limited herein.

[0083] Specifically, the processor 93 is configured to control itself and the camera 91 and the memory 92 to implement the steps in any of the above gesture recognition method embodiments, or to implement the steps in any of the above gesture interaction method embodiments. The processor 93 can also be referred to as a CPU (Central Processing Unit). The processor 93 can be an integrated circuit chip with processing capability. The processor 93 can also be a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 93 can be jointly implemented by an integrated circuit chip.

[0084] The above scheme, on the one hand, since the possibility that the target image pixel points respectively belong to a plurality of preset categories is directly predicted, i.e. without the need for two-stage tasks of person detection and gesture recognition, the influence of the number of persons on the calculation load can be reduced as much as possible, and the applicability to multi-person gesture interaction can be improved. On the other hand, since the gesture recognition process does not depend on the sensor, the influence of factors such as walking and light on gesture recognition can be reduced as much as possible, and the accuracy of gesture recognition can be improved. On the other hand, since the possibility that the target image pixel points respectively belong to a plurality of preset categories is directly predicted in the gesture recognition process, and does not depend on hand key point detection, even if the person is far away from the camera, the accuracy of gesture recognition can be ensured as much as possible. Therefore, the accuracy of gesture recognition can be further improved while ensuring applicability to multi-person gesture interaction, and the calculation load of gesture recognition can be reduced as much as possible, and the applicability in long-distance use scenarios can be improved.

[0085] Please refer to Figure 10, Figure 10 is a schematic diagram of a framework of an embodiment of the computer readable storage medium 100. The computer readable storage medium 100 stores program instructions 101 capable of being executed by a processor, the program instructions 101 being used to implement the steps in any of the gesture recognition method embodiments described above, or implement the steps in any of the gesture interaction method embodiments described above.

[0086] The above scheme, on the one hand, since the likelihoods that the target image pixel points respectively belong to a plurality of preset categories are directly predicted, i.e., without the need for two-stage tasks of person detection and gesture recognition, the influence of the number of persons on the computing load can be reduced as much as possible, and thus the applicability to the multi-person gesture interaction situation can be improved. On the other hand, since the gesture recognition process does not depend on a sensor, the influence of factors such as walking and light on gesture recognition can be reduced as much as possible, and thus the accuracy of gesture recognition can be improved. On the other hand, since the likelihoods that the target image pixel points respectively belong to a plurality of preset categories are directly predicted in the gesture recognition process, without the need for hand key point detection, even if the persons are far away from the camera, the accuracy of gesture recognition can be ensured as much as possible. Therefore, while ensuring the applicability to the multi-person gesture interaction situation, the accuracy of gesture recognition can be further improved, the computing load of gesture recognition can be reduced as much as possible, and the applicability in a long-distance use scenario can be improved.

[0087] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or contains modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0088] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0089] In several embodiments provided by the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the device embodiment described above is only schematic. The division of the modules or units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0090] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0091] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0092] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present application essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0093] If the technical scheme of the present application involves personal information, the product applying the technical scheme of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical scheme of the present application involves sensitive personal information, the product applying the technical scheme of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as a camera, a clear and conspicuous mark is set to inform that the personal information collection range has been entered, and the personal information will be collected. If the person voluntarily enters the collection range, it is regarded as consent to collect the personal information. Or, on the device for processing personal information, the personal information processing rules are informed by using obvious marks / information, and the personal authorization is obtained by means of pop-up information or asking the person to upload his / her personal information. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

Claims

1. A gesture recognition method, characterized in that: include: Extracting target image features of a target image in a captured video, and extracting reference image features of a reference image located before the target image in the captured video; Prediction is performed based on the target image features and the reference image features to obtain predicted images of several preset categories; wherein the several preset categories include no gesture and at least one gesture category, the pixel value of a first pixel in the predicted image represents: the possibility that a second pixel corresponding to the position of the first pixel in the target image belongs to a target category, and the target category is a preset category to which the predicted image belongs, and the predicted image is predicted by a gesture prediction model, the gesture prediction model including a preset number of subnetworks connected sequentially, and a prediction network connected to the last of the subnetworks, and the subnetwork including a feature extraction network and a timing offset module; Based on the analysis of each of the predicted images, a gesture recognition result of the target image is obtained; wherein the gesture recognition result at least includes: whether there is a gesture in the target image, and when the gesture exists in the target image, the gesture category of each of the gestures.

2. The method according to claim 1, characterized in that The target image feature includes a plurality of sub-features arranged in channel order, and the reference image feature includes a plurality of sub-features arranged in channel order; and performing prediction based on the target image feature and the reference image feature to obtain a plurality of preset categories of predicted images includes: Replacing the sub-feature at the target sequence position in the target image feature with the sub-feature at the target sequence position in the first image feature to update the target image feature; wherein the first image feature is a reference image feature of a reference image adjacent to the target image; Prediction is performed based on the latest updated target image features to obtain predicted images of the plurality of preset categories.

3. The method according to claim 2, characterized in that Before performing prediction based on the latest updated target image features to obtain the predicted images of the plurality of preset categories, the method further includes: Determine whether the total number of times the target image feature performs feature replacement is less than a preset value; In response to the total number of times being less than the preset value, each of the most recently acquired reference image features is used as a second image feature, and a sub-feature in the second image feature located at the target sequence is replaced with a sub-feature in the third image feature located at the target sequence, so as to update the second image feature; wherein the third image feature is: a most recently acquired reference image feature of the reference image preceding the reference image to which the second image feature belongs; Feature extraction is performed based on the most recently updated target image feature to update the target image feature, and feature extraction is performed based on the most recently updated reference image feature to update the reference image feature, and the step of replacing the sub-feature located at the target sequence in the target image feature with the sub-feature located at the target sequence in the first image feature to update the target image feature and subsequent steps are re-executed.

4. The method according to claim 2, characterized in that The prediction network is used to predict the predicted images of the preset categories based on the latest updated target image features, the feature extraction network is used to perform feature extraction, and the temporal offset module is used to perform feature replacement.

5. The method according to claim 3 or 4, characterized in that The captured video is acquired by a camera, and the preset value is determined based on the frame rate of the camera.

6. The method according to claim 1, characterized in that The analyzing each of the predicted images to obtain a gesture recognition result of the target image includes: Performing post-processing on each of the predicted images to obtain a post-processed image corresponding to each of the predicted images; wherein the post-processing includes: denoising, closing operation and watershed operation; A gesture recognition result of the target image is obtained based on the post-processed images corresponding to each of the predicted images.

7. The method according to claim 6, characterized in that Obtaining a gesture recognition result of the target image based on the post-processed images corresponding to the respective predicted images includes: Performing contour detection on the post-processed images corresponding to the respective predicted images to obtain contour detection results corresponding to the respective predicted images; wherein the contour detection results at least include: whether a contour exists in the post-processed images; A gesture recognition result of the target image is obtained based on a contour detection result corresponding to the predicted image and a preset category to which the predicted image belongs.

8. A gesture interaction method, characterized in that: include: Obtaining a gesture recognition result of a target image in a captured video; wherein the gesture recognition result is obtained based on the gesture recognition method according to any one of claims 1 to 7, and the gesture recognition result includes: a gesture category of each gesture in the target image; Based on the gesture categories of the gestures, the gestures are responded to respectively.

9. A gesture recognition device, characterized in that: include: an extraction module, configured to extract target image features of a target image in a captured video, and extract reference image features of a reference image preceding the target image in the captured video; A prediction module, configured to perform prediction based on the target image features and the reference image features to obtain predicted images of several preset categories; wherein the several preset categories include no gesture and at least one gesture category, the pixel value of a first pixel in the predicted image represents: the probability that a second pixel corresponding to the position of the first pixel in the target image belongs to a target category, and the target category is a preset category to which the predicted image belongs, and the predicted image is predicted by a gesture prediction model, the gesture prediction model including a preset number of subnetworks connected sequentially, and a prediction network connected to the last of the subnetworks, and the subnetwork including a feature extraction network and a timing offset module; An analysis module is used to perform analysis based on each of the predicted images to obtain a gesture recognition result of the target image; wherein the gesture recognition result at least includes: whether there is a gesture in the target image, and when the gesture exists in the target image, the gesture category of each of the gestures.

10. A gesture interaction device, characterized in that: include: a result acquisition module, configured to acquire a gesture recognition result of a target image in a captured video; wherein the gesture recognition result is obtained based on the gesture recognition device according to claim 9, and the gesture recognition result includes: a gesture category of each gesture in the target image; The gesture response module is configured to respond to each gesture based on the gesture category of each gesture.

11. An electronic device, characterized in that: It includes a camera, a memory and a processor, the camera and the memory are respectively coupled to the processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the gesture recognition method described in any one of claims 1 to 7, or to implement the gesture interaction method described in claim 8.

12. A computer-readable storage medium, characterized in that Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the gesture recognition method described in any one of claims 1 to 7, or the gesture interaction method described in claim 8.

Images