Video gesture recognition method and device based on deep learning and electronic equipment

Abstract

The application discloses a video gesture recognition method and device based on deep learning and electronic equipment, the method is applied to the field of information security, and the method comprises the steps of determining a first image frame set and a target image frame from a gesture video pre-recorded by a target object; determining a second image frame set and a third image frame set from a target video; processing image frames in the second image frame set and the target image frame; determining a fourth image frame set according to the processed second image frame set and the processed target image frame; obtaining image frames in the third image frame set and image frames in the first image frame set, inputting the image frames into a recurrent neural network to obtain a second similarity set through calculation; and determining a gesture recognition result of the target video according to the fourth image frame set and the first similarity set. Through the application, the problem that the accuracy of a gesture recognition result is low due to background interference when gestures are recognized from a video in the related art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video recognition, in particular to a video gesture recognition method and device based on deep learning and electronic equipment. BACKGROUND

[0002] At present, when recognizing gestures in a video, the video is often decomposed into a plurality of image frames, and then each image frame is processed and analyzed. However, in addition to the gestures to be compared, the video also contains different backgrounds. The complex background leads to an inaccurate recognition result of the video gestures.

[0003] One of the existing processing methods is to use an openCV function to calculate a plurality of similarities between each image frame of a to-be-recognized video and each image frame of an original video, and obtain a recognition result after weighted calculation. In addition, another method is to calculate each image frame of the to-be-recognized video and a preset gesture image frame of the original video through a neural network, and then obtain a recognition result after weighted calculation of a plurality of similarities. However, the above methods do not take into account that the background of the gestures in the two captured videos is difficult to be completely the same, so when calculating the similarity between the image frames, a large error will occur due to the interference of the background.

[0004] At present, there is no effective solution to the problem of low accuracy of gesture recognition results caused by background interference when recognizing gestures from a video in the related art. SUMMARY

[0005] The main purpose of the present application is to provide a video gesture recognition method and device based on deep learning and electronic equipment, so as to solve the problem of low accuracy of gesture recognition results caused by background interference when recognizing gestures from a video in the related art.

[0006] In order to achieve the above object, according to one aspect of the present application, a video gesture recognition method based on deep learning is provided, which comprises: determining a first image frame set and a target image frame from a gesture video pre-recorded by a target object, wherein each image frame in the first image frame set is an image with a gesture, and the target image frame is an image frame representing a gesture determined by the target object; obtaining a target video, and determining a second image frame set and a third image frame set from the target video, wherein the similarity between the image frames in the second image frame set and the target image frame is higher than the similarity between the image frames in the third image frame set and the target image frame, and the target video is a video to be subjected to gesture recognition; processing the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame; determining a fourth image frame set according to the processed second image frame set and the processed target image frame; inputting the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network for similarity comparison, outputting the similarity between the image frames in the third image frame set and the image frames in the first image frame set, and obtaining a first similarity set; and determining a gesture recognition result of the target video according to the fourth image frame set and the first similarity set.

[0007] Further, the determination of the first image frame set and the target image frame from the gesture video pre-recorded by the target object comprises: obtaining a target image frame set by the target object for the pre-recorded gesture video; extracting each image frame in the pre-recorded gesture video of the target object to obtain a fifth image frame set; processing the fifth image frame set by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set, wherein the similarity between the image frames in the sixth image frame set and the target image frame is higher than the similarity between the image frames in the seventh image frame set and the target image frame; screening out image frames containing hands from the seventh image frame set by using a pre-trained hand detection model, adding the image frames containing hands in the seventh image frame set to the sixth image frame set, and taking the added sixth image frame set as the first image frame set.

[0008] Further, the obtaining of the target video and the determination of the second image frame set and the third image frame set from the target video comprise: extracting each image frame in the target video to obtain an eighth image frame set; calculating the second similarity between each image frame in the eighth image frame set and the target image frame by using a pre-trained first convolutional neural network to obtain a second similarity set; obtaining images with a second similarity greater than or equal to a first threshold value to obtain the second image frame set; and obtaining images with a second similarity less than the first threshold value to obtain the third image frame set.

[0009] Further, the image frames in the second image frame set and the target image frame are processed to obtain a processed second image frame set and a processed target image frame, including: identifying, by the pre-trained second convolutional neural network, a first area range occupied by the hand of the target object in the image frames in the second image frame set and a second area range occupied by the hand of the target object in the target image frame; covering the first area range and the second area range with a first color, and covering the remaining area ranges in the image frames in the second image frame set except the first area range and the remaining area ranges in the target image frame except the second area range with a second color, to obtain the processed second image frame set and the processed target image frame.

[0010] Further, the gesture recognition result of the target video is determined according to the fourth image frame set and the first similarity set, including: processing the values in the first similarity set to obtain a second similarity; calculating the sum of the number of image frames in the second image frame set and the number of image frames in the third image frame set to obtain a total number of image frames; calculating the ratio of the number of image frames in the fourth image frame set to the total number of image frames; if it is determined that the ratio is greater than or equal to a second threshold value, and it is determined that the second similarity is greater than or equal to a third threshold value, then it is determined that the gesture recognition result of the target video is that the gesture in the target video is the gesture in the pre-recorded gesture video of the target object.

[0011] Further, the fourth image frame set is determined according to the processed second image frame set and the processed target image frame, including: inputting the processed second image frame set and the processed target image frame into the first convolutional neural network to output a third similarity between the image frames in the processed second image frame set and the processed target image frame; and selecting, from the processed second image frame set, image frames with a third similarity greater than a fourth threshold value with the processed target image frame to obtain the fourth image frame set.

[0012] Further, the fifth image frame set is processed by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set. The fifth image frame set is clustered by using the clustering algorithm to obtain a ninth image frame set and a tenth image frame set, wherein the ninth image frame set and the tenth image frame set contain different pictures. The ninth image frame set, the tenth image frame set, and the target image frame are subjected to histogram analysis to determine the average similarity between the image frames in the ninth image frame set and the target image frame and the average similarity between the image frames in the tenth image frame set and the target image frame. If the average similarity between the image frames in the ninth image frame set and the target image frame is higher than the average similarity between the image frames in the tenth image frame set and the target image frame, the ninth image frame set is taken as the sixth image frame set, and the tenth image frame set is taken as the seventh image frame set. If the average similarity between the image frames in the ninth image frame set and the target image frame is lower than the average similarity between the image frames in the tenth image frame set and the target image frame, the tenth image frame set is taken as the sixth image frame set, and the ninth image frame set is taken as the seventh image frame set.

[0013] To achieve the above object, according to another aspect of the present application, a video gesture recognition device based on deep learning is provided, which comprises: a first determination unit configured to determine a first image frame set and a target image frame from a gesture video pre-recorded by a target object, wherein each image in the first image frame set is an image with a gesture, and the target image frame is an image frame representing a gesture determined by the target object; a second determination unit configured to obtain a target video and determine a second image frame set and a third image frame set from the target video, wherein the similarity between the image frames in the second image frame set and the target image frame is higher than the similarity between the image frames in the third image frame set and the target image frame, and the target video is a video to be subjected to gesture recognition; a first processing unit configured to process the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame; a third determination unit configured to determine a fourth image frame set according to the processed second image frame set and the processed target image frame; a first acquisition unit configured to input the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network for similarity comparison, output the similarity between the image frames in the third image frame set and the image frames in the first image frame set, and obtain a first similarity set; and a fourth determination unit configured to determine a gesture recognition result of the target video according to the fourth image frame set and the first similarity set.

[0014] Further, the first determining unit comprises: a first obtaining sub-unit, configured to obtain a target image frame set for the pre-recorded gesture video of the target object; a second obtaining sub-unit, configured to extract each image frame in the pre-recorded gesture video of the target object to obtain a fifth image frame set; a first processing sub-unit, configured to process the fifth image frame set by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set, wherein the similarity between the image frames in the sixth image frame set and the target image frame is higher than the similarity between the image frames in the seventh image frame set and the target image frame; and a third obtaining sub-unit, configured to filter out image frames containing hands from the seventh image frame set by using a pre-trained hand detection model, add the image frames containing hands in the seventh image frame set to the sixth image frame set, and take the added sixth image frame set as the first image frame set.

[0015] Further, the second determining unit comprises: a fourth obtaining sub-unit, configured to extract each image frame in the target video to obtain an eighth image frame set; a first calculating sub-unit, configured to calculate the second similarity between each image frame in the eighth image frame set and the target image frame by using a pre-trained first convolutional neural network to obtain a second similarity set; a fifth obtaining sub-unit, configured to obtain images with the second similarity greater than or equal to a first threshold value to obtain a second image frame set; and a sixth obtaining sub-unit, configured to obtain images with the second similarity less than the first threshold value to obtain a third image frame set.

[0016] Further, the first processing unit comprises: a first identifying sub-unit, configured to identify the first area range occupied by the hand of the target object in the image frames of the second image frame set and the second area range occupied by the hand of the target object in the target image frame by using a pre-trained second convolutional neural network; and a first covering sub-unit, configured to cover the first area range and the second area range with a first color, and cover the remaining area range in the image frames of the second image frame set except the first area range and the remaining area range in the target image frame except the second area range with a second color to obtain a processed second image frame set and a processed target image frame.

[0017] Further, the fourth determining unit comprises: a second processing sub-unit, configured to process the values in the first similarity set to obtain a second similarity; a second calculating sub-unit, configured to calculate the sum of the number of image frames in the second image frame set and the number of image frames in the third image frame set to obtain a total number of image frames; a third calculating sub-unit, configured to calculate the ratio of the number of image frames in the fourth image frame set to the total number of image frames; and a first determining sub-unit, configured to determine that the gesture recognition result of the target video is that the gesture in the target video is the gesture in the pre-recorded gesture video of the target object if it is determined that the ratio is greater than or equal to a second threshold value and it is determined that the second similarity is greater than or equal to a third threshold value.

[0018] Further, the third determining unit comprises: a first output sub-unit, configured to input the processed second image frame set and the processed target image frame into the first convolutional neural network, and output a third similarity between an image frame in the processed second image frame set and the processed target image frame; and a first screening sub-unit, configured to screen, from the processed second image frame set, an image frame having a third similarity greater than a fourth threshold value to the processed target image frame, to obtain a fourth image frame set.

[0019] Further, the first processing sub-unit comprises: a processing module, configured to perform clustering processing on the fifth image frame set using a clustering algorithm to obtain a ninth image frame set and a tenth image frame set, wherein the ninth image frame set and the tenth image frame set contain different pictures; a first determining module, configured to perform histogram analysis on the ninth image frame set, the tenth image frame set and the target image frame, to determine a mean similarity between an image frame in the ninth image frame set and the target image frame, and a mean similarity between an image frame in the tenth image frame set and the target image frame; a second determining module, configured to determine the ninth image frame set as the sixth image frame set and the tenth image frame set as the seventh image frame set if the mean similarity between the image frame in the ninth image frame set and the target image frame is higher than the mean similarity between the image frame in the tenth image frame set and the target image frame; and a third determining module, configured to determine the tenth image frame set as the sixth image frame set and the ninth image frame set as the seventh image frame set if the mean similarity between the image frame in the ninth image frame set and the target image frame is lower than the mean similarity between the image frame in the tenth image frame set and the target image frame.

[0020] To achieve the above object, according to an aspect of the present application, a processor is provided, the processor being used for running a program, wherein the program, when being executed, performs any one of the above deep learning-based video gesture recognition methods.

[0021] To achieve the above object, according to an aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory being configured to store one or more programs, wherein the one or more programs, when being executed by the one or more processors, cause the one or more processors to implement any one of the above deep learning-based video gesture recognition methods.

[0022] By the present application, the following steps are adopted: determining a first image frame set and a target image frame from a pre-recorded gesture video of a target object, wherein each image frame in the first image frame set is an image with a gesture, and the target image frame is an image frame representing a gesture determined by the target object; obtaining a target video and determining a second image frame set and a third image frame set from the target video, wherein the similarity of the image frames in the second image frame set to the target image frame is higher than the similarity of the image frames in the third image frame set to the target image frame, wherein the target video is a video to be subjected to gesture recognition; processing the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame; determining a fourth image frame set according to the processed second image frame set and the processed target image frame; inputting the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network for similarity comparison, outputting the similarity of the image frames in the third image frame set and the image frames in the first image frame set, and obtaining a first similarity set; determining a gesture recognition result for the target video according to the fourth image frame set and the first similarity set, thereby solving the problem of low accuracy of gesture recognition results caused by background interference when recognizing gestures from a video. By processing the image frames in the pre-recorded gesture video and the to-be-recognized video before calculating the similarity between the pre-recorded gesture video and the target video (to-be-recognized video), the interference of the background in the image frames is reduced, then the similarity comparison is performed, and finally the recognition result is obtained, thereby avoiding the influence of complex background interference on the recognition result in video recognition, and further achieving the effect of improving the recognition accuracy of recognizing gestures in a video. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application illustrated in the drawings, and their description, are presented to explain the application and not to limit or define it. In the drawings:

[0024] Figure 1 is a flowchart of a deep learning-based video gesture recognition method provided according to an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of an optional deep learning-based video gesture recognition method provided according to an embodiment of the present application Figure 1 ;

[0026] Figure 3 is a schematic diagram of an optional deep learning-based video gesture recognition method provided according to an embodiment of the present application Figure 2 ;

[0027] Figure 4 is a schematic diagram of an optional deep learning-based video gesture recognition method provided according to an embodiment of the present applicationFigure 3 ;

[0028] Figure 5 is a schematic diagram of a video gesture recognition device based on deep learning provided according to an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of an electronic device for video gesture recognition based on deep learning provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0031] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0034] The present application will be described below in conjunction with preferred implementation steps, Figure 1 is a flowchart of a video gesture recognition method based on deep learning provided according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0035] Step S101, determining a first image frame set and a target image frame from a pre-recorded gesture video of a target object, wherein each image frame in the first image frame set is an image with a gesture, and the target image frame is an image frame representing a gesture determined by the target object.

[0036] In order to reduce the influence of complex background in the pre-recorded gesture video, in the embodiment, the pre-recorded gesture video of the user (i.e. the target object) is pre-processed, the pre-recorded gesture video is first converted into a plurality of image frames and pre-processed to obtain a first image frame set, wherein only image frames containing hands exist in the first image frame set. At the same time, a preset image frame (i.e. the target image frame) needs to be obtained from the pre-recorded gesture video of the user, which is similar to the video cover and is an image frame selected by the user from the pre-recorded gesture video to represent the gesture of the pre-recorded gesture video.

[0037] Step S102, obtaining a target video and determining a second image frame set and a third image frame set from the target video, wherein the similarity between the image frames in the second image frame set and the target image frame is higher than the similarity between the image frames in the third image frame set and the target image frame, and the target video is a video to be recognized.

[0038] Specifically, the above-mentioned target video can be a video to be recognized (i.e. the target video) shot by the user. In order to obtain a more accurate video gesture recognition result, in the present application, the target video is converted into two image frame sets, one with a relatively high similarity to the target image frame (i.e. the second image frame set), and the other with a relatively low similarity to the target image frame (i.e. the third image frame set).

[0039] Step S103, processing the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame.

[0040] Specifically, in order to reduce the influence of the background on the recognition of the video gesture, the second image frame set and the preset gesture picture (i.e. the target image frame) of the pre-recorded gesture video need to be processed to obtain a processed second image frame set and a processed target image frame.

[0041] Step S104, determining a fourth image frame set according to the processed second image frame set and the processed target image frame.

[0042] In order to identify whether the gestures in the target video and the gestures in the pre-recorded gesture video are consistent, in the embodiment, the processed target image frames are used to process the processed second image frame set to obtain a fourth image frame set, so as to subsequently compare the gestures in the target video and the gestures in the pre-recorded gesture video and obtain a video recognition result.

[0043] In step S105, the image frames in the third image frame set and the image frames in the first image frame set are input into the recurrent neural network for similarity comparison, and the similarity of the image frames in the third image frame set and the image frames in the first image frame set is output to obtain a first similarity set.

[0044] In the prior art, the image frame set converted from the target video and the target image frame preset by the user are compared to obtain a recognition result. This method only uses a single target image frame for comparison. If the target image frame is blurred, has noise or has low resolution, the recognition result may be affected by the target image frame, and the accuracy of the video gesture recognition result is reduced.

[0045] In order to avoid the influence of comparing only a single picture on the video gesture recognition result, in the embodiment, the pre-trained recurrent convolutional neural network is used to calculate the time sequence-based similarity between the first image frame set and the third image frame set to obtain a plurality of similarities (i.e., the first similarity set). The time sequence-based similarity calculation is to calculate the similarity of two image frame sets, i.e., a time sequence number is assigned to each image frame according to the time sequence in which the image frame appears in the video, and then the same time sequence number is selected from the two image frame sets to input into the recurrent convolutional neural network to calculate the similarity between the two image frames, and the first similarity set is obtained.

[0046] For example, the image frames from the 3rd second to the 5th second in the first image frame set and the third image frame set are extracted, wherein three image frames are extracted from the 3rd second, the 4th second and the 5th second in the first image frame set and are named as A1, A2 and A3, and three image frames are extracted from the 3rd second, the 4th second and the 5th second in the third image frame set and are named as B1, B2 and B3. Then, A1 and B1 are input into the pre-trained recurrent convolutional neural network to calculate the similarity C1 between the A1 image frame and the B1 image frame, and then the similarity C2 between the A2 image frame and the B2 image frame and the similarity C3 between the A3 image frame and the B3 image frame are calculated. The first similarity set is composed of the similarity C1, the similarity C2 and the similarity C3.

[0047] In step S106, the gestures in the target video are identified according to the fourth image frame set and the first similarity set.

[0048] In order to obtain more accurate gesture recognition results, in the embodiment, the fourth image frame set and the first similarity set calculated by the recurrent convolutional network are processed respectively to obtain the gesture recognition result of the target video.

[0049] In summary, the video gesture recognition method based on deep learning provided by the embodiment of the application determines the first image frame set and the target image frame from the pre-recorded gesture video of the target object, wherein each image in the first image frame set is a gesture image, and the target image frame is an image frame representing a gesture determined by the target object; obtains a target video, and determines a second image frame set and a third image frame set from the target video, wherein the similarity of the image frames in the second image frame set to the target image frame is higher than the similarity of the image frames in the third image frame set to the target image frame, and the target video is a video to be subjected to gesture recognition; processes the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame; determines a fourth image frame set according to the processed second image frame set and the processed target image frame; inputs the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network for similarity comparison, outputs the similarity of the image frames in the third image frame set and the image frames in the first image frame set, and obtains a first similarity set; determines the gesture recognition result of the target video according to the fourth image frame set and the first similarity set, and solves the problem of low accuracy of gesture recognition results caused by background interference when recognizing gestures from a video. By processing the image frames in the pre-recorded gesture video and the to-be-recognized video before calculating the similarity between the pre-recorded gesture video and the target video (to-be-recognized video), the interference of the background in the image frames is reduced, then the similarity is compared, and finally the recognition result is obtained, which avoids the influence of complex background interference on the recognition result in video recognition, and thus improves the recognition accuracy of the recognized video gesture.

[0050] Optionally, in the deep learning-based video gesture recognition method provided in the embodiments of the present application, the first image frame set and the target image frame are determined from the pre-recorded gesture video of the target object, including: obtaining a target image frame set by the target object for the pre-recorded gesture video; extracting each image frame in the pre-recorded gesture video of the target object to obtain a fifth image frame set; processing the fifth image frame set by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set, wherein the similarity between the image frames in the sixth image frame set and the target image frame is higher than the similarity between the image frames in the seventh image frame set and the target image frame; filtering out the image frames containing hands from the seventh image frame set by using a pre-trained hand detection model, adding the image frames containing hands in the seventh image frame set to the sixth image frame set, and taking the added sixth image frame set as the first image frame set.

[0051] In order to reduce the influence of complex background in the pre-recorded gesture video, the image frames containing only background information can be reduced in the image frames used to calculate the similarity of the image frames, that is, the image frames not containing hand features are removed from the image frame set. In the present embodiment, the pre-processing process of the pre-recorded gesture video is as shown in Figure 2 As shown in the figure, in step S10, the pre-recorded gesture video of the user (i.e. the target object) is pre-processed; in step S11, the pre-recorded gesture video is converted into a plurality of image frames to obtain a fifth image frame set, the fifth image frame set is divided into two image frame sets by using a clustering algorithm to obtain a gesture frame set FS (i.e. a sixth image frame set) and a process frame set F (i.e. a seventh image frame set); in step S12, then a pre-trained hand detection model is used to filter out a plurality of image frames with hand features from the seventh image frame set and add them to the sixth image frame set to obtain an image frame set FP, and the image frame set FP is determined as the first image frame set. While obtaining the first image frame set, the gesture image frame representing the gesture in the pre-recorded gesture video pre-set by the user is also obtained, and the gesture image frame is determined as the target image frame.

[0052] By using the pre-trained hand detection model to process the fifth image frame set obtained by converting the pre-recorded gesture video, the first image frame set not containing background information is obtained, which realizes the effect of reducing the influence of background interference on the accuracy of the recognition result, and further achieves the effect of improving the accuracy of video recognition gesture.

[0053] Optionally, in the method for recognizing video gestures based on deep learning provided in the embodiments of the present application, the target video is obtained, and the second image frame set and the third image frame set are determined from the target video, which includes: extracting each image frame in the target video to obtain an eighth image frame set; calculating the second similarity between each image frame in the eighth image frame set and the target image frame by using the first pre-trained convolutional neural network to obtain a second similarity set; obtaining the image frame set with the second similarity greater than or equal to the first threshold value to obtain the second image frame set; and obtaining the image frame set with the second similarity less than the first threshold value to obtain the third image frame set.

[0054] In order to analyze the target video more comprehensively, in the embodiments, the target video is converted into the second image frame set and the third image frame set with different similarities to the target image frame. First, the image frames of the target video are extracted to obtain the eighth image frame set; then the similarity set (i.e., the second similarity set) between each image frame in the eighth image frame set and the target image frame is calculated by using the pre-trained convolutional neural network; finally, the image frames with the second similarity greater than or equal to a certain threshold value (i.e., the first threshold value) are selected from the eighth image frame set to form the second image frame set, and the other image frames in the eighth image frame set except the second image frame set form the third image frame set. By processing the target video, two image frame sets with different similarities to the target image frame are obtained, which can analyze the target video in more detail and make the recognition result of recognizing video gestures more accurate and reasonable.

[0055] Optionally, in the method for recognizing video gestures based on deep learning provided in the embodiments of the present application, the image frames in the second image frame set and the target image frame are processed to obtain the processed second image frame set and the processed target image frame, which includes: identifying the first area range occupied by the hand of the target object in the image frames of the second image frame set and the second area range occupied by the hand of the target object in the target image frame by using the second pre-trained convolutional neural network; covering the first area range and the second area range with the first color, and covering the remaining area range of the image frames of the second image frame set except the first area range and the remaining area range of the target image frame except the second area range with the second color to obtain the processed second image frame set and the processed target image frame.

[0056] In the above scheme, the number of image frames containing only background is reduced from the image frame set converted from the pre-recorded gesture video by the hand detection model. Further, the image frame set converted from the target video (i.e., the second image frame set) can also be processed, i.e., in the second image frame set, the area range other than the hand (i.e., the area range where the background is located) is covered with a color, which greatly weakens the cluttered background information in the image frame, and thus reduces the impact of background interference on the recognition result of the gesture video.

[0057] To process the second image frame set, first, the area range occupied by the hand feature in the second image frame set (i.e., the first area range) and the area range occupied by the hand feature in the target image frame (i.e., the second area range) are identified by the pre-trained convolutional neural network. Then, the first area range and the second area range are covered with a color (i.e., the first color), and the remaining area range in the image frame of the second image frame set other than the first area range and the remaining area range in the target image frame other than the second area range are covered with another color (i.e., the second color), to obtain the processed second image frame set and the processed target image frame. For example, the first area range and the second area range are covered with white, and the remaining area range in the image frame of the second image frame set other than the first area range and the remaining area range in the target image frame other than the second area range are covered with black, to obtain the processed second image frame set and the processed target image frame.

[0058] Optionally, in the deep learning-based video gesture recognition method provided in the embodiments of the present application, determining the gesture recognition result of the target video according to the fourth image frame set and the first similarity set comprises: processing the values in the first similarity set to obtain a second similarity; calculating the sum of the number of image frames in the second image frame set and the number of image frames in the third image frame set to obtain a total number of image frames; calculating the ratio of the number of image frames in the fourth image frame set to the total number of image frames; if it is determined that the ratio is greater than or equal to a second threshold value and the second similarity is greater than or equal to a third threshold value, it is determined that the gesture recognition result of the target video is that the gesture in the target video is the gesture in the pre-recorded gesture video of the target object.

[0059] In order to obtain a more accurate identification result of the target video, in the embodiment, two judgments need to be made according to the number of image frames in the fourth image frame set and the value of the first similarity, and only when both judgment conditions are met can it be considered that the gesture in the target video is consistent with the gesture in the pre-recorded gesture video: first, a plurality of similarity values in the first similarity set are calculated to obtain a similarity value (i.e., a second similarity), wherein the plurality of similarity values can be calculated to obtain the second similarity according to the actual needs of the video, for example, the mean of the plurality of similarity values in the first similarity set can be calculated to obtain the second similarity, in addition, the plurality of similarity values in the first similarity set can be assigned weights, and the weighted mean can be calculated as the second similarity, or other methods can be used to calculate the second similarity; then the total number of image frames extracted from the target video is calculated, that is, the sum of the number of image frames in the second image frame set and the number of image frames in the third image frame set; then the ratio of the number of image frames in the fourth image frame set to the total number of image frames is calculated; finally, if the ratio is greater than a preset second threshold value and the second similarity is greater than a preset third threshold value, it is considered that the gesture in the target video is consistent with the gesture in the pre-recorded gesture video.

[0060] Optionally, in the deep learning-based video gesture recognition method provided in the embodiment of the application, the fourth image frame set is determined according to the processed second image frame set and the processed target image frame, comprising: inputting the processed second image frame set and the processed target image frame into the first convolutional neural network to output a third similarity between the image frames in the processed second image frame set and the processed target image frame; and selecting, from the processed second image frame set, image frames having a third similarity greater than a fourth threshold value with the processed target image frame to obtain the fourth image frame set.

[0061] In the embodiment, first, the third similarity between the image frames in the processed second image frame set and the processed target image frame is calculated by the convolutional neural network; then, the image frames having a third similarity greater than a preset fourth threshold value are selected from the processed second image frame set to obtain the fourth image frame set. By processing the image frames after removing the background information, the fourth image frame set with high similarity to the target image frame is obtained from the second image frame set, so as to subsequently judge whether the gesture of the target video is consistent with the gesture of the pre-recorded gesture video through the fourth image frame set, thereby achieving the effect of reducing the influence of background interference and making the identification result of recognizing the video gesture more accurate.

[0062] Optionally, in the deep learning-based video gesture recognition method provided in the embodiments of the present application, the fifth image frame set is processed by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set. Specifically, the fifth image frame set is clustered by using the clustering algorithm to obtain a ninth image frame set and a tenth image frame set, wherein the ninth image frame set and the tenth image frame set contain different pictures; the ninth image frame set, the tenth image frame set and the target image frame are subjected to histogram analysis to determine the average similarity between the image frames in the ninth image frame set and the target image frame and the average similarity between the image frames in the tenth image frame set and the target image frame; if the average similarity between the image frames in the ninth image frame set and the target image frame is higher than the average similarity between the image frames in the tenth image frame set and the target image frame, the ninth image frame set is taken as the sixth image frame set and the tenth image frame set is taken as the seventh image frame set; if the average similarity between the image frames in the ninth image frame set and the target image frame is lower than the average similarity between the image frames in the tenth image frame set and the target image frame, the tenth image frame set is taken as the sixth image frame set and the ninth image frame set is taken as the seventh image frame set.

[0063] In order to reduce the influence of the background in the pre-recorded gesture video, in the embodiments, the fifth image frame set is divided into two image frame sets. First, the fifth image frame set is clustered by using a clustering algorithm to obtain two image frame sets (i.e., the ninth image frame set and the tenth image frame set); then, the ninth image frame set and the tenth image frame set are subjected to histogram analysis to calculate the average similarity between the ninth image frame set and the target image frame and the average similarity between the tenth image frame set and the target image frame; then, the average similarity between the ninth image frame set and the target image frame and the average similarity between the tenth image frame set and the target image frame are compared, the image frame set with the higher average similarity is determined as the sixth image frame set, and the other image frame set with the lower average similarity is determined as the seventh image frame set.

[0064] In summary, the method for recognizing gestures in a video based on deep learning provided by the embodiments of the present application determines a first image frame set and a target image frame from a pre-recorded gesture video of a target object, wherein each image frame in the first image frame set is an image with a gesture, and the target image frame is an image frame representing a gesture determined by the target object; acquires a target video, and determines a second image frame set and a third image frame set from the target video, wherein the similarity between the image frames in the second image frame set and the target image frame is higher than the similarity between the image frames in the third image frame set and the target image frame, and the target video is a video to be subjected to gesture recognition; processes the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame; determines a fourth image frame set based on the processed second image frame set and the processed target image frame; inputs the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network for similarity comparison, and outputs the similarity between the image frames in the third image frame set and the image frames in the first image frame set to obtain a first similarity set; and determines a gesture recognition result for the target video based on the fourth image frame set and the first similarity set, thereby solving the problem of low accuracy of gesture recognition results caused by background interference when recognizing gestures from a video. The image frames in the pre-recorded gesture video and the video to be recognized (i.e., the target video) are processed before the similarity between the pre-recorded gesture video and the target video is calculated, so as to reduce the interference of the background in the image frames, and then the similarity is compared, and finally the recognition result is obtained, thereby avoiding the influence of complex background interference on the recognition result in video recognition, and further achieving the effect of improving the recognition accuracy of gestures in a video.

[0065] Optionally, in the present embodiment, the processing process of the video to be recognized (i.e., the target video) of the present solution can be as shown in Figure 3 Step S21: Acquire a target recognition; Step S22: Input the plurality of image frames obtained by converting the target video and the target image frame into a convolutional neural network for calculation to obtain an image frame set FH with high similarity between the image frames in the target video and the target image frame and an image frame set FL with low similarity between the image frames in the target video and the target image frame, and then input the image frame set FL and the process frame set FP obtained by pre-processing the pre-recorded gesture video into a recurrent neural network to calculate the similarity, thereby obtaining a first similarity set; Step S23: Process the first similarity set to obtain a second similarity, and determine whether the gestures in the target video and the gestures in the pre-recorded gesture video are consistent according to the second similarity, thereby obtaining a recognition result of the gesture video.

[0066] Optionally, in the present embodiment, the process of determining whether the gestures in the target video and the gestures in the pre-recorded gesture video are consistent of the present solution can be as shown in Figure 4As shown, in step S31, a pre-recorded gesture video and a target video are acquired; the pre-recorded gesture video is processed to obtain a process frame set FP; in step S32, the target video is input into a convolutional neural network frame by frame and a target image frame preset by a user to obtain an image frame set FH in which image frames in the target video have high similarity with the target image frame and an image frame set FL in which image frames in the target video have low similarity with the target image frame; in step S33, the trained convolutional neural network is used to locate a region range in which hands are contained in the image frame set FH and the target image frame, an OpenCV function is used to cover the region range in which the hands are located with white and cover the region range other than the hands with black, the processed image frame set FH and the processed target image frame are input into the convolutional neural network to calculate their similarity, and an image frame set FH' in which the similarity is greater than a fourth threshold value is obtained; in step S34, the image frame set FL and the process frame set FP are input into a recurrent convolutional neural network to calculate a time sequence-based similarity; in step S35, finally, a judgment is made according to the similarity of the image frame set FH' and the image frame set FL with the process frame set FP, if the similarity of the image frame set FL with the process frame set FP is greater than or equal to a third threshold value and a ratio of the number of image frames in the image frame set FH' to the number of image frames in an image frame set converted from the target video is greater than or equal to a second threshold value, a recognition result that the gesture of the target video is consistent with the gesture of the pre-recorded gesture video is output.

[0067] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0068] The embodiment of the present application also provides a video gesture recognition device based on deep learning. It should be noted that the video gesture recognition device based on deep learning of the embodiment of the present application can be used to execute the method for video gesture recognition based on deep learning provided by the embodiment of the present application. The video gesture recognition device based on deep learning provided by the embodiment of the present application is introduced as follows.

[0069] Figure 5 is a schematic diagram of the video gesture recognition device based on deep learning according to the embodiment of the present application. As shown in the figure, the device comprises a first determination unit 501, a second determination unit 502, a first processing unit 503, a third determination unit 504, a first acquisition unit 505 and a fourth determination unit 506. Figure 5

[0070] ​Specifically, the first determining unit 501 is configured to determine a first image frame set and a target image frame from a gesture video pre-recorded by the target object, wherein each image frame in the first image frame set is a gesture image, and the target image frame is a gesture-representing image frame determined by the target object.

[0071] The second determining unit 502 is configured to obtain a target video, and determine a second image frame set and a third image frame set from the target video, wherein the similarity between the image frames in the second image frame set and the target image frame is higher than the similarity between the image frames in the third image frame set and the target image frame, and the target video is a video to be subjected to gesture recognition.

[0072] The first processing unit 503 is configured to process the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame.

[0073] The third determining unit 504 is configured to determine a fourth image frame set according to the processed second image frame set and the processed target image frame.

[0074] The first obtaining unit 505 is configured to input the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network to compare the similarity between the image frames in the third image frame set and the image frames in the first image frame set, and output a first similarity set.

[0075] The fourth determining unit 506 is configured to determine a gesture recognition result of the target video according to the fourth image frame set and the first similarity set.

[0076] The embodiment of the application provides a video gesture recognition device based on deep learning, which comprises a first determining unit 501, a second determining unit 502, a first processing unit 503, a third determining unit 504 and a first acquiring unit 505. The first determining unit 501 is used for determining a first image frame set and a target image frame from a gesture video pre-recorded by a target object, wherein each image frame in the first image frame set is an image with a gesture, and the target image frame is an image frame representing a gesture determined by the target object. The second determining unit 502 is used for acquiring a target video and determining a second image frame set and a third image frame set from the target video, wherein the similarity between the image frames in the second image frame set and the target image frame is higher than the similarity between the image frames in the third image frame set and the target image frame, and the target video is a video to be subjected to gesture recognition. The first processing unit 503 is used for processing the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame. The third determining unit 504 is used for determining a fourth image frame set according to the processed second image frame set and the processed target image frame. The first acquiring unit 505 is used for inputting the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network to compare the similarity between the image frames in the third image frame set and the image frames in the first image frame set, outputting the similarity between the image frames in the third image frame set and the image frames in the first image frame set, and obtaining a first similarity set. The fourth determining unit 506 is used for determining a gesture recognition result of the target video according to the fourth image frame set and the first similarity set, thereby solving the problem that the accuracy of the gesture recognition result is low due to background interference when gestures are recognized from a video. The image frames in the pre-recorded gesture video and the to-be-recognized video are processed before the similarity between the pre-recorded gesture video and the target video (to-be-recognized video) is calculated, so that the interference of the background in the image frames is reduced, then the similarity is compared, and finally the recognition result is obtained, thereby avoiding the influence of the complex background on the recognition result in video recognition, and the effect of improving the recognition accuracy of the recognized video gesture is achieved.

[0077] Optionally, in the deep learning-based video gesture recognition apparatus provided by the embodiment of the present application, the first determining unit 501 comprises: a first obtaining subunit configured to obtain a target image frame set for a pre-recorded gesture video of the target object; a second obtaining subunit configured to extract each image frame in the pre-recorded gesture video of the target object to obtain a fifth image frame set; a first processing subunit configured to process the fifth image frame set by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set, wherein the similarity between the image frames in the sixth image frame set and the target image frame is higher than the similarity between the image frames in the seventh image frame set and the target image frame; and a third obtaining subunit configured to filter out image frames containing hands from the seventh image frame set by using a pre-trained hand detection model, add the image frames containing hands in the seventh image frame set to the sixth image frame set, and take the added sixth image frame set as the first image frame set.

[0078] Optionally, in the deep learning-based video gesture recognition apparatus provided by the embodiment of the present application, the second determining unit 502 comprises: a fourth obtaining subunit configured to extract each image frame in the target video to obtain an eighth image frame set; a first calculating subunit configured to calculate the second similarity between each image frame in the eighth image frame set and the target image frame by using a pre-trained first convolutional neural network to obtain a second similarity set; a fifth obtaining subunit configured to obtain images with the second similarity greater than or equal to a first threshold value to obtain a second image frame set; and a sixth obtaining subunit configured to obtain images with the second similarity less than the first threshold value to obtain a third image frame set.

[0079] Optionally, in the deep learning-based video gesture recognition apparatus provided by the embodiment of the present application, the first processing unit 503 comprises: a first recognition subunit configured to recognize the first area range occupied by the hand of the target object in the image frames of the second image frame set and the second area range occupied by the hand of the target object in the target image frame by using a pre-trained second convolutional neural network; and a first covering subunit configured to cover the first area range and the second area range with a first color, cover the remaining area range in the image frames of the second image frame set except the first area range and the remaining area range in the target image frame except the second area range with a second color, and obtain a processed second image frame set and a processed target image frame.

[0080] Optionally, in the deep learning-based video gesture recognition apparatus provided by the embodiment of the present application, the fourth determination unit 506 comprises: a second processing subunit, configured to process the values in the first similarity set to obtain a second similarity; a second calculation subunit, configured to calculate the sum of the number of image frames in the second image frame set and the number of image frames in the third image frame set to obtain a total number of image frames; a third calculation subunit, configured to calculate the ratio of the number of image frames in the fourth image frame set to the total number of image frames; and a first determination subunit, configured to determine that the gesture recognition result of the target video is that the gesture in the target video is the gesture in the pre-recorded gesture video of the target object, if it is determined that the ratio is greater than or equal to the second threshold value and the second similarity is greater than or equal to the third threshold value.

[0081] Optionally, in the deep learning-based video gesture recognition apparatus provided by the embodiment of the present application, the third determination unit 504 comprises: a first output subunit, configured to input the processed second image frame set and the processed target image frame into the first convolutional neural network to output a third similarity between the image frames in the processed second image frame set and the processed target image frame; and a first screening subunit, configured to screen the image frames with the third similarity greater than the fourth threshold value from the processed second image frame set to obtain the fourth image frame set.

[0082] Optionally, in the deep learning-based video gesture recognition apparatus provided by the embodiment of the present application, the first processing subunit comprises: a processing module, configured to use a clustering algorithm to perform clustering processing on the fifth image frame set to obtain a ninth image frame set and a tenth image frame set, wherein the ninth image frame set and the tenth image frame set contain different pictures; a first determination module, configured to perform histogram analysis on the ninth image frame set, the tenth image frame set and the target image frame to determine the average similarity between the image frames in the ninth image frame set and the target image frame and the average similarity between the image frames in the tenth image frame set and the target image frame; a second determination module, configured to determine the ninth image frame set as the sixth image frame set and the tenth image frame set as the seventh image frame set if the average similarity between the image frames in the ninth image frame set and the target image frame is higher than the average similarity between the image frames in the tenth image frame set and the target image frame; and a third determination module, configured to determine the tenth image frame set as the sixth image frame set and the ninth image frame set as the seventh image frame set if the average similarity between the image frames in the ninth image frame set and the target image frame is lower than the average similarity between the image frames in the tenth image frame set and the target image frame.

[0083] The deep learning-based video gesture recognition device includes a processor and a memory. The first determining unit 501, the second determining unit 502, the first processing unit 503, the third determining unit 504, the first acquiring unit 505, and the fourth determining unit 506 are all stored in the memory as program units. The processor executes the program units stored in the memory to achieve the corresponding functions.

[0084] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the accuracy of video gesture recognition.

[0085] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0086] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements a deep learning-based video gesture recognition method.

[0087] This invention provides a processor for running a program, wherein the program executes a deep learning-based video gesture recognition method during runtime.

[0088] like Figure 5 As shown, this embodiment of the invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: determining a first set of image frames and a target image frame from a pre-recorded gesture video of a target object, wherein each frame in the first set of image frames is an image containing a gesture, and the target image frame is an image frame representing the gesture determined by the target object; acquiring a target video, and determining a second set of image frames and a third set of image frames from the target video, wherein the similarity between image frames in the second set and the target image frame is higher than that between image frames in the third set and the target image frame. Similarity is calculated, where the target video is the video for which gesture recognition is to be performed. Image frames in the second image frame set and the target image frame are processed to obtain a processed second image frame set and a processed target image frame. Based on the processed second image frame set and the processed target image frame, a fourth image frame set is determined. Image frames in the third image frame set and image frames in the first image frame set are input into a recurrent neural network for similarity comparison, and the similarity between the image frames in the third image frame set and the image frames in the first image frame set is output to obtain a first similarity set. Based on the fourth image frame set and the first similarity set, the gesture recognition result for the target video is determined.

[0089] The processor, when executing the program, further implements the following steps: determining the first image frame set and the target image frame from the pre-recorded gesture video of the target object includes: obtaining the target image frame set by the target object for the pre-recorded gesture video; extracting each image frame in the pre-recorded gesture video of the target object to obtain a fifth image frame set; processing the fifth image frame set by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set, wherein the similarity between the image frames in the sixth image frame set and the target image frame is higher than the similarity between the image frames in the seventh image frame set and the target image frame; filtering out the image frames containing hands from the seventh image frame set by using a pre-trained hand detection model, and adding the image frames containing hands in the seventh image frame set to the sixth image frame set, and taking the added sixth image frame set as the first image frame set.

[0090] The processor, when executing the program, further implements the following steps: obtaining the target video, and determining the second image frame set and the third image frame set from the target video includes: extracting each image frame in the target video to obtain an eighth image frame set; calculating the second similarity between each image frame in the eighth image frame set and the target image frame by using a pre-trained first convolutional neural network to obtain a second similarity set; obtaining the image frames with the second similarity greater than or equal to a first threshold value to obtain the second image frame set; and obtaining the image frames with the second similarity less than the first threshold value to obtain the third image frame set.

[0091] The processor, when executing the program, further implements the following steps: processing the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame includes: identifying the first area range occupied by the hand of the target object in the image frames in the second image frame set and the second area range occupied by the hand of the target object in the target image frame by using a pre-trained second convolutional neural network; covering the first area range and the second area range by using a first color, and covering the remaining area range in the image frames in the second image frame set except the first area range and the remaining area range in the target image frame except the second area range by using a second color, to obtain the processed second image frame set and the processed target image frame.

[0092] The processor further implements the following steps when executing the program: determining the gesture recognition result of the target video according to the fourth image frame set and the first similarity set, including: processing the values in the first similarity set to obtain a second similarity; calculating the sum of the number of image frames in the second image frame set and the number of image frames in the third image frame set to obtain a total number of image frames; calculating the ratio of the number of image frames in the fourth image frame set to the total number of image frames; if it is determined that the ratio is greater than or equal to a second threshold value, and it is determined that the second similarity is greater than or equal to a third threshold value, then it is determined that the gesture recognition result of the target video is that the gesture in the target video is the gesture in the pre-recorded gesture video of the target object.

[0093] The processor further implements the following steps when executing the program: determining the fourth image frame set according to the processed second image frame set and the processed target image frame, including: inputting the processed second image frame set and the processed target image frame into the first convolutional neural network to output a third similarity between the image frames in the processed second image frame set and the processed target image frame; selecting the image frames with the third similarity greater than a fourth threshold value from the processed second image frame set to obtain the fourth image frame set.

[0094] The processor further implements the following steps when executing the program: processing the fifth image frame set using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set, including: using the clustering algorithm to cluster the fifth image frame set to obtain a ninth image frame set and a tenth image frame set, wherein the ninth image frame set and the tenth image frame set contain different pictures; performing histogram analysis on the ninth image frame set, the tenth image frame set, and the target image frame to determine the average similarity between the image frames in the ninth image frame set and the target image frame, and the average similarity between the image frames in the tenth image frame set and the target image frame; if the average similarity between the image frames in the ninth image frame set and the target image frame is higher than the average similarity between the image frames in the tenth image frame set and the target image frame, then the ninth image frame set is taken as the sixth image frame set, and the tenth image frame set is taken as the seventh image frame set; if the average similarity between the image frames in the ninth image frame set and the target image frame is lower than the average similarity between the image frames in the tenth image frame set and the target image frame, then the tenth image frame set is taken as the sixth image frame set, and the ninth image frame set is taken as the seventh image frame set.

[0095] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0096] The application also provides a computer program product, which is adapted to execute the program steps of initializing the following method when executed on a data processing device: determining a first image frame set and a target image frame from a pre-recorded gesture video of a target object, wherein each image frame in the first image frame set is a gesture image, and the target image frame is a gesture-representing image frame determined by the target object; obtaining a target video and determining a second image frame set and a third image frame set from the target video, wherein the similarity of the image frames in the second image frame set to the target image frame is higher than the similarity of the image frames in the third image frame set to the target image frame, and the target video is a video to be subjected to gesture recognition; processing the image frames in the second image frame set and the target image frame to obtain a processed second image frame set and a processed target image frame; determining a fourth image frame set according to the processed second image frame set and the processed target image frame; inputting the image frames in the third image frame set and the image frames in the first image frame set into a recurrent neural network for similarity comparison, outputting the similarity of the image frames in the third image frame set and the image frames in the first image frame set, and obtaining a first similarity set; and determining a gesture recognition result of the target video according to the fourth image frame set and the first similarity set.

[0097] When executed on a data processing device, the program is also adapted to execute the program steps of initializing the following method: determining a first image frame set and a target image frame from a pre-recorded gesture video of a target object includes: obtaining a target image frame set by the target object for the pre-recorded gesture video; extracting each image frame in the pre-recorded gesture video of the target object to obtain a fifth image frame set; processing the fifth image frame set by using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set, wherein the similarity between the image frames in the sixth image frame set and the target image frame is higher than the similarity between the image frames in the seventh image frame set and the target image frame; filtering out image frames containing hands from the seventh image frame set by using a pre-trained hand detection model, adding the image frames containing hands in the seventh image frame set to the sixth image frame set, and taking the added sixth image frame set as the first image frame set.

[0098] When executed on a data processing device, the program is also adapted to execute the program steps of initializing the following method: obtaining a target video and determining a second image frame set and a third image frame set from the target video includes: extracting each image frame in the target video to obtain an eighth image frame set; calculating the second similarity of each image frame in the eighth image frame set and the target image frame by using a pre-trained first convolutional neural network to obtain a second similarity set; obtaining images with a second similarity greater than or equal to a first threshold value to obtain the second image frame set; and obtaining images with a second similarity less than the first threshold value to obtain the third image frame set.

[0099] When executed on the data processing device, the program is further adapted to execute the steps of the method to process the image frames in the second image frame set and the target image frame to obtain the processed second image frame set and the processed target image frame, comprising: identifying, by the pre-trained second convolutional neural network, a first region range occupied by the hand of the target object in the image frames in the second image frame set and a second region range occupied by the hand of the target object in the target image frame; covering the first region range and the second region range with a first color, covering the remaining region ranges in the image frames in the second image frame set except the first region range and the remaining region ranges in the target image frame except the second region range with a second color, to obtain the processed second image frame set and the processed target image frame.

[0100] When executed on the data processing device, the program is further adapted to execute the steps of the method to determine the gesture recognition result of the target video according to the fourth image frame set and the first similarity set, comprising: processing the values in the first similarity set to obtain a second similarity; calculating the sum of the number of image frames in the second image frame set and the number of image frames in the third image frame set to obtain a total number of image frames; calculating the ratio of the number of image frames in the fourth image frame set to the total number of image frames; if it is determined that the ratio is greater than or equal to a second threshold value and the second similarity is greater than or equal to a third threshold value, it is determined that the gesture recognition result of the target video is that the gesture in the target video is the gesture in the pre-recorded gesture video of the target object.

[0101] When executed on the data processing device, the program is further adapted to execute the steps of the method to determine the fourth image frame set according to the processed second image frame set and the processed target image frame, comprising: inputting the processed second image frame set and the processed target image frame into the first convolutional neural network to output a third similarity between the image frames in the processed second image frame set and the processed target image frame; selecting, from the processed second image frame set, image frames with a third similarity greater than a fourth threshold value with the processed target image frame to obtain the fourth image frame set.

[0102] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: processing the fifth image frame set using a clustering algorithm to obtain the sixth and seventh image frame sets, including: using a clustering algorithm to cluster the fifth image frame set to obtain the ninth and tenth image frame sets, wherein the ninth and tenth image frame sets contain different images; performing histogram analysis on the ninth, tenth, and target image frame sets to determine the mean similarity between the image frames in the ninth image frame set and the target image frame, and the mean similarity between the image frames in the tenth image frame set and the target image frame. The average similarity between the image frames in the ninth image frame set and the target image frame; if the average similarity between the image frames in the ninth image frame set and the target image frame is higher than the average similarity between the image frames in the tenth image frame set and the target image frame, then the ninth image frame set is taken as the sixth image frame set, and the tenth image frame set is taken as the seventh image frame set; if the average similarity between the image frames in the ninth image frame set and the target image frame is lower than the average similarity between the image frames in the tenth image frame set and the target image frame, then the tenth image frame set is taken as the sixth image frame set, and the ninth image frame set is taken as the seventh image frame set.

[0103] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a processFigure 1 one or more processes and / or blocks Figure 1 the function(s) specified in the block or blocks.

[0106] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more processes and / or blocks Figure 1 the function(s) specified in the block or blocks.

[0107] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0108] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information about an operating system, application software, and / or the like. Memory is an example of computer readable media.

[0109] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0111] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code thereon for use by a computer or computer system.

[0112] The foregoing is merely illustrative of the embodiments of the present application and is not used to limit the present application. The present application can have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A video gesture recognition method based on deep learning, characterized in that, include: From the pre-recorded gesture video of the target object, a first image frame set and a target image frame are determined, wherein each image in the first image frame set is an image with a gesture, and the target image frame is an image frame representing the gesture determined by the target object. A target video is acquired, and a second set of image frames and a third set of image frames are determined from the target video, wherein the similarity between the image frames in the second set of image frames and the target image frames is higher than the similarity between the image frames in the third set of image frames and the target image frames, wherein the target video is the video to be used for gesture recognition; The image frames in the second image frame set and the target image frame are processed to obtain a processed second image frame set and a processed target image frame; wherein, the processing of the image frames in the second image frame set and the target image frame includes using a second convolutional neural network to identify the hand region in the second image frame set and the target image frame; The fourth image frame set is determined based on the processed second image frame set and the processed target image frame; Image frames in the third image frame set and image frames in the first image frame set are input into a recurrent neural network for similarity comparison, and the similarity between the image frames in the third image frame set and the image frames in the first image frame set is output to obtain the first similarity set; Based on the fourth image frame set and the first similarity set, the gesture recognition result for the target video is determined; The fourth image frame set is determined based on the processed second image frame set and the processed target image frame, including: The processed second image frame set and the processed target image frame are input into the first convolutional neural network, and the third similarity between the image frames in the processed second image frame set and the processed target image frame is output. From the processed second image frame set, select image frames whose third similarity to the processed target image frame is greater than the fourth threshold to obtain the fourth image frame set.

2. The method according to claim 1, characterized in that, From the pre-recorded gesture video of the target object, the first set of image frames and the target image frames are determined to include: Obtain the target image frame set for the pre-recorded gesture video; Extract each image frame from the pre-recorded gesture video of the target object to obtain a fifth image frame set; The fifth image frame set is processed using a clustering algorithm to obtain a sixth image frame set and a seventh image frame set. The similarity between the image frames in the sixth image frame set and the target image frame is higher than the similarity between the image frames in the seventh image frame set and the target image frame. Using a pre-trained hand detection model, image frames containing hands are selected from the seventh image frame set. These image frames are then added to the sixth image frame set, and the sixth image frame set is used as the first image frame set.

3. The method according to claim 1, characterized in that, Acquiring the target video and determining the second set of image frames and the third set of image frames from the target video includes: Extract each image frame from the target video to obtain the eighth image frame set; The second similarity between each image frame in the eighth image frame set and the target image frame is calculated using a pre-trained first convolutional neural network to obtain a second similarity set. Obtain images whose second similarity is greater than or equal to the first threshold to obtain the second image frame set; Images with a similarity less than the first threshold are obtained to form the third set of image frames.

4. The method according to claim 1, characterized in that, Processing the image frames in the second image frame set and the target image frame yields the processed second image frame set and the processed target image frame, including: The pre-trained second convolutional neural network identifies the first region occupied by the hand of the target object in the image frames of the second image frame set and the second region occupied by the hand of the target object in the target image frame. The first region and the second region are covered with a first color, and the remaining regions of the image frames in the second image frame set except for the first region and the remaining regions of the target image frame except for the second region are covered with a second color, to obtain the processed second image frame set and the processed target image frame.

5. The method according to claim 1, characterized in that, Based on the fourth image frame set and the first similarity set, the gesture recognition result for the target video is determined as follows: The numerical values ​​in the first similarity set are processed to obtain the second similarity; wherein the processing method includes at least calculating the mean and weighted mean; The total number of image frames is obtained by summing the number of image frames in the second image frame set and the number of image frames in the third image frame set. Calculate the ratio of the number of image frames in the fourth image frame set to the total number of image frames; If it is determined that the ratio is greater than or equal to the second threshold, and the second similarity is greater than or equal to the third threshold, then the gesture recognition result of the target video is determined to be the gesture in the target video as the gesture in the pre-recorded gesture video of the target object.

6. The method according to claim 2, characterized in that, The fifth image frame set is processed using a clustering algorithm to obtain the sixth and seventh image frame sets, including: The fifth image frame set is clustered using a clustering algorithm to obtain the ninth image frame set and the tenth image frame set, wherein the ninth image frame set and the tenth image frame set contain different images; Histogram analysis is performed on the ninth image frame set, the tenth image frame set, and the target image frame to determine the mean similarity between the image frames in the ninth image frame set and the target image frame, and the mean similarity between the image frames in the tenth image frame set and the target image frame. If the average similarity between the image frames in the ninth image frame set and the target image frame is higher than the average similarity between the image frames in the tenth image frame set and the target image frame, then the ninth image frame set is designated as the sixth image frame set, and the tenth image frame set is designated as the seventh image frame set. If the average similarity between the image frames in the ninth image frame set and the target image frame is lower than the average similarity between the image frames in the tenth image frame set and the target image frame, then the tenth image frame set is designated as the sixth image frame set, and the ninth image frame set is designated as the seventh image frame set.

7. A video gesture recognition device based on deep learning, characterized in that, include: The first determining unit is used to determine a first image frame set and a target image frame from a pre-recorded gesture video of the target object, wherein each image in the first image frame set is an image with a gesture, and the target image frame is an image frame representing the gesture determined by the target object. The second determining unit is used to acquire a target video and determine a second set of image frames and a third set of image frames from the target video, wherein the similarity between the image frames in the second set of image frames and the target image frames is higher than the similarity between the image frames in the third set of image frames and the target image frames, wherein the target video is a video to be used for gesture recognition. The first processing unit is configured to process the image frames in the second image frame set and the target image frame to obtain the processed second image frame set and the processed target image frame; wherein, the processing of the image frames in the second image frame set and the target image frame includes using a second convolutional neural network to identify the hand region in the second image frame set and the target image frame; The third determining unit is used to determine the fourth image frame set based on the processed second image frame set and the processed target image frame; The first acquisition unit is used to input image frames in the third image frame set and image frames in the first image frame set into a recurrent neural network for similarity comparison, and output the similarity between the image frames in the third image frame set and the image frames in the first image frame set to obtain a first similarity set. The fourth determining unit is used to determine the gesture recognition result of the target video based on the fourth image frame set and the first similarity set; The third determining unit includes: a first output subunit, used to input the processed second image frame set and the processed target image frame into the first convolutional neural network, and output the third similarity between the image frames in the processed second image frame set and the processed target image frame; and a first filtering subunit, used to filter out image frames from the processed second image frame set whose third similarity with the processed target image frame is greater than a fourth threshold, to obtain a fourth image frame set.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the deep learning-based video gesture recognition method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, The method includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the deep learning-based video gesture recognition method according to any one of claims 1 to 6.

Images