A gesture recognition method, device and equipment

Abstract

The present application provides a gesture recognition method, which relates to the field of human-computer interaction technology. The method includes: extracting M images from a first video segment in a video stream; performing gesture recognition on the M images through a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment, and fusing the gesture recognition results of N consecutive video segments including the first video segment to obtain a fused gesture recognition result. In the above recognition process, there is no need to segment and track the gestures in the video stream, but to recognize the actions of each stage through a deep learning algorithm with a faster computing speed, and then fuse the actions of each stage, so as to achieve the effect of increasing the speed of gesture recognition and reducing the delay of gesture recognition.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a gesture recognition method, device and equipment. Background Art

[0002] Gesture input is an indispensable key technology for achieving natural and direct human-computer interaction. Gesture recognition methods based on computer vision have become a hot topic of research today because they are independent of devices, provide more natural human-computer interaction effects, and provide better immersion.

[0003] In the related art, the gesture recognition scheme based on computer vision is as follows: first, the gesture image video stream is captured by a camera and the video stream is converted into image frames; then, the shape, features and position information of the gesture are extracted from the image frames by segmenting and tracking according to a specific image tracking algorithm; finally, the gesture is recognized based on the extracted shape, features and position information of the gesture combined with pre-established classification criteria.

[0004] In the related art, when extracting the shape, features and position information of a gesture from an image frame, the gesture in the image needs to be segmented and tracked. However, the segmentation and tracking process consumes a lot of processing time and causes excessive delay. Summary of the invention

[0005] In order to reduce the delay of gesture recognition, embodiments of the present application provide a gesture recognition method, apparatus and device.

[0006] In a first aspect, a gesture recognition method is provided, the method comprising: acquiring M images, the M images being extracted from a first video segment in a video stream, wherein the first video segment is any video segment in the video stream, and M is an integer greater than or equal to 2; performing gesture recognition on the M images through a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment; after obtaining gesture recognition results of N consecutive video segments including the first video segment in the video stream, fusing the gesture recognition results of the N consecutive video segments to obtain a fused gesture recognition result; N≥2, and N is an integer.

[0007] In the above gesture recognition method, for each video segment in the video stream, M images in the video segment are obtained, and gesture recognition is performed on the M images through a deep learning algorithm to obtain a gesture recognition result corresponding to the video segment. Finally, the gesture recognition results of N consecutive video segments including the video segment are fused to obtain the gesture recognition result of the N consecutive video segments. That is, in the above recognition process, there is no need to segment and track the gestures in the video stream, but a deep learning algorithm with a faster computing speed is used to identify the actions in each stage, and the actions in each stage are fused, so as to achieve the effect of improving the speed of gesture recognition and reducing the delay of gesture recognition.

[0008] In a possible implementation scheme, fusing the gesture recognition results of the N consecutive video segments to obtain a fused gesture recognition result includes:

[0009] The gesture recognition results of the N consecutive video segments are input into a pre-trained first machine learning model to obtain the fused gesture recognition result, the first machine learning model is used to determine the overall gesture movement trend constituted by the input N consecutive gesture recognition results, and output the gesture corresponding to the overall gesture movement trend as the fused gesture recognition result.

[0010] In actual applications, when a user performs a gesture operation, he or she may make a gesture that does not conform to the current gesture operation within a short period of time during the gesture operation. Through the above possible implementation scheme, after identifying the gesture recognition results of each video segment, the final gesture recognition result can be obtained according to the gesture movement trend indicated by the gesture recognition results of multiple consecutive video segments, thereby eliminating the influence of the user's erroneous gestures in a short period of time on the final gesture recognition result, thereby improving the accuracy of gesture recognition.

[0011] In a possible implementation, the first machine learning model is a neural network model, and the number of neurons in the neural network model is N; or, the first machine learning model is a support vector machine SVM model.

[0012] In a possible implementation scheme, fusing the gesture recognition results of the N consecutive video segments to obtain a fused gesture recognition result includes:

[0013] Obtaining preset weight coefficients corresponding to the gesture recognition results of the N consecutive video segments;

[0014] According to the weight coefficients corresponding to the gesture recognition results of the N consecutive video segments, the gesture recognition results of the N consecutive video segments are weighted averaged to obtain the fused gesture recognition result.

[0015] Through the above possible implementation scheme, after identifying the gesture recognition results of each video segment, the gesture recognition results of multiple consecutive video segments can be weighted averaged according to pre-set weights to reduce the impact of the user's erroneous gestures in a short period of time on the final gesture recognition results, thereby improving the accuracy of gesture recognition.

[0016] In a possible implementation, performing gesture recognition on the M images by using a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment includes:

[0017] Performing image processing on the M images to obtain an optical flow information image corresponding to the first video segment, the optical flow information image including optical flow information between a first image among the M images and a p-th image before the first image, the first image being any one of the M images, the optical flow information including instantaneous velocity vector information of pixels in the image, performing gesture recognition on the optical flow information image using a first deep learning algorithm to obtain a first recognition result, where p is an integer greater than or equal to 1; performing image processing on the M images to obtain a color information image corresponding to the first video segment, the color information image including color information of the M images, the color information including color values ​​of each pixel in the image, performing gesture recognition on the color information image using a second deep learning algorithm to obtain a second recognition result; and fusing the first recognition result and the second recognition result to obtain a gesture recognition result of the first video segment.

[0018] The above possible implementation scheme extracts optical flow information and color information of the video segment according to M images, performs gesture recognition respectively according to the extracted optical flow information and color information, and then fuses the gesture recognition results respectively identified, thereby improving the problem of inaccurate gestures recognized by a single deep learning algorithm, so as to improve the accuracy of gesture recognition results for the video segment.

[0019] In a possible implementation scheme, performing image processing on the M images to obtain an optical flow information image corresponding to the first video segment includes:

[0020] For the first image, obtain the p-th image before the first image in the video stream according to a preset rule; calculate the optical flow information between the first image and the p-th image, and generate an optical flow information image containing the optical flow information between the first image and the p-th image; wherein the time interval between the first image and the p-th image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image;

[0021] or,

[0022] For the first image, all p images before the first image in the video stream are obtained according to preset rules; the optical flow information between the first image and each two adjacent images in the M images is calculated, and the optical flow information between each two adjacent images is accumulated to generate an optical flow information image containing the accumulated optical flow information; wherein the time interval between the first image and the p-th image before the first image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image.

[0023] In the above possible implementation scheme, based on the currently acquired image and the p images before the current image, the optical flow information image between the current image and the p-th image before the current image can be obtained, so that gesture recognition can be performed on the optical flow information image through a deep learning algorithm later, without the need to segment and track the gestures in the image, thereby simplifying the processing of the gesture recognition results, improving the speed of gesture recognition, and reducing the delay of gesture recognition.

[0024] In the above possible implementation scheme, performing image processing on the M images to obtain a color information image corresponding to the first video segment includes:

[0025] Extracting color information of m images among the M images, generating color information images corresponding to the m images according to the extracted color information, and acquiring the color information images corresponding to the m images as color information images corresponding to the first video segment; the m images are m random images among the M images, or the m images are m images among the M images that have the largest changes relative to their respective previous images in the video stream, where m is an integer greater than or equal to 1;

[0026] Alternatively, the pixel positions of the M images whose image contents change with time are detected, the average value of the color information at the corresponding identified pixel positions in the M images is calculated, new color information at the identified pixel positions is obtained, and a color information image corresponding to the first video segment is generated based on the new color information at the identified pixel positions.

[0027] In a possible implementation, before acquiring the M images, the method further includes:

[0028] Determine a time window of a preset time length in the video stream, where the end time of the time window is within the time period corresponding to the first video segment; determine whether an action occurs in the first video segment based on the last image in the time window and at least one reference image, where the at least one reference image is any image other than the last image in the time window; if the judgment result is that an action occurs in the first video segment, execute the step of acquiring M images.

[0029] Since gesture operations inevitably involve gesture actions, through the above possible implementation scheme, before performing gesture recognition on a video segment, it is first necessary to determine whether an action has occurred in the video segment through an image in the video segment and at least one image before the image, and only when it is determined that an action has occurred, perform subsequent recognition operations, thereby reducing unnecessary recognition steps, saving computing resources, and improving recognition efficiency.

[0030] In a possible implementation, the determining, according to the last image in the time window and the at least one reference image, whether an action occurs in the first video segment includes:

[0031] For each reference image in the at least one reference image, a partial derivative image of the last image is calculated, wherein the value of each pixel in the partial derivative image is the partial derivative of the value of the corresponding pixel in the last image relative to the value of the corresponding pixel in the reference image; the values ​​of each pixel in the partial derivative image are normalized to obtain a normalized partial derivative image; according to a preset binarization threshold, the normalized partial derivative image is binarized to obtain a binarized image of the partial derivative image, wherein the value of each pixel in the binarized image is 0 or 1; the sum of the grayscale values ​​of each pixel in the binarized image is calculated; when the sum of the grayscale values ​​is greater than 0, it is determined that action occurs in the first video segment.

[0032] In a possible implementation scheme, the fusing the first recognition result and the second recognition result to obtain the gesture recognition result of the first video segment includes:

[0033] An average value is calculated for the first recognition result and the second recognition result, and a gesture recognition result of the first video segment is obtained according to a calculation result of the average value calculation; or, the first recognition result and the second recognition result are input into a pre-trained second machine learning model to obtain a gesture recognition result of the first video segment.

[0034] In a second aspect, a gesture recognition device is provided, which has the function of implementing the gesture recognition method provided by the first aspect and the possible implementation scheme of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more units corresponding to the above functions.

[0035] In a third aspect, a gesture recognition device is provided, comprising: a processor and a memory; the processor in the device executes a program or instruction stored in the memory to implement the gesture recognition method provided by the first aspect and possible implementation scheme of the first aspect.

[0036] In a fourth aspect, a computer-readable storage medium is provided, which stores an executable program, and the executable program is executed by a processor to implement the gesture recognition method provided by the first aspect and the possible implementation scheme of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is an architectural diagram of a gesture recognition system involved in this application;

[0038] Figure 2 yes Figure 1 A schematic diagram of gesture recognition related to the illustrated embodiment;

[0039] Figure 3 is a method flow chart of a gesture recognition method provided by an exemplary embodiment of the present application;

[0040] Figure 4 yes Figure 3 Schematic diagram of two time window spans involved in the illustrated embodiment;

[0041] Figure 5 yes Figure 3 The illustrated embodiment relates to a schematic diagram of fusing recognition results;

[0042] Figure 6 yes Figure 3 A schematic diagram of a flow chart of gesture recognition involved in the illustrated embodiment;

[0043] Figure 7 is a structural schematic diagram of a gesture recognition device provided by an exemplary embodiment of the present application;

[0044] Figure 8 It is a structural block diagram of a gesture recognition device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] Figure 1 1 is a system architecture diagram of a gesture recognition system involved in an embodiment of the present application. The gesture recognition system may include the following devices: an image acquisition device 110 and a gesture recognition device 120 .

[0047] The image acquisition device 110 may be a camera. For example, the image acquisition device 110 may be a single camera, or the image acquisition device 110 may be a camera module composed of two or more cameras.

[0048] The image acquisition device 110 may be fixedly arranged, or the image acquisition device 110 may be integrated with a servo motor, which may drive the image acquisition device 110 to rotate or move under the control of the gesture recognition device to change the shooting angle or shooting position of the image acquisition device 110 .

[0049] The gesture recognition device 120 may be a general-purpose computer, or the gesture recognition device may also be an embedded computing device.

[0050] The image acquisition device 110 and the gesture recognition device 120 may be independent devices, and the image acquisition device 110 and the gesture recognition device 120 are connected via a wired or wireless network.

[0051] Alternatively, the image acquisition device 110 and the gesture recognition device 120 may also be integrated into the same physical device, and the image acquisition device 110 and the gesture recognition device 120 are connected via a communication bus.

[0052] Optionally, after identifying the gesture in the video stream, the gesture recognition device 120 transmits the identified gesture to the control device 130, and the control device 130 determines a corresponding control instruction based on the identified gesture, and performs a corresponding control operation based on the determined control instruction, such as controlling the graphic display based on the control instruction, or controlling the controlled device to perform a certain operation based on the control instruction, etc.

[0053] In the embodiment of the present application, the image acquisition device 110 transmits the acquired video stream to the gesture recognition device 120, and the gesture recognition device 120 performs image analysis and gesture recognition on the video stream to instantly recognize the gestures in the video stream. Figure 2 , which shows a gesture recognition schematic diagram involved in the embodiment of the present application. Figure 2As shown, when performing gesture recognition, the gesture recognition device 120 can extract M images (where M is an integer greater than or equal to 2) from a video segment of the video stream. The gesture recognition device 120 performs gesture recognition on the M images through a deep learning algorithm to obtain a gesture recognition result corresponding to the video segment. After obtaining the gesture recognition results of N consecutive video segments including the video segment in the video stream, the gesture recognition results of the N consecutive video segments are fused to obtain a fused gesture recognition result.

[0054] That is, in the scheme shown in the embodiment of the present application, a complete gesture action is divided into multiple stage actions, each stage action is recognized by a deep learning algorithm, and finally the recognized stage actions are merged into a complete gesture action. During the recognition process, there is no need to segment and track the gestures in the video stream. Instead, a deep learning algorithm with faster computing speed is used to recognize the various stage actions, thereby achieving the effect of increasing the speed of gesture recognition and reducing the delay of gesture recognition.

[0055] Taking the above deep learning algorithm as an example, which is a dual-channel deep learning algorithm based on optical flow information and color information, please refer to Figure 3 , which is a method flow chart of a gesture recognition method provided by an exemplary embodiment of the present application. Figure 3 As shown, the gesture recognition method may include the following steps:

[0056] Step 301: for a first video segment in a video stream, determine a time window of a preset time length in the video stream, where the end time of the time window is within a time period corresponding to the first video segment.

[0057] The first video segment is any video segment in the video stream. In the embodiment of the present application, the gesture recognition device may divide the video stream into a plurality of video segments connected end to end, and perform gesture recognition on each video segment respectively.

[0058] A video stream is composed of a series of video images corresponding to different time points. In an embodiment of the present application, the above-mentioned time window can be a time window between time points corresponding to two different video images, that is, the time length between the time point corresponding to the first image of the video stream in the time window and the time point corresponding to the last image in the time window is the above-mentioned preset time length.

[0059] The last image in the time window is an image in the first video segment to be identified, and other images of the video stream in the time window may be images in the first video segment or images before the first video segment in the video stream.

[0060] For example, please refer to Figure 4, which shows two time window span schematic diagrams involved in the embodiments of the present application. Figure 4 In the example, the start time of the time window is t1 and the end time is t2, while the start time of the first video segment is t3 and the end time is t4.

[0061] like Figure 4 As shown in (a), in a possible implementation, t1 and t2 are between t3 and t4, that is, the time window is completely within the first video segment.

[0062] like Figure 4 As shown in (b), in another possible implementation, t2 is between t3 and t4, and t1 is before t3, that is, part of the time window is within the first video segment, and the other part is before the first video segment.

[0063] In addition, the above-mentioned preset time length can be preset in the gesture recognition device by a developer.

[0064] Step 302, judging whether there is any action in the first video segment according to the last image in the time window and at least one reference image; if so, proceeding to step 303, otherwise, returning to step 301 to determine the next time window of a preset time length.

[0065] The at least one reference image is any other image except the last image in the time window.

[0066] In the embodiment of the present application, the gesture recognition device determines whether an action occurs in the first video segment based on the difference between the last image of the video stream in the time window and at least one other image of the video stream in the time window.

[0067] In practical applications, the step of determining whether an action occurs in the first video segment based on the last image in the time window and at least one other image in the time window can be divided into the following sub-steps:

[0068] Step 302a, for each reference image in the at least one reference image, calculate the partial derivative image of the last image, the value of each pixel in the partial derivative image is the partial derivative of the value of the corresponding pixel in the last image with respect to the value of the corresponding pixel in the reference image.

[0069] In the embodiment of the present application, the image in the input video stream can be defined as f(x, y, t), where x is the horizontal component of the image, y is the vertical component of the image; t represents time t=1, 2, ..., t0, .... For two frames of images f(x, y, t0) and f(x, y, t0-q) of the input video stream, q=1 for two adjacent frames of images.

[0070] Definition: A frame of image at time t0 is f(x, y, t0), where the image at time t0 is the last image in the above time window, and the image at the previous q moments is f(x, y, t0-q). The gesture recognition device calculates the partial derivative of the video stream at time t0 relative to time t0-q:

[0071]

[0072] Step 302b, normalize the values ​​of each pixel in the partial derivative image to obtain a normalized partial derivative image.

[0073] The gesture recognition device may normalize g(x, y, t0) to the range [a, b]. For example, the normalization range is [a, b] = [0, 1]. That is, the gesture recognition device normalizes the value of each pixel in g(x, y, t0) to a value in the interval [0, 1].

[0074] Step 302c, binarizing the normalized partial derivative image according to a preset binarization threshold value to obtain a binarized image of the partial derivative image, wherein the value of each pixel in the binarized image is 0 or 1.

[0075] In the embodiment of the present application, after obtaining the normalized partial derivative image, the normalized partial derivative image can be binarized according to the size relationship between the value of each pixel in the normalized partial derivative image and the preset binarization threshold, and the value of each pixel in the normalized partial derivative image is binarized to 0 or 1. The binarization formula is as follows:

[0076]

[0077] In the above formula 2, Z is the preset binarization threshold. For the normalized partial derivative image g b The value of the pixel in (x, y, t0) is binarized to 1 when the value of the pixel is greater than Z, and is binarized to 0 when the value of the pixel is less than or equal to T.

[0078] The preset binarization threshold is a preset value between (0, 1), for example, the preset binarization threshold may be 0.5, or the preset binarization threshold may also be 0.4 or 0.6, etc. The binarization threshold may be preset by the developer according to the actual processing effect.

[0079] Step 302d, calculating the sum of the grayscale values ​​of each pixel in the binary image.

[0080] Step 302e: when the sum of the grayscale values ​​is greater than 0, it is determined that an action occurs in the first video segment.

[0081] In the embodiment of the present application, after obtaining the binary image gb(x, y, t0), the gesture recognition device calculates the grayscale value sum Sum(t0) of gb(x, y, t0). When the sum Sum(t0) is greater than 0, it can be determined that there is an action in the first video segment. Otherwise, it is considered that there is "no action" in the first video segment. The formula is as follows:

[0082] Sum(t0)=∑ (x，y) g b (x, y, t0) (3)

[0083] If Sum(t0)>0, it is determined that an action has occurred, and the process goes to step 303 ; if Sum(t0)≤0, it is determined that no action has occurred, and the process goes back to step 301 .

[0084] Step 303: Acquire M images, where the M images are extracted from the first video segment.

[0085] When it is determined in step 302 that an action occurs in the first video segment, the gesture recognition device may extract M images from the first video segment, where M is an integer greater than or equal to 2.

[0086] In practical applications, the gesture recognition device may extract each image in the first video segment to obtain the M images. Alternatively, the gesture recognition device may extract an image every other one or more images in the first video segment to obtain the M images.

[0087] Step 304: perform image processing on the M images to obtain an optical flow information image corresponding to the first video segment.

[0088] The above-mentioned optical flow information image includes optical flow information between a first image among M images and the pth image before the first image, where the first image is any one of the M images, and the optical flow information includes instantaneous velocity vector information of pixel points in the image, where p is an integer greater than or equal to 1.

[0089] Optical flow is the instantaneous speed of the pixel movement of a spatial moving object on the observation imaging plane. The gesture recognition device can use the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous image and the current image, so as to calculate the motion information of the object between the two images. The calculated motion information of the object between the two images is the optical flow information between the two images. The above method of calculating the motion information of the object between the two images is called the optical flow method. The optical flow information is also called the optical flow field, which refers to the apparent motion of the grayscale mode of the image. It is a two-dimensional vector field. The information it contains is the instantaneous motion speed vector information of each pixel. Therefore, the optical flow information can be expressed as a two-channel image with the same size as the original image.

[0090] When extracting the optical flow information image, the gesture recognition device can use the RGB image sequence in the first video segment to obtain an optical flow information image (regardless of the number of frames contained in the first video segment). In the embodiment of the present application, the optical flow information image corresponding to the first video segment can be obtained in the following two ways:

[0091] 1. For a first image among the M images, obtain the p-th image before the first image in the video stream according to a preset rule; calculate the optical flow information between the first image and the p-th image, and generate an optical flow information image including the optical flow information between the first image and the p-th image.

[0092] The time interval between the first image and the pth image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image. The first deep learning algorithm is the algorithm used by the gesture recognition device to subsequently recognize gestures based on the optical flow information image.

[0093] The preset rules may be rules set by developers or users, for example, developers or users may manually set the value of p. Alternatively, the gesture recognition device may also set the value of p according to the preset rules and the processing performance of the device; for example, the gesture recognition device may pre-run the forward calculation of the first deep learning algorithm and the calculation of the optical flow information image, and record the forward calculation time and the time for calculating the optical flow information image, and set the value of p according to the forward calculation time, the time for calculating the optical flow information image, and the frame rate of the video stream (i.e., the number of images contained in each second of the video). Specifically, for example, the gesture recognition device may determine the number of images in the video stream corresponding to the larger value of the forward calculation time and the time for calculating the optical flow information image, and set the value corresponding to the determined number of images as the value of p.

[0094] For real-time video, assuming that T is the time interval between a certain image and the pth image before the image, the minimum value of T can be the larger value of the time required for the deep learning network of the gesture recognition device to perform forward calculation of gesture recognition through the optical flow information image and the time required for the gesture recognition device to calculate the optical flow information image.

[0095] Assuming OF[·] represents an optical flow algorithm, in a possible implementation, the gesture recognition device can use the Eulerian motion field algorithm to identify an image I among p images. t (x, y), and I t (x, y) before the pth image I t-T (x, y), directly calculate and obtain the optical flow information corresponding to one of the M images, and generate an optical flow information image containing the calculated optical flow information, and use the optical flow information image as the optical flow information image corresponding to the first video segment. Its calculation formula can be simply expressed as follows:

[0096] U t (x, y) = OF[I t-T (x,y),I t (x, y)] (4)

[0097] In the above formula (4), U t (x, y) is image I t The optical flow information image corresponding to (x, y). OF[·] represents the above-mentioned Euler optical flow algorithm.

[0098] 2. For the first image among the M images, obtain all p images before the first image in the video stream according to a preset rule; calculate the optical flow information between the first image and every two adjacent images among the p images, accumulate the optical flow information between every two adjacent images, and generate an optical flow information image containing the accumulated optical flow information.

[0099] In another possible implementation, the gesture recognition device may calculate an image I among the M images by using a Lagrangian motion field algorithm. t (x, y), and I t (x, y) before p images I t-1 (x,y),I t-2 (x,y),...,I t-T The optical flow information between every two adjacent images in (x, y) is then accumulated to generate an image U containing the accumulated optical flow information. t(x, y). The process of accumulating optical flow information involves missing data interpolation, and you can choose linear, bilinear, cubic and other interpolation methods. The calculation formula can be simply expressed as follows:

[0100]

[0101] In the above formula (4), U t (x, y) is image I t The optical flow information image corresponding to (x, y). OF[·] represents the above-mentioned Lagrangian optical flow algorithm.

[0102] For method 1, the optical flow field only needs to be calculated once, so a more accurate optical flow field algorithm needs to be selected. For method 2, the optical flow field needs to be calculated multiple times, and an optical flow field algorithm with low accuracy but high speed can be used.

[0103] Step 305: perform image processing on the M images to obtain a color information image corresponding to the first video segment.

[0104] The color information image includes the color information of the M images, and the color information includes the color value of each pixel in the image.

[0105] The gesture recognition device processes the image sequence in the first video segment and outputs m color information images, such as RGB (red green blue) images, to represent the color information image corresponding to the first video segment, where m is an integer greater than or equal to 1. Assume that all images I in the first video segment are input t-T (x,y),I t-T+1 (x, y), ..., I t-1 (x,y),I t (x, y), and the output is m images at the end of the video segment To represent the color information of the image in the first video segment.

[0106] When performing image processing on the M images to obtain the color information image corresponding to the first video segment, the gesture recognition device may obtain the color information image by the following method:

[0107] 1) Extracting color information of m images among the M images, generating color information images corresponding to the m images according to the extracted color information, and acquiring the color information images corresponding to the m images as the color information images corresponding to the first video segment. For example, after the gesture recognition device extracts color information of any one of the m images, it generates a color information image corresponding to the any one image, and the generated color information image includes the color information of the any one image.

[0108] The m images are random m images from the M images. For example, taking a single color information image as an example, in the embodiment of the present application, when the time length of the first video segment is relatively short, a color information image corresponding to an image can be randomly selected from the first video segment to represent Right now:

[0109]

[0110] Wherein, tT is the time point corresponding to the first image in the first video segment, and t is the time point corresponding to the last image in the first video segment.

[0111] Optionally, in addition to randomly selecting the color information images of the m images as the color information images corresponding to the first video segment, the gesture recognition device may also select the color information images of the m images as the color information images corresponding to the first video segment through other strategies. For example, the gesture recognition device may select the color information images of the m images that are at the front or the back of the corresponding time among the above M images as the color information images corresponding to the first video segment.

[0112] In another possible implementation, the m images may be the m images among the M images that have the largest changes relative to their respective previous images in the video stream.

[0113] For example, for each of the M images, the gesture recognition device can detect pixels in the image that have changed compared to an image that is before the image in the video stream; the gesture recognition device can obtain the color information image corresponding to the m images in the M images that have the largest number of pixels that have changed compared to their respective previous images as the color information image corresponding to the first video segment.

[0114] 2) Detecting pixel positions in the M images where image content changes over time, calculating an average value of color information at corresponding identified pixel positions in the M images, obtaining new color information at the identified pixel positions, and generating a color information image corresponding to the first video segment based on the new color information at the identified pixel positions.

[0115] In an embodiment of the present application, the gesture recognition device may also compare pixel points at the same position in the M images to identify pixel positions in the M images where image content changes over time (the pixel position may be the coordinates of the pixel point in the image), and average the color information of the pixel points at the corresponding identified pixel positions in the M images to obtain new color information at the corresponding identified pixel positions, and generate a new color information image, wherein in the new color information image, the color information corresponding to the above-identified pixel position is the new color information obtained by the above-mentioned averaging.

[0116] Among them, the above-mentioned algorithm for detecting pixels in an image that have changed compared with the previous image, and the algorithm for detecting pixel positions in an image that have changed over time can be collectively referred to as a spatiotemporal saliency image detection algorithm.

[0117] 3) Extract color information of all or part of the M images to obtain color information images corresponding to all or part of the images, calculate the average value of color information at each pixel in the color information images corresponding to all or part of the images, and obtain the color information image corresponding to the first video segment.

[0118] For example, taking the above-mentioned M images as all images in the frequency band and obtaining a single color information image as an example, the calculation formula for taking the average value of the color information of each image in the first video segment as the color information image of the first video segment can be as follows:

[0119]

[0120] Wherein, tT is the time point corresponding to the first image in the first video segment, t is the time point corresponding to the last image in the first video segment; and n is the number of images in the first video segment.

[0121] 4) Extracting color information of all or part of the M images, generating color information images corresponding to all or part of the images, calculating the average value of the color information at each pixel in the color information images corresponding to all or part of the images, and then subtracting the color information of each pixel in the color information images corresponding to all or part of the images from the color information of each pixel calculated above, and using the obtained color information image as the color information image corresponding to the first video segment.

[0122] Since in the above-mentioned M images, the pixels that change are usually the foreground part of the image (i.e., the part corresponding to the human hand), and the pixels corresponding to the background part are usually unchanged, therefore, in the color information images corresponding to all or part of the above-mentioned images, the color information of the pixels corresponding to the background part is usually the same or similar to the average value of the color information at that location, while the color information of the pixels corresponding to the background part is usually quite different from the average value of the color information at that location. Therefore, in the embodiment of the present application, the color information of each pixel in the color information image corresponding to all or part of the images can also be subtracted from the average value of the corresponding pixel position to obtain the color information image corresponding to all or part of the images after removing the background part. The gesture recognition device can use the color information image corresponding to all or part of the images after removing the background part as the color information image corresponding to the first video segment.

[0123] Step 306: Perform gesture recognition on the optical flow information image using a first deep learning algorithm to obtain a first recognition result, and perform gesture recognition on the color information image using a second deep learning algorithm to obtain a second recognition result.

[0124] In an embodiment of the present application, a color information image (such as an RGB image) and an optical flow information image can be obtained in a previous step based on an input video stream, and in this step 306, two deep learning models are used to perform gesture recognition respectively, and the recognition results of the two deep learning models are fused in the next step.

[0125] The embodiment of the present application uses a dual-channel deep learning model for gesture recognition, one of which is a Temporal stream (corresponding to the first deep learning algorithm mentioned above), which inputs an optical flow information image and finally outputs a gesture recognition result for the current optical flow information image; for example, in the above step 304, for each of the M images, after the gesture recognition device obtains the optical flow information image of the image, it caches the optical flow information image, and when performing gesture recognition on the optical flow information image, the gesture recognition device inputs the most recently stored X optical flow information images into the deep learning channel Temporal stream to output gesture recognition results corresponding to the X optical flow information images, and uses the output gesture recognition result as the result of performing gesture recognition on the optical flow information image of the first video segment.

[0126] In the above-mentioned dual-channel deep learning model, the other channel is the Spatial stream (corresponding to the above-mentioned second deep learning algorithm), whose input is the at least one color information image obtained in step 305, representing the first video segment, and the output is the gesture recognition result of the at least one color information image.

[0127] Among them, the above-mentioned dual-channel deep learning model is a pre-trained machine learning model.

[0128] Step 307: fuse the first recognition result and the second recognition result to obtain a gesture recognition result of the first video segment.

[0129] In the embodiment of the present application, since the gesture recognition result of the optical flow information image and the gesture recognition result included in the color information image obtained in the above step 306 are gesture recognition results for the same video segment, after the gesture recognition device obtains the gesture recognition result of the optical flow information image and the gesture recognition result included in the color information image, the two results can be merged to obtain the gesture recognition result of the first video segment.

[0130] There are two ways to fuse the first recognition result and the second recognition result:

[0131] One is to calculate an average value of the first recognition result and the second recognition result, and obtain the gesture recognition result of the first video segment according to the calculation result of the average value.

[0132] Another method is to input the first recognition result and the second recognition result into a pre-trained second machine learning model, such as a linear support vector machine (SVM) model, to obtain the gesture recognition result of the first video segment. The second machine learning model is a learning model for determining a single recognition result based on the two input recognition results, and the second machine learning model can be obtained by training a video segment with gestures pre-labeled. For example, the two recognition results can be two numerical values, and the gesture recognition device can input the two numerical values ​​into the second machine learning model. The second machine learning model calculates a fused numerical value according to the pre-trained calculation formula and the two input numerical values, and outputs the fused numerical value as the gesture recognition result of the first video segment.

[0133] For each video segment, the gesture recognition device obtains a phased gesture recognition result corresponding to the video segment in real time, and stores it in a temporary gesture recognition result library.

[0134] Step 308: After obtaining the gesture recognition results of N consecutive video segments including the first video segment in the video stream, the gesture recognition results of the N consecutive video segments are fused to obtain a fused gesture recognition result.

[0135] Wherein, N≥2, and N is an integer.

[0136] In an embodiment of the present application, when fusing the gesture recognition results of N consecutive video segments, the gesture recognition device may input the gesture recognition results of the N consecutive video segments into a pre-trained first machine learning model to obtain the fused gesture recognition result, and the first machine learning model is used to determine the overall gesture movement trend formed by the input N consecutive gesture recognition results, and output the gesture corresponding to the overall gesture movement trend as the fused gesture recognition result. Specifically, for example, the N consecutive gesture recognition results may be N numerical values, and the gesture recognition device may input the N numerical values ​​into the first machine learning model in the time sequence of the N video segments, and the first machine learning model calculates a fused numerical value according to the pre-trained calculation formula and the N numerical values ​​input successively, and outputs the fused numerical value as the fused gesture recognition result.

[0137] Among them, the first machine learning model is a neural network model, and the number of neurons in the neural network model is N; or, the first machine learning model is a support vector machine SVM model.

[0138] Alternatively, in an embodiment of the present application, when the gesture recognition results of the N consecutive video segments are fused, the gesture recognition device may obtain pre-set weight coefficients corresponding to each of the gesture recognition results of the N consecutive video segments; and perform weighted averaging on the gesture recognition results of the N consecutive video segments according to the weight coefficients corresponding to each of the gesture recognition results of the N consecutive video segments to obtain the fused gesture recognition result.

[0139] In actual applications, when a user performs a single gesture operation, the overall gesture movement trend is consistent with the gesture action that the user wants to make, but there may be a short period of time when the gesture does not conform to the gesture action that the user wants to make. For example, taking the gesture operation that the user wants to make as an example of raising the hand upward, the user makes a gesture operation of raising the hand upward within 1s, but in a very short time period (such as 0.2s) within this 1s, the user does not raise his hand upward, but slightly presses his hand downward, and after this very short time period, the user continues to raise his hand upward. At this time, the gesture recognition result in the above very short time period recognized by the gesture recognition device does not conform to the gesture operation that the user currently wants to perform. Therefore, in an embodiment of the present application, in order to improve the accuracy of gesture recognition, the gesture recognition device can fuse the gesture recognition results of multiple consecutive video segments (i.e., a sequence of gesture recognition results), and use the overall gesture movement trend reflected by the gesture recognition results of multiple video segments as the fused gesture recognition result.

[0140] Specifically, in the real-time continuous video stream, within N*T1 time (T1 is the time length of a video segment), the gesture recognition device calculates N action recognition interim results, and uses the fusion decision of these N recognition interim results (after N*T1 time) to give a final recognition result. According to the average length of time a user makes a gesture, N*T1 can be taken as about 1 second. There are many ways to implement the fusion of N recognition interim results, such as the following two methods:

[0141] (1) Linear combination:

[0142] Result=α1r1+α2r2+……+α N r N (8)

[0143] Here, r1, r2, …, r N is the staged recognition result, and the weight coefficient before each result is α1, α2, …, α N ,These weight coefficients can be coefficients determined in advance by a machine learning algorithm, and different combinations of coefficients will produce different combinations.

[0144] (2) Support vector machine method SVM:

[0145] Please refer to Figure 5 , which shows a schematic diagram of a recognition result fusion involved in an embodiment of the present application. Figure 5 As shown, in the embodiment of the present application, the phased recognition results r1, r2, ..., r N Input to a pre-trained machine learning model, i.e. Figure 5 In the SVM module shown (the SVM kernel function in the SVM module is a pre-set or trained kernel function), the fusion result is output.

[0146] After obtaining the fused gesture recognition results, the gesture recognition device can call corresponding modules (eg, slide show presentation, full-screen picture playback, etc.) according to the gesture recognition results to achieve the purpose of human-computer interaction.

[0147] In the embodiment of the present application, if it is determined that no action occurs in the video segment, the gesture recognition device may not perform gesture recognition on the video segment to reduce the frequency of gesture recognition and avoid unnecessary recognition processes. Specifically, the gesture recognition device may directly set the gesture recognition result in the video segment to null, or not set the gesture recognition result for the video segment.

[0148] Please refer to Figure 6 , which shows a flow chart of a gesture recognition process involved in an embodiment of the present application. Figure 1 As an example, the gesture recognition system shown in Figure 6 As shown, the image acquisition device inputs the acquired video stream into the gesture recognition device. After the gesture recognition device extracts an image from the video stream, it determines whether an action has occurred in the image relative to the at least one previous image according to the method shown in the above step 302, based on the image and at least one image within a period of time before the image. If it is determined that an action has occurred, the gesture recognition device extracts the optical flow information image and the color information image of the video segment according to the methods of steps 304 and 305 for each image in a video segment where the current image in the video stream is located (or can also be for part of the images), and performs gesture recognition on the optical flow information image and the color information image according to the method shown in step 306, and then fuses the gesture recognition results obtained by performing gesture recognition on the optical flow information image and the color information image according to the method shown in step 307 to obtain the staged gesture recognition result corresponding to the video segment. After N staged gesture recognition results are obtained continuously, the gesture recognition device fuses the N gesture recognition results according to the method shown in step 308 to obtain a fused gesture recognition result.

[0149] Optionally, the machine learning models involved above (including the first machine learning model, the second machine learning model, and the dual-channel deep learning model in step 306, etc.) can be obtained by machine training with video samples that are pre-labeled with corresponding gestures.

[0150] The above-mentioned machine training process can be implemented by a model training device. Specifically, for example, taking the above-mentioned first machine learning model, the second machine learning model and the dual-channel deep learning model as examples, all of which are obtained through machine training, in a possible implementation method, the developer can input several video stream samples into the model training device, each video stream sample contains a gesture, and the developer pre-marks the gestures in each video stream sample, and the developer divides each video stream into multiple video segments and marks the stage gestures corresponding to each video segment. When performing machine training, the model training device extracts the optical flow information image and the color information image for each video segment according to the scheme shown in steps 304 and 305, and inputs the optical flow information image and the color information image of the video segment into the dual-channel deep learning model, and inputs the two recognition results output by the dual-channel deep learning model, as well as the stage gestures of the video segment that have been marked, into the second machine learning model, so as to perform model training on the dual-channel deep learning model and the second machine learning model. In addition, for each video stream sample, the model training device inputs the stage gestures of each video segment in the video stream sample and the pre-marked gestures of the video stream sample into the first machine learning model for machine training to obtain the first machine learning model.

[0151] For another example, taking the above-mentioned first machine learning model, second machine learning model and dual-channel deep learning model as being obtained through machine training, in another possible implementation method, the developer can input several video stream samples into the model training device, each video stream sample contains a gesture, and the developer pre-labels the gesture in each video stream sample, the model training device divides the video stream into multiple video segments, and extracts the optical flow information image and the color information image for each video segment, and inputs the optical flow information image and the color information image of the video segment into the dual-channel deep learning model, inputs the two recognition results output by the dual-channel deep learning model into the second machine learning model, and then inputs the staged gesture recognition results for multiple video segments output by the second machine learning model into the first machine learning model. At the same time, the model training device inputs the labeled gesture corresponding to the video stream into the first machine learning model, so as to simultaneously perform machine training on the first machine learning model, the second machine learning model and the dual-channel deep learning model.

[0152] It should be noted that the method shown in the embodiment of the present invention is illustrated by taking the above-mentioned dual-channel deep learning model as an example. In actual applications, when the gesture recognition device recognizes each video segment, it can recognize the gesture of a single video segment through other deep learning algorithms. For example, the gesture recognition device can only recognize the gesture recognition result corresponding to the video segment through the optical flow information image, or the gesture recognition device can also only recognize the gesture recognition result corresponding to the video segment through the color information image. The embodiment of the present invention does not limit the above-mentioned deep learning algorithm for recognizing the gesture recognition result of the video segment.

[0153] In summary, the method shown in the embodiment of the present application, for each video segment in the video stream, extracts an optical flow information image and a color information image for the video segment respectively, and performs gesture recognition on the optical flow information image and the color information image respectively through a deep learning algorithm. After gesture recognition, the gesture recognition results corresponding to the two images are fused to determine the gesture recognition result corresponding to the video segment, and finally the gesture recognition results of N consecutive video segments including the video segment are fused to obtain the gesture recognition results of the N consecutive video segments. That is, in the above method, a complete gesture action is divided into multiple stage actions, each stage action is identified by a deep learning algorithm, and finally the identified stage actions are fused into a complete gesture action. During the recognition process, there is no need to segment and track the gestures in the video stream, but to identify the various stage actions by a deep learning algorithm with a faster computing speed, thereby achieving the effect of improving the speed of gesture recognition and reducing the delay of gesture recognition.

[0154] Figure 7is a schematic diagram of the structure of a gesture recognition device 70 provided by an exemplary embodiment of the present application. The gesture recognition device 70 can be implemented as Figure 1 The gesture recognition device 120 in the system shown. Figure 7 As shown, the gesture recognition device 70 may include: a processor 71 and a memory 73 .

[0155] The processor 71 may include one or more processing units, which may be a central processing unit (CPU) or a network processor (NP).

[0156] Optionally, the gesture recognition device 70 may further include a memory 73. The memory 73 may be used to store a software program, and the software program may be executed by the processor 71. In addition, the memory 73 may also store various types of business data or user data. The software program may include an image acquisition module, a recognition module, and a fusion module; optionally, the software program may also include a time window determination module and a judgment module;

[0157] The image acquisition module is executed by the processor 71 to achieve the above Figure 3 The illustrated embodiment is a function of obtaining M images extracted from the first video segment of a video stream.

[0158] The identification module is executed by the processor 71 to implement the above Figure 3 The embodiment shown is a function of identifying a gesture recognition result corresponding to a first video segment.

[0159] The fusion module is executed by the processor 71 to achieve the above Figure 3 The illustrated embodiment is related to the function of fusing the gesture recognition results of N consecutive video segments.

[0160] The time window determination module is executed by the processor 71 to implement the above Figure 3 The illustrated embodiment relates to the function of determining a time window.

[0161] The judgment module is executed by the processor 71 to implement the above Figure 3 The illustrated embodiment is related to the function of determining whether an action occurs in the first video segment.

[0162] Optionally, the gesture recognition device 70 may further include a communication interface 74, and the communication interface 74 may include a network interface. The network interface is used to connect to the image acquisition device. Specifically, the network interface may include a wired network interface, such as an Ethernet interface or an optical fiber interface, or the network interface may also include a wireless network interface, such as a wireless local area network interface or a cellular mobile network interface. The gesture recognition device 70 communicates with other devices through the network interface 74.

[0163] Optionally, the processor 71 may be connected to the memory 73 and the communication interface 74 via a bus.

[0164] Optionally, the gesture recognition device 70 may further include an output device 75 and an input device 77. The output device 75 and the input device 77 are connected to the processor 71. The output device 75 may be a display for displaying information, a power amplifier device for playing sound, or a printer, etc. The output device 75 may also include an output controller for providing output to the display screen, the power amplifier device, or the printer. The input device 77 may be a device such as a mouse, a keyboard, an electronic stylus, or a touch panel for user input of information, and the input device 77 may also include an output controller for receiving and processing input from devices such as a mouse, a keyboard, an electronic stylus, or a touch panel.

[0165] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0166] Figure 8 is a structural block diagram of a gesture recognition device provided by an exemplary embodiment of the present application. The gesture recognition device can be implemented as part or all of a gesture recognition device through a hardware circuit or a combination of software and hardware. The gesture recognition device can be the above-mentioned Figure 1 The gesture recognition device 120 in the illustrated embodiment may include: an image acquisition unit 801 , a recognition unit 802 , and a fusion unit 803 ; optionally, the software program may also include a time window determination unit 804 and a judgment unit 805 .

[0167] The image acquisition unit 801 is executed by a processor to achieve the above Figure 3 The illustrated embodiment is a function of obtaining M images extracted from the first video segment of a video stream.

[0168] The identification unit 802 is executed by the processor to implement the above Figure 3 The embodiment shown is related to the function of obtaining the gesture recognition result corresponding to the first video segment.

[0169] The fusion unit 803 is executed by the processor to achieve the above Figure 3 The illustrated embodiment is related to the function of fusing the gesture recognition results of N consecutive video segments.

[0170] The time window determination unit 804 is executed by the processor to implement the above Figure 3 The illustrated embodiment relates to the function of determining a time window.

[0171] The judgment unit 805 is executed by the processor to implement the above Figure 3 The illustrated embodiment is related to the function of determining whether an action occurs in the first video segment.

[0172] It should be noted that: the gesture recognition device provided in the above embodiment only uses the division of the above functional units as an example when performing gesture recognition. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device is divided into different functional units to complete all or part of the functions described above. In addition, the gesture recognition device provided in the above embodiment and the method embodiment of the gesture recognition method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0173] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0174] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0175] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A gesture recognition device, characterized in that: The device comprises: a processor and a memory; The processor is used to determine a time window of a preset time length in a video stream, calculate a partial derivative image of a last image in the time window for each reference image of at least one reference image in the time window, the value of each pixel in the partial derivative image is the partial derivative of the value of the corresponding pixel in the last image relative to the value of the corresponding pixel in the reference image, perform normalization processing and binarization processing on the partial derivative image in sequence to obtain a binarized image of the partial derivative image, judge whether an action occurs in a first video segment according to the sum of the grayscale values ​​of each pixel in the binarized image, and if the judgment result is that an action occurs in the first video segment, obtain M images, and the M images are extracted from the first video segment, wherein the first video segment is any video segment in the video stream, the end moment of the time window is within the time period corresponding to the first video segment, and M is an integer greater than or equal to 2; The processor is configured to perform gesture recognition on an optical flow information image corresponding to the first video segment and a color information image corresponding to the first video segment respectively through a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment, wherein the optical flow information image and the color information image are respectively obtained by performing image processing on the M images, and the deep learning algorithm used for gesture recognition on the optical flow information image is different from the deep learning algorithm used for gesture recognition on the color information image; The processor is configured to, after obtaining gesture recognition results of N consecutive video segments including the first video segment in the video stream, fuse the gesture recognition results of the N consecutive video segments to obtain a fused gesture recognition result, where N is an integer greater than or equal to 2; wherein the gesture recognition result of each video segment in the N video segments is a stage action, the fused gesture recognition result is a gesture action, and the gesture action includes multiple stage actions.

2. The device according to claim 1, characterized in that When fusing the gesture recognition results of the N consecutive video segments to obtain the fused gesture recognition result, the processor is specifically configured to: The gesture recognition results of the N consecutive video segments are input into a pre-trained first machine learning model to obtain the fused gesture recognition result, the first machine learning model is used to determine the overall gesture movement trend constituted by the input N consecutive gesture recognition results, and output the gesture corresponding to the overall gesture movement trend as the fused gesture recognition result.

3. The device according to claim 2, characterized in that The first machine learning model is a neural network model, and the number of neurons in the neural network model is N; or, The first machine learning model is a support vector machine (SVM) model.

4. The device according to claim 1, characterized in that When fusing the gesture recognition results of the N consecutive video segments to obtain the fused gesture recognition result, the processor is specifically configured to: Obtaining preset weight coefficients corresponding to the gesture recognition results of the N consecutive video segments; According to the weight coefficients corresponding to the gesture recognition results of the N consecutive video segments, the gesture recognition results of the N consecutive video segments are weighted averaged to obtain the fused gesture recognition result.

5. The device according to claim 1, characterized in that When performing gesture recognition on the optical flow information image corresponding to the first video segment and the color information image corresponding to the first video segment respectively through a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment, the processor is specifically configured to: Performing image processing on the M images to obtain the optical flow information image corresponding to the first video segment, the optical flow information image including optical flow information between a first image among the M images and a p-th image before the first image, the first image being any one of the M images, the optical flow information including instantaneous velocity vector information of pixel points in the image, and performing gesture recognition on the optical flow information image through a first deep learning algorithm to obtain a first recognition result, where p is an integer greater than or equal to 1; Performing image processing on the M images to obtain the color information image corresponding to the first video segment, the color information image including color information of the M images, the color information including color values ​​of each pixel in the image, and performing gesture recognition on the color information image using a second deep learning algorithm to obtain a second recognition result; The first recognition result and the second recognition result are fused to obtain a gesture recognition result of the first video segment.

6. The device according to claim 5, characterized in that When performing image processing on the M images to obtain the optical flow information image corresponding to the first video segment, the processor is specifically configured to: For the first image, obtain the p-th image before the first image in the video stream according to a preset rule; calculate the optical flow information between the first image and the p-th image, and generate an optical flow information image containing the optical flow information between the first image and the p-th image; wherein the time interval between the first image and the p-th image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image; or, For the first image, all p images before the first image in the video stream are obtained according to preset rules; the optical flow information between the first image and each two adjacent images in the M images is calculated, and the optical flow information between each two adjacent images is accumulated to generate an optical flow information image containing the accumulated optical flow information; wherein the time interval between the first image and the p-th image before the first image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image.

7. The device according to claim 5, characterized in that When performing image processing on the M images to obtain the color information image corresponding to the first video segment, the processor is specifically configured to: Extracting color information of m images among the M images, generating color information images corresponding to the m images according to the extracted color information, and acquiring the color information images corresponding to the m images as color information images corresponding to the first video segment; the m images are m random images among the M images, or the m images are m images among the M images that have the largest changes relative to their respective previous images in the video stream, where m is an integer greater than or equal to 1; Alternatively, the pixel positions of the M images whose image contents change with time are detected, the average value of the color information at the corresponding identified pixel positions in the M images is calculated, new color information at the identified pixel positions is obtained, and a color information image corresponding to the first video segment is generated based on the new color information at the identified pixel positions.

8. The device according to any one of claims 1 to 7, characterized in that: The reference image is any other image except the last image in the time window.

9. The device according to any one of claims 1 to 7, characterized in that: When the partial derivative image is subjected to normalization processing and binarization processing in sequence to obtain a binarized image of the partial derivative image, the processor is specifically configured to: Normalizing the values ​​of each pixel in the partial derivative image to obtain a normalized partial derivative image; According to a preset binarization threshold, the normalized partial derivative image is binarized to obtain a binarized image of the partial derivative image, wherein the value of each pixel in the binarized image is 0 or 1; When judging whether an action occurs in the first video segment according to the sum of the grayscale values ​​of each pixel in the binary image, the processor is specifically configured to: Calculating the sum of the grayscale values ​​of each pixel in the binary image; When the sum of the grayscale values ​​is greater than 0, it is determined that an action occurs in the first video segment.

10. The device according to any one of claims 5 to 7, characterized in that: The processor, when fusing the first recognition result and the second recognition result to obtain the gesture recognition result of the first video segment, is specifically configured to: Calculating an average value of the first recognition result and the second recognition result, and obtaining a gesture recognition result of the first video segment according to a calculation result of the average value calculation; or, The first recognition result and the second recognition result are input into a pre-trained second machine learning model to obtain a gesture recognition result of the first video segment.

11. A gesture recognition device, characterized in that: The device comprises: A time window determination unit, configured to determine a time window of a preset time length in a video stream, wherein an end time of the time window is within a time period corresponding to a first video segment, and the first video segment is any video segment in the video stream; a judgment unit, configured to calculate, for each reference image of at least one reference image in the time window, a partial derivative image of a last image in the time window, wherein a value of each pixel in the partial derivative image is a partial derivative of a value of a corresponding pixel in the last image relative to a value of a corresponding pixel in the reference image, perform normalization processing and binarization processing on the partial derivative image in sequence to obtain a binarized image of the partial derivative image, and judge whether an action occurs in the first video segment according to a sum of grayscale values ​​of each pixel in the binarized image; an image acquisition unit, configured to acquire M images when a judgment result shows that an action occurs in the first video segment, wherein the M images are extracted from the first video segment, and M is an integer greater than or equal to 2; a recognition unit, configured to perform gesture recognition on an optical flow information image corresponding to the first video segment and a color information image corresponding to the first video segment respectively through a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment, wherein the optical flow information image and the color information image are respectively obtained by performing image processing on the M images, and the deep learning algorithm used for gesture recognition on the optical flow information image is different from the deep learning algorithm used for gesture recognition on the color information image; A fusion unit is used to fuse the gesture recognition results of N consecutive video segments including the first video segment in the video stream after obtaining the gesture recognition results of the N consecutive video segments to obtain a fused gesture recognition result; N≥2, and N is an integer; wherein the gesture recognition result of each video segment in the N video segments is a stage action, the fused gesture recognition result is a gesture action, and the gesture action includes multiple stage actions.

12. The device according to claim 11, characterized in that When fusing the gesture recognition results of the N consecutive video segments to obtain the fused gesture recognition result, the fusion unit is specifically used to: The gesture recognition results of the N consecutive video segments are input into a pre-trained first machine learning model to obtain the fused gesture recognition result, the first machine learning model is used to determine the overall gesture movement trend constituted by the input N consecutive gesture recognition results, and output the gesture corresponding to the overall gesture movement trend as the fused gesture recognition result.

13. The device according to claim 12, characterized in that The first machine learning model is a neural network model, and the number of neurons in the neural network model is N; or, The first machine learning model is a support vector machine (SVM) model.

14. The device according to claim 11, characterized in that When fusing the gesture recognition results of the N consecutive video segments to obtain the fused gesture recognition result, the fusion unit is specifically used to: Obtaining preset weight coefficients corresponding to the gesture recognition results of the N consecutive video segments; According to the weight coefficients corresponding to the gesture recognition results of the N consecutive video segments, the gesture recognition results of the N consecutive video segments are weighted averaged to obtain the fused gesture recognition result.

15. The device according to claim 11, characterized in that The identification unit is specifically used for: Performing image processing on the M images to obtain the optical flow information image corresponding to the first video segment, the optical flow information image including optical flow information between a first image among the M images and a p-th image before the first image, the first image being any one of the M images, the optical flow information including instantaneous velocity vector information of pixel points in the image, and performing gesture recognition on the optical flow information image through a first deep learning algorithm to obtain a first recognition result, where p is an integer greater than or equal to 1; Performing image processing on the M images to obtain the color information image corresponding to the first video segment, the color information image including color information of the M images, the color information including color values ​​of each pixel in the image, and performing gesture recognition on the color information image using a second deep learning algorithm to obtain a second recognition result; The first recognition result and the second recognition result are fused to obtain a gesture recognition result of the first video segment.

16. The device according to claim 15, characterized in that When performing image processing on the M images to obtain the optical flow information image corresponding to the first video segment, the recognition unit is specifically configured to: For the first image, obtain the p-th image before the first image in the video stream according to a preset rule; calculate the optical flow information between the first image and the p-th image, and generate an optical flow information image containing the optical flow information between the first image and the p-th image; wherein the time interval between the first image and the p-th image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image; or, For the first image, all p images before the first image in the video stream are obtained according to preset rules; the optical flow information between the first image and each two adjacent images in the M images is calculated, and the optical flow information between each two adjacent images is accumulated to generate an optical flow information image containing the accumulated optical flow information; wherein the time interval between the first image and the p-th image before the first image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image.

17. The device according to claim 15, characterized in that When performing image processing on the M images to obtain the color information image corresponding to the first video segment, the recognition unit is specifically used to: Extracting color information of m images among the M images, generating color information images corresponding to the m images according to the extracted color information, and acquiring the color information images corresponding to the m images as color information images corresponding to the first video segment; the m images are m random images among the M images, or the m images are m images among the M images that have the largest changes relative to their respective previous images in the video stream, where m is an integer greater than or equal to 1; Alternatively, the pixel positions of the M images whose image contents change with time are detected, the average value of the color information at the corresponding identified pixel positions in the M images is calculated, new color information at the identified pixel positions is obtained, and a color information image corresponding to the first video segment is generated based on the new color information at the identified pixel positions.

18. The device according to any one of claims 11 to 17, characterized in that The reference image is any other image except the last image in the time window.

19. The device according to any one of claims 11 to 17, characterized in that When the partial derivative image is subjected to normalization processing and binarization processing in sequence to obtain a binarized image of the partial derivative image, the judging unit is specifically configured to: Normalizing the values ​​of each pixel in the partial derivative image to obtain a normalized partial derivative image; According to a preset binarization threshold, the normalized partial derivative image is binarized to obtain a binarized image of the partial derivative image, wherein the value of each pixel in the binarized image is 0 or 1; When judging whether an action occurs in the first video segment according to the sum of the grayscale values ​​of each pixel in the binary image, the judging unit is specifically configured to: Calculating the sum of the grayscale values ​​of each pixel in the binary image; When the sum of the grayscale values ​​is greater than 0, it is determined that an action occurs in the first video segment.

20. The device according to any one of claims 15 to 17, characterized in that When the first recognition result and the second recognition result are merged to obtain the gesture recognition result of the first video segment, the recognition unit is specifically configured to: Calculating an average value of the first recognition result and the second recognition result, and obtaining a gesture recognition result of the first video segment according to a calculation result of the average value calculation; or, The first recognition result and the second recognition result are input into a pre-trained second machine learning model to obtain a gesture recognition result of the first video segment.

21. A gesture recognition method, characterized in that: The method comprises: Determine a time window of a preset time length in the video stream, where an end time of the time window is within a time period corresponding to a first video segment, and the first video segment is any video segment in the video stream; For each of at least one reference image in the time window, a partial derivative image of the last image in the time window is calculated, wherein the value of each pixel in the partial derivative image is the partial derivative of the value of the corresponding pixel in the last image relative to the value of the corresponding pixel in the reference image, the partial derivative image is subjected to normalization processing and binarization processing in sequence to obtain a binarized image of the partial derivative image, and whether an action occurs in the first video segment is determined according to the sum of the grayscale values ​​of each pixel in the binarized image; If the judgment result is that an action occurs in the first video segment, then obtaining M images, where the M images are extracted from the first video segment, and M is an integer greater than or equal to 2; Performing gesture recognition on an optical flow information image corresponding to the first video segment and a color information image corresponding to the first video segment respectively through a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment, wherein the optical flow information image and the color information image are respectively obtained by performing image processing on the M images, and the deep learning algorithm used for gesture recognition on the optical flow information image is different from the deep learning algorithm used for gesture recognition on the color information image; After obtaining gesture recognition results of N consecutive video segments including the first video segment in the video stream, the gesture recognition results of the N consecutive video segments are fused to obtain a fused gesture recognition result; N≥2, and N is an integer; wherein the gesture recognition result of each video segment in the N video segments is a stage action, the fused gesture recognition result is a gesture action, and the gesture action includes multiple stage actions.

22. The method according to claim 21, characterized in that The step of fusing the gesture recognition results of the N consecutive video segments to obtain a fused gesture recognition result includes: The gesture recognition results of the N consecutive video segments are input into a pre-trained first machine learning model to obtain the fused gesture recognition result, the first machine learning model is used to determine the overall gesture movement trend constituted by the input N consecutive gesture recognition results, and output the gesture corresponding to the overall gesture movement trend as the fused gesture recognition result.

23. The method according to claim 22, characterized in that The first machine learning model is a neural network model, and the number of neurons in the neural network model is N; or, The first machine learning model is a support vector machine (SVM) model.

24. The method according to claim 21, characterized in that The step of fusing the gesture recognition results of the N consecutive video segments to obtain a fused gesture recognition result includes: Obtaining preset weight coefficients corresponding to the gesture recognition results of the N consecutive video segments; According to the weight coefficients corresponding to the gesture recognition results of the N consecutive video segments, the gesture recognition results of the N consecutive video segments are weighted averaged to obtain the fused gesture recognition result.

25. The method according to claim 21, characterized in that The performing gesture recognition on the optical flow information image corresponding to the first video segment and the color information image corresponding to the first video segment respectively by a deep learning algorithm to obtain a gesture recognition result corresponding to the first video segment includes: Performing image processing on the M images to obtain the optical flow information image corresponding to the first video segment, the optical flow information image including optical flow information between a first image among the M images and a p-th image before the first image, the first image being any one of the M images, the optical flow information including instantaneous velocity vector information of pixel points in the image, and performing gesture recognition on the optical flow information image through a first deep learning algorithm to obtain a first recognition result, where p is an integer greater than or equal to 1; Performing image processing on the M images to obtain the color information image corresponding to the first video segment, the color information image including color information of the M images, the color information including color values ​​of each pixel in the image, and performing gesture recognition on the color information image using a second deep learning algorithm to obtain a second recognition result; The first recognition result and the second recognition result are fused to obtain a gesture recognition result of the first video segment.

26. The method according to claim 25, characterized in that The performing image processing on the M images to obtain the optical flow information image corresponding to the first video segment includes: For the first image, obtain the p-th image before the first image in the video stream according to a preset rule; calculate the optical flow information between the first image and the p-th image, and generate an optical flow information image containing the optical flow information between the first image and the p-th image; wherein the time interval between the first image and the p-th image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image; or, For the first image, all p images before the first image in the video stream are obtained according to preset rules; the optical flow information between the first image and each two adjacent images in the M images is calculated, and the optical flow information between each two adjacent images is accumulated to generate an optical flow information image containing the accumulated optical flow information; wherein the time interval between the first image and the p-th image before the first image is not less than the forward calculation time of the first deep learning algorithm and the time required to calculate the optical flow information image.

27. The method according to claim 25, characterized in that The performing image processing on the M images to obtain the color information image corresponding to the first video segment includes: Extracting color information of m images among the M images, generating color information images corresponding to the m images according to the extracted color information, and acquiring the color information images corresponding to the m images as color information images corresponding to the first video segment; the m images are m random images among the M images, or the m images are m images among the M images that have the largest changes relative to their respective previous images in the video stream, where m is an integer greater than or equal to 1; Alternatively, the pixel positions of the M images whose image contents change with time are detected, the average value of the color information at the corresponding identified pixel positions in the M images is calculated, new color information at the identified pixel positions is obtained, and a color information image corresponding to the first video segment is generated based on the new color information at the identified pixel positions.

28. The method according to any one of claims 21 to 27, characterized in that: The reference image is any other image except the last image in the time window.

29. The method according to any one of claims 21 to 27, characterized in that: The step of sequentially performing normalization processing and binarization processing on the partial derivative image to obtain a binarized image of the partial derivative image includes: Normalizing the values ​​of each pixel in the partial derivative image to obtain a normalized partial derivative image; According to a preset binarization threshold, the normalized partial derivative image is binarized to obtain a binarized image of the partial derivative image, wherein the value of each pixel in the binarized image is 0 or 1; The determining whether an action occurs in the first video segment according to the sum of the grayscale values ​​of each pixel in the binary image includes: Calculating the sum of the grayscale values ​​of each pixel in the binary image; When the sum of the grayscale values ​​is greater than 0, it is determined that an action occurs in the first video segment.

30. The method according to any one of claims 25 to 27, characterized in that: The fusing the first recognition result and the second recognition result to obtain the gesture recognition result of the first video segment includes: Calculating an average value of the first recognition result and the second recognition result, and obtaining a gesture recognition result of the first video segment according to a calculation result of the average value calculation; or, The first recognition result and the second recognition result are input into a pre-trained second machine learning model to obtain a gesture recognition result of the first video segment.

31. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an executable program, and the executable program is executed by a processor to implement the gesture recognition method according to any one of claims 21 to 30.

Images