Gesture recognition system and gesture recognition method

Abstract

A gesture recognition system and a gesture recognition method are disclosed. The gesture recognition system comprises a skeleton key point extraction module, a skeleton vector construction module, a time sequence feature extraction module and a full connection layer connected in sequence. The skeleton key point extraction module is used to extract a first skeleton key point set of a human body in a first image based on a detector and a tracker. The skeleton vector construction module is used to determine a first skeleton vector set based on part of the first skeleton key points in the first skeleton key point set, and determine a length feature of a plurality of first skeleton vectors based on a torso length of the human body in the first image. The time sequence feature extraction module is used to extract a time sequence feature of a skeleton vector of the human body in a video. The full connection layer is used to classify a gesture of the human body in the first image based on the time sequence feature of the skeleton vector of the human body in the video. The gesture recognition system can realize gesture recognition at low cost, high efficiency and high accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and in particular to a gesture recognition system and a gesture recognition method. Background Art

[0002] At intersections controlled by traffic police, autonomous vehicles must accurately recognize and obey the officers' gestures to maintain traffic order and safety. In everyday driving, interactions between vehicles and pedestrians are also inevitable. Therefore, gesture recognition systems that can intelligently interpret the gestures of both traffic police and pedestrians have become an essential component of intelligent driving technology.

[0003] In related technologies, a gesture recognition system includes: a skeleton key point recognition sensor and a gesture recognition module. The skeleton key point sensor is used to recognize the key points of the human skeleton, and the gesture recognition module is used to perform gesture recognition based on the recognized key points of the human skeleton.

[0004] However, skeleton key point sensors are expensive, which leads to high costs for gesture recognition systems. Summary of the Invention

[0005] The present disclosure provides a gesture recognition system and a gesture recognition system that can realize gesture recognition at low cost, efficiently and accurately. The technical solution includes at least the following solutions:

[0006] In a first aspect, a gesture recognition system is provided, including a skeleton key point extraction module, a skeleton vector construction module, a temporal feature extraction module and a fully connected layer connected in sequence, wherein the skeleton key point extraction module is used to extract a first skeleton key point set of a human body in a first image based on a detector and a tracker, wherein the first skeleton key point set includes multiple first skeleton key points, the detector is used to determine a first ROI region of interest in the first image, the first ROI is an area where the first skeleton key points exist, the tracker is used to determine the first skeleton key point set based on the first ROI, and the first image is an image of a human body in the first frame of a video; the skeleton vector construction module is used to extract a first skeleton key point set based on the first skeleton key point set. The first skeleton key points in the image are used to determine a first skeleton vector set, wherein the first skeleton vector set includes multiple first skeleton vectors, and the multiple first skeleton vectors constitute a first human skeleton vector image, and based on the torso length of the human body in the first image, the length characteristics of the multiple first skeleton vectors are determined; the temporal feature extraction module is used to extract the temporal characteristics of the skeleton vector of the human body in the video based on the length characteristics of the skeleton vector of the human body in each frame of the continuous multi-frame images, and the continuous multi-frame images are continuous multi-frame images in the video, and the continuous multi-frame images include the first image; the fully connected layer is used to classify the gestures of the human body in the first image based on the temporal characteristics of the skeleton vector of the human body in the video.

[0007] Optionally, the tracker includes an encoder, a decoder and a regression encoder, and the skeletal key point extraction module is used to implement the extraction of skeletal key points of the human body in the first image based on the detector and the tracker in the following manner: based on the detector, with the hip center of the human body in the first image as a reference, determine the size of the first human body circumscribed circle, and rotate the first human body circumscribed circle so that in the rotated first human body circumscribed circle, the line connecting the shoulder center and the hip center of the human body in the first image is perpendicular to the ground, thereby obtaining the first ROI; based on the encoder, determine all first joint features of the human body in the first ROI; based on the decoder, determine a heat map of each first joint feature, and the heat map of each first joint feature is used to make a first prediction of the first joint feature; based on the regression encoder, make a second prediction based on the heat map of each first joint feature to determine the first skeletal key point set.

[0008] Optionally, the skeletal key point extraction module is also used to: based on the tracker, determine a second ROI in the second image based on the first skeletal key point set, the second image is the next frame image of the first image, and the second ROI is the area where the second skeletal key points of the human body in the second image exist; based on the encoder, determine all second joint features of the human body in the second ROI; based on the decoder, determine a heat map of each second joint feature, and the heat map of each second joint feature is used to perform a first prediction of the second joint feature; based on the regression encoder, perform a second prediction based on the heat map of each second joint feature to determine the second skeletal key point set of the human body in the second image.

[0009] Optionally, the multiple first bone vectors include: a first left upper arm vector, a first right upper arm vector, a first left forearm vector, a first right forearm vector, a first left torso vector, a first right torso vector, a first left thigh vector, a first right thigh vector, a first left calf vector and a first right calf vector.

[0010] Optionally, the gesture recognition system is connected to a training module, which is used to preprocess multiple videos in a training data set, each of which includes multiple frames of temporally continuous images; wherein the training module is used to implement the preprocessing of multiple videos in the training data set in the following manner: for the third image in the first video, the third image is divided into multiple sub-blocks arranged in an array, each sub-block is different from each other, the first video is any one of the multiple videos, and the third image is any frame image in the first video; grayscale histogram equalization is performed on the multiple sub-blocks to obtain a grayscale mapping table for each sub-block; based on the grayscale mapping table of each sub-block, the grayscale value of each pixel in the third image is determined.

[0011] Optionally, the training module is used to implement the grayscale mapping table based on each sub-block in the following manner to determine the grayscale value of each pixel in the third image: determine the first dependent sub-block of the first pixel, where the first pixel is any pixel in the first sub-block, and the first sub-block is any one of the multiple sub-blocks; based on the first dependent sub-block, determine the second dependent sub-block, the third dependent sub-block and the fourth dependent sub-block; based on the grayscale value of the first pixel before the grayscale histogram equalization processing and the grayscale mapping table of each sub-block, determine the grayscale mapping value of the first pixel in the first dependent sub-block, the second dependent sub-block, the third dependent sub-block and the fourth dependent sub-block respectively to obtain four grayscale mapping values; based on the four grayscale mapping values, perform bilinear interpolation on the grayscale value of the first pixel before the grayscale histogram equalization processing to obtain the grayscale value of the first pixel.

[0012] Optionally, the training module is further configured to implement the preprocessing of the multiple videos in the training data set in the following manner:

[0013] Performing bilateral filtering denoising processing on the pixel value of each pixel in the third image based on a Gaussian filter in the spatial domain and a Gaussian filter in the value range to obtain a corrected pixel value of each pixel;

[0014] The Gaussian filter in the spatial domain is implemented using the following formula:

[0015]

[0016] The Gaussian filter of the range is implemented using the following formula:

[0017]

[0018] Among them, G s is a Gaussian filter in the spatial domain, G r is a Gaussian filter with a value range, e is a natural base, o is the coordinate of pixel o, q is the coordinate of pixel q, the pixel o and the pixel q are any two adjacent pixels in the third image in the spatial domain, I p is the pixel value of pixel p, I q is the pixel value of pixel q, is the smoothing parameter.

[0019] Optionally, each of the multiple videos in the training data set corresponds to a label in a label set, the label set is synchronized with the timing of the multiple videos, and the labels in the label set are used to mark multiple frames of images in the video containing gestures; the training module is also used to: when training the gesture recognition system based on the multiple videos, adjust the timing of the label set and the multiple videos so that the timing of the label set lags behind the timing of the multiple videos by a first time length.

[0020] Optionally, the skeleton vector construction module is also used to determine the angular characteristics of each first skeleton vector based on the sine value and cosine value of each first skeleton vector and the gravity vector; the timing feature extraction module is also used to extract the timing characteristics of the skeleton vector of the human body in the video based on the length characteristics and angle characteristics of the skeleton vector of the human body in each frame image in multiple consecutive frames of images.

[0021] In a second aspect, a gesture recognition method is provided, comprising: inputting a video to be recognized into the gesture recognition system described in the first aspect, and obtaining a gesture type of the video to be recognized.

[0022] The beneficial effects of the technical solutions provided by the embodiments of the present disclosure include at least:

[0023] In the disclosed embodiment, the gesture recognition system 100 extracts a set of first skeletal key points of a human body from a first image using the detector and tracker in the skeletal key point extraction module 11. Compared to skeletal point sensors, this approach is more economical, suitable for mobile devices, and ensures the speed of skeletal point recognition. The skeletal vector construction module 12 determines the length features of multiple first skeletal vectors based on the torso length of the human body in the first image, thereby improving the accuracy of the determined length features. The fully connected layer 14 classifies gestures based on the temporal features of the first skeletal vectors, ensuring recognition accuracy while effectively reducing computational costs, avoiding reliance on high-performance hardware, and making it more suitable for real-time mobile scenarios.

[0024] In addition, gesture recognition through skeleton point data can reduce the impact of lighting, background noise, etc. on the recognition results, express human body movements more simply and completely, and obtain higher robustness on small sample data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A schematic structural diagram of a gesture recognition system provided by an exemplary embodiment of the present disclosure is shown;

[0027] Figure 2 is a schematic diagram of gesture types;

[0028] Figure 3 A structural diagram of a gesture recognition system provided by another exemplary embodiment of the present disclosure is shown;

[0029] Figure 4 This is a workflow diagram of the skeleton key point extraction module;

[0030] Figure 5 It is a schematic diagram of determining the length features and angle features of multiple first bone vectors. DETAILED DESCRIPTION

[0031] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning understood by persons of ordinary skill in the field to which the present disclosure belongs. The words “first”, “second”, “third” and similar terms used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as “a” or “an” do not indicate a quantity limitation, but rather indicate the presence of at least one. Words such as “include” or “comprise” mean that the elements or objects appearing before “include” or “comprises” include the elements or objects listed after “include” or “comprises” and their equivalents, and do not exclude other elements or objects. Words such as “connect” or “connected” are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Up”, “down”, “left”, “right” and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0032] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0033] Figure 1 A schematic diagram of the structure of a gesture recognition system provided by an exemplary embodiment of the present disclosure is shown. Figure 1 The gesture recognition system 100 includes a skeleton key point extraction module 11, a skeleton vector construction module 12, a temporal feature extraction module 13 and a fully connected layer 14, which are connected in sequence.

[0034] The skeleton key point extraction module 11 is used to extract a first skeleton key point set of a human body in a first image based on a detector and a tracker.

[0035] The first skeleton key point set includes a plurality of first skeleton key points. The first skeleton key point set is a set of all skeleton key points of the human body in the first image.

[0036] Among them, the detector is used to determine the first ROI (Region Of Interest) in the first image, the first ROI is the area where the first skeleton key point exists, the tracker is used to determine the first skeleton key point set based on the first ROI, and the first image is the image of the human body in the first frame of the video.

[0037] For example, the detector and tracker can be implemented based on the BlazePose network within the MediaPipe framework. BlazePose is a lightweight convolutional neural network architecture suitable for single-person skeletal keypoint detection, offering fast recognition speed. The implementation of BlazePose is well documented in related technologies and will not be detailed here.

[0038] The first image is the first frame in a video containing a human figure. Typically, gesture recognition is performed based on the video. After a gesture recognition system obtains a video, it can split the video into multiple temporally consecutive frames. It must first determine that the first frame containing a human figure is the first image before subsequent gesture recognition can proceed. In the disclosed embodiment, a detector is used to determine the first image among these multiple frames.

[0039] Optionally, the detector determines the first image in multiple frames of images in the following manner: identify whether there is a face in multiple frames of images in the video in chronological order; if there is no face in a frame image, identify whether there is a face in the next image, until a frame image with a face is identified, which is the first image.

[0040] The skeleton vector construction module 12 is used to determine the first skeleton vector set based on some of the first skeleton key points in the first skeleton key point set, where the first skeleton vector set includes multiple first skeleton vectors, and the multiple first skeleton vectors constitute a first human skeleton vector map, and determine the length characteristics of the multiple first skeleton vectors based on the torso length of the human body in the first image.

[0041] Here, multiple first bone vectors constitute a first human skeleton vector map, that is, multiple first bone vectors are connected into the first human skeleton vector map according to the joint hinge structure of the human body.

[0042] Optionally, when the first skeletal key point set includes a nasal bone key point, a left hip key point and a right hip key point, the torso length of the human body in the first image can be the length of the line between the midpoint of the line connecting the left hip key point and the right hip key point in the first skeletal key point set and the nasal bone key point.

[0043] The temporal feature extraction module 13 is used to extract the temporal features of the human body's bone vectors in the video based on the length features of the human body's bone vectors in each frame of the continuous multi-frame images.

[0044] The continuous multi-frame images are continuous multi-frame images in a video, and the continuous multi-frame images include the first image.

[0045] Here, for the consecutive multiple frames in a video, the human body in each frame is the same human body, that is, the human body in the first image. For each frame in the consecutive multiple frames, the same processing method as the first image can be used to obtain the length characteristics of the human bone vector in each frame.

[0046] The fully connected layer 14 is used to classify the gestures of the human body in the first image based on the temporal features of the skeleton vectors of the human body in the video.

[0047] Figure 2 is a schematic diagram of gesture types, such as Figure 2 As shown in the figure, there are 10 types of gestures, including 8 traffic police gestures and 2 pedestrian gestures. In order from (1) to (10), these 10 gestures are stop, go straight, turn left, wait for left turn, turn right, slow down, change lanes, pull over, wave to stop, and wave to stop.

[0048] In the disclosed embodiment, the gesture recognition system 100 extracts a first set of skeletal key points of a human body in a first image through the detector and tracker in the skeletal key point extraction module 11. Compared to skeletal point sensors, this method is more economical and suitable for mobile devices, and the recognition speed of skeletal points can also be guaranteed. The skeletal vector construction module 12 determines the length features of multiple first skeletal vectors based on the torso length of the human body in the first image, thereby improving the accuracy of the determined length features. The temporal feature extraction module accurately extracts the temporal features of the human body in the video, and the fully connected layer 14 classifies gestures based on the temporal features of the human body in the video. This ensures recognition accuracy while effectively reducing computational costs, avoiding dependence on high-performance hardware, and is more suitable for real-time mobile scenarios.

[0049] In addition, gesture recognition through skeleton point data can reduce the impact of lighting, background noise, etc. on the recognition results, express human body movements more simply and completely, and obtain higher robustness on small sample data sets.

[0050] Figure 3 A schematic diagram of the structure of a gesture recognition system provided by another exemplary embodiment of the present disclosure is shown. Figure 3 The gesture recognition system 100 includes a skeleton key point extraction module 11, a skeleton vector construction module 12, a temporal feature extraction module 13 and a fully connected layer 14, which are connected in sequence, and the gesture recognition system 100 is also connected to a training module 15.

[0051] The training module 15 is used to pre-process multiple videos in the training data set.

[0052] Each video includes multiple temporally consecutive image frames.

[0053] In the embodiment of the present disclosure, the multiple videos in the training data set include videos of 8 types of traffic police gestures and 2 types of pedestrian gestures. In other words, the gesture types in the videos in the training data set are the same as the gesture types that the gesture recognition system needs to classify.

[0054] Optionally, the training data set may also include videos of gestures in complex environments, such as videos of different gestures under different lighting conditions and backgrounds. For example, videos of different gestures in complex environments such as complex intersections, facing away from the sun, and under dappled tree shadows, which may affect the clear presentation of the human body.

[0055] The sizes and frame rates of these multiple videos may differ, resulting in different image sizes and numbers in the two videos. Therefore, these multiple videos need to be normalized so that they have the same size and frame rate. For example, the size of each video can be adjusted to a first size, and the frame rate of each video can be adjusted to the first frame rate. For videos smaller than the first size, the videos are padded to the first size; and for videos larger than the first size, the videos are cropped to the first size.

[0056] For example, the first size can be 1080 by 1080, meaning that any frame image in any video includes 1080 rows of pixels and 1080 columns of pixels. The first frame rate can be 15 frames per second. A frame rate that is too low will result in the inability to extract continuous motion trajectory information from the video, while a frame rate that is too high will affect the final judgment due to redundant details. By uniformly adjusting the frame rate of each video to 15 frames per second, the integrity of the information can be ensured while avoiding excessive redundant details.

[0057] Optionally, the multiple videos in the training data set can be divided into a training set and a test set in a ratio of 7:3.

[0058] Optionally, the training module 15 is configured to pre-process the multiple videos in the training data set using steps ac:

[0059] Step a: for a third image in the first video, divide the third image into a plurality of sub-blocks arranged in an array.

[0060] Each sub-block is the same size and contains different images. The first video is any one of the multiple videos, and the third image is any frame image in the first video. The third image can be the same as or different from the first image, and this is not limited in the present embodiment.

[0061] Step b: performing grayscale histogram equalization processing on multiple sub-blocks to obtain a grayscale mapping table for each sub-block.

[0062] For any of the multiple sub-blocks, the grayscale histogram is used to represent the number of pixels with a certain grayscale level in the sub-block. First, the original grayscale value of each pixel in the sub-block can be determined, and thus the cumulative distribution probability of each grayscale level in the sub-block can be determined. Based on the cumulative distribution probability, the grayscale mapping function of the sub-block is constructed. Based on the grayscale mapping function, the original grayscale histogram of the sub-block, that is, the original grayscale mapping table, can be determined.

[0063] If the grayscale distribution of the sub-block is uneven, for example, the grayscale distribution of the sub-block is concentrated in a narrow range, the contrast of the sub-block is low and the details are not clear. However, by applying grayscale histogram equalization to the sub-block, the grayscale distribution of the sub-block can be made uniform, thereby increasing the contrast and making the details of the sub-block clearer, thereby enhancing the sub-block.

[0064] Grayscale histogram equalization processing is to set a grayscale threshold in the process of determining the mapping function of the sub-block. If the grayscale value of a grayscale level in the sub-block is greater than the grayscale threshold, the part exceeding the grayscale threshold will be evenly distributed to the other grayscale levels of the sub-block, and finally a grayscale mapping table after equalization processing can be obtained. Grayscale histogram equalization processing can avoid image distortion or increased noise caused by excessive modification of the contrast of certain local areas. The grayscale value of each pixel in the grayscale mapping table after equalization processing is uncertain, and it is necessary to map the grayscale mapping table after equalization processing back to the sub-block through the mapping function of the sub-block to obtain the pixel value of each pixel in the sub-block, and then the grayscale value of each pixel point can be determined based on the pixel value of each pixel point in the sub-block.

[0065] Here, the grayscale mapping table of each sub-block obtained in step b is also the grayscale mapping table of each sub-block after equalization processing.

[0066] Step c: determining the grayscale value of each pixel in the third image based on the grayscale mapping table of each sub-block.

[0067] Optionally, step c is implemented using the following four steps:

[0068] In the first step, a first dependent sub-block of a first pixel is determined, where the first pixel is any pixel in the first sub-block, and the first sub-block is any one of the multiple sub-blocks.

[0069] Optionally, the first dependent sub-block of the first pixel is determined in the following manner:

[0070] Assume the coordinates of the first pixel are (m, n), that is, the first pixel is the pixel in the mth row and nth column of the first sub-block. Subtract the row coordinates of the first pixel from Column coordinates minus Get the coordinates of the second pixel The sub-block where the second pixel is located is the first dependent sub-block. Wherein, A is the number of rows of all pixels in the first sub-block, and B is the number of columns of all pixels in the first sub-block.

[0071] In the second step, based on the first dependent sub-block, a second dependent sub-block, a third dependent sub-block and a fourth dependent sub-block are determined.

[0072] Assume that the first dependent sub-block is P(x, y), where x represents the xth row of the first dependent sub-block among the multiple sub-blocks, and y represents the yth column of the first dependent sub-block among the multiple sub-blocks. Then the second dependent sub-block is P(x+1, y), which means that the second dependent sub-block is adjacent to and to the right of the first dependent sub-block; the third dependent sub-block is P(x, y+1), which means that the third dependent sub-block is adjacent to and above the first dependent sub-block; and the fourth dependent sub-block is P(x+1, y+1), which means that the fourth dependent sub-block is adjacent to both the second and third dependent sub-blocks and does not overlap with the first dependent sub-block.

[0073] In the third step, based on the grayscale value of the first pixel before grayscale histogram equalization processing and the grayscale mapping table of each sub-block, the grayscale mapping values ​​of the first pixel in the first dependent sub-block, the second dependent sub-block, the third dependent sub-block and the fourth dependent sub-block are determined respectively to obtain four grayscale mapping values.

[0074] When the mapping function of each sub-block is obtained in step b, the grayscale mapping values ​​of the first pixel in the first dependent sub-block, the second dependent sub-block, the third dependent sub-block and the fourth dependent sub-block are determined respectively based on the grayscale value of the first pixel before grayscale histogram equalization processing and the grayscale mapping table of each sub-block, including: substituting the pixel value of the first pixel into the mapping functions of the first dependent sub-block, the second dependent sub-block, the third dependent sub-block and the fourth dependent sub-block respectively to obtain four grayscale mapping values.

[0075] In the fourth step, based on the four grayscale mapping values, bilinear interpolation is performed on the grayscale value of the first pixel before the grayscale histogram equalization process to obtain the grayscale value of the first pixel.

[0076] The grayscale value of the first pixel before the grayscale histogram equalization process is also the original grayscale value of the first pixel.

[0077] When performing bilinear interpolation, linear interpolation can be performed first in the x-direction and then in the y-direction; or linear interpolation can be performed first in the y-direction and then in the x-direction. This disclosure does not limit this. The implementation of bilinear interpolation is well known in the art and will not be described in detail here.

[0078] In the disclosed embodiment, a sub-block located at the boundary of the third image cannot determine four dependent sub-blocks, but only two dependent sub-blocks, and thus two grayscale mapping values. In this case, the pixel values ​​in this sub-block only need to be linearly interpolated based on the two grayscale mapping values.

[0079] After executing step c, the grayscale value of each pixel can be obtained. However, there is often noise in the image, which will affect the accuracy of the final calculated result. In order to remove the noise in the image, the training module 15 is also used to pre-process the multiple videos in the training data set in the following manner:

[0080] Based on a Gaussian filter in the spatial domain and a Gaussian filter in the value range, a bilateral filtering denoising process is performed on the pixel value of each pixel in the third image to obtain a corrected pixel value of each pixel.

[0081] In the embodiment of the present disclosure, the bilateral filtering denoising process is implemented based on the filter BF in formula (1).

[0082]

[0083] In formula (1), BF is a filter, p∈S represents any pixel point p in the filter window S, G s (p) represents the Gaussian filter in the spatial domain, G r (p) represents the Gaussian filter of the range, I p is the pixel value of pixel point p, W q is the weighted sum of each pixel value in the filter window S, used for weight normalization, W q =∑ p∈S G s (p)G r (p).

[0084] Here, the filtering window may be the third image, or any frame image of any one of the multiple videos in the training data set, which is not limited in the embodiment of the present disclosure.

[0085] The Gaussian filter in the spatial domain is implemented using formula (2):

[0086]

[0087] The Gaussian filter of the range is implemented using formula (3):

[0088]

[0089] In formula (2) and formula (3), e is the natural base, p is the coordinate of pixel point p, q is the coordinate of pixel point q, pixel point p and pixel point q are any two adjacent pixels in the third image in the spatial domain, Ip is the gray value of pixel p, I q is the grayscale value of pixel q, is the smoothing parameter.

[0090] In the flat area, the G of each pixel in the filter r The values ​​are similar, then G s Dominant filtering effect; in the edge area, G on the same side of the edge r The values ​​are similar and much larger than G on the other side of the edge r Value, at this time the G of the pixel on the other side r Value and G s The value has almost no effect on the filtering result, so that the edge information is protected. In this way, the filter BF shows a certain degree of adaptability.

[0091] When a noise point appears in a flat area, the weight of the signal around the noise point is very small. After normalization, the weight of the signal around the noise point is increased, so the filter BF also has a filtering effect on the noise point. Through bilateral filtering denoising, the pixel value of each pixel can be corrected, achieving image smoothing while avoiding blurred edges.

[0092] Optionally, the training module 15 is further configured to establish a label set based on the training data set. Each of the multiple videos in the training data set corresponds to a label in the label set, the label set is synchronized with the time sequence of the multiple videos, and the labels in the label set are used to annotate multiple frames of images in the video that contain gestures.

[0093] For example, a video has a frame rate of 15 frames per second, and the video is 15 seconds long. A gesture begins in the first frame at the 3rd second of the video and ends in the third frame at the 12th second. The gesture in the video is a left turn. In the tag set, the tag corresponding to the video is the tag corresponding to the left turn gesture. The tag corresponding to the left turn gesture also begins at the time corresponding to the first frame at the 3rd second (i.e., at the 3.0th second of the video) and disappears at the time corresponding to the third frame at the 12th second (i.e., at the 12.2th second of the video).

[0094] Optionally, the training module 15 is further configured to train the gesture recognition system 100 based on a plurality of videos in the training data set.

[0095] When a label set exists, the training module 15 is further configured to adjust the timing of the label set and the multiple videos when training the gesture recognition system based on the multiple videos, so that the timing of the label set lags behind the timing of the multiple videos by a first time length.

[0096] The timing of the labels in the original label set is synchronized with the timing of the video, meaning that each frame in the video containing a gesture is labeled with a corresponding label. By adjusting the timing of the label set and the multiple videos so that the timing of the label set lags behind the timing of the multiple videos by a first time length, the first N frames in any video containing a gesture are not labeled. Where N is a positive integer.

[0097] For example, N can range from 1 to 30, such as 1, 3, 15, 20, or 30, which is not limited in the present embodiment. The first time length is determined based on N and the frame rate of the video. For example, if the frame rate is 15 frames per second and N is 15, the first time length is 1 second.

[0098] When the gesture recognition system 100 recognizes gestures, errors may occur. For example, a person may simply raise their hand rather than perform a complete gesture, but the gesture recognition system may recognize the person as performing a specific gesture based on this hand-raising motion. Therefore, in the disclosed embodiments, by adjusting the timing between the tag set and multiple videos, recognition errors caused by similar gestures in the early, short period of time are avoided, thereby improving the accuracy and efficiency of gesture recognition.

[0099] The skeleton key point extraction module 11 is used to extract a first skeleton key point set of a human body in a first image based on a detector and a tracker.

[0100] Optionally, the tracker includes an encoder, a decoder, and a regression encoder. The skeleton key point extraction module 11 is configured to extract skeleton key points of a human body in the first image based on the detector and the tracker using the following four steps:

[0101] In the first step, based on the detector, the size of the first human body circumscribed circle is determined with the hip center of the human body in the first image as the reference, and the first human body circumscribed circle is rotated so that in the rotated first human body circumscribed circle, the line connecting the shoulder center and the hip center of the human body in the first image is perpendicular to the ground, thereby obtaining the first ROI.

[0102] After the first image is input into the detector, the detector can first determine the hip center of the human body, and then determine a rough circumscribed circle of the human body based on the hip center. The circumscribed circle is the ROI of the human body in the first image.

[0103] Different human bodies may have different postures. For example, when some pedestrians make gestures, their upper bodies may be tilted instead of perpendicular to the ground. However, in the traffic police's action specifications, the upper body of the traffic police is perpendicular to the ground, which will lead to differences in the angles of the upper body of the identified human body. Therefore, after determining the human body's circumscribed circle, it is also necessary to determine whether the line connecting the center of the shoulder and the center of the hip of the human body is perpendicular to the ground. If it is perpendicular to the ground, no processing is required. If it is not perpendicular to the ground, it is necessary to rotate the first human body's circumscribed circle so that in the rotated first human body's circumscribed circle, the line connecting the center of the shoulder and the center of the hip of the human body in the first image is perpendicular to the ground. In this way, the upper bodies of the identified human bodies are all perpendicular to the ground, which facilitates subsequent unified recognition in the tracker. This action is also called human body alignment.

[0104] In the second step, based on the encoder, all first joint features of the human body in the first ROI are determined.

[0105] The first joint feature is also the coordinate range of the possible bone points in the ROI.

[0106] In the third step, based on the decoder, a heatmap of each first joint feature is determined, and the heatmap of each first joint feature is used to make a first prediction of the first joint feature.

[0107] The decoder determines the heat map of each first joint feature, that is, converts the coordinate range of possible joints in the ROI into the form of a heat map.

[0108] The heat map obtained in this step can be converted into a rough coordinate range, which needs further processing to obtain a more accurate coordinate of the skeleton key point, that is, the first prediction of the first joint feature.

[0109] In the fourth step, based on the regression encoder, a second prediction is made based on the heat map of each first joint feature to determine the first skeleton key point set.

[0110] The regression encoder can determine the coordinates, offsets, and confidence levels of the skeletal keypoints in each first joint based on the heat map of each first joint feature. Here, when determining the coordinates of the skeletal keypoints, the regression encoder actually outputs the position corresponding to the maximum probability of a skeletal keypoint appearing as the coordinate of the skeletal keypoint. This maximum probability is also the confidence level of the coordinates of the skeletal keypoint.

[0111] For example, the first skeleton key point set includes 33 skeleton key points, and each skeleton key point is stored in the skeleton key point set in the form of (x, y, confidence). In this way, the first skeleton key point set includes 33 times 3 key data. Here, (x, y, confidence) indicates that the row coordinate of a skeleton key point in the first image is x, the column coordinate is y, and the confidence is confidence.

[0112] For any video, the video includes multiple frames of images, each frame of image corresponds to a skeleton key point set, and the skeleton key point sets corresponding to the multiple frames of image of the same video can be stored in the same file.

[0113] In the disclosed embodiments, both the detector and the tracker are capable of determining whether a person exists in an image. For example, if a person does not exist in an image, the tracker will output a detection failure result, indicating that a person does not exist in the image. Furthermore, if the tracker detects that a person does not exist in a frame, the next frame after that frame is fed into the detector. This continues until the detector detects an image with a person. At that point, the image with the person present is fed into the tracker, and the next frame after that image with the person present is fed directly into the tracker without passing through the detector.

[0114] Optionally, the skeletal key point extraction module 11 is also used to: based on the tracker, determine the second ROI in the second image based on the first skeletal key point set, the second image is the next frame image of the first image, and the second ROI is the area where the second skeletal key points of the human body in the second image exist; based on the encoder, determine all second joint features of the human body in the second ROI; based on the decoder, determine the heat map of each second joint feature, and the heat map of each second joint feature is used to perform a first prediction of the second joint feature; based on the regression encoder, perform a second prediction based on the heat map of each second joint feature to determine the second skeletal key point set of the human body in the second image.

[0115] Determining the second ROI in the second image based on the first skeleton key point set includes: adding 10% of the human body motion range to the first skeleton key point set to obtain the second ROI.

[0116] The processing of the second image by the tracker is, that is, the processing of the next frame of the image containing the human body by the tracker when the next frame of the image containing the human body is directly input into the tracker without passing through the detector.

[0117] In common human pose estimation algorithms, to achieve higher prediction accuracy, most key point extraction methods have turned to heatmap-based techniques rather than earlier direct regression algorithms. This heatmap-based pose estimation algorithm maintains good results even in multi-person recognition scenarios, but its high computational overhead poses a challenge in real-time inference applications on mobile devices. BlazePose, used in this solution, provides a lighter-weight pose estimation solution that combines heatmap and regression methods, significantly reducing computational overhead while ensuring accuracy. It not only supports GPU computing but also achieves near real-time operation in CPU environments, successfully meeting the needs of real-time human skeletal point detection in complex backgrounds in intelligent driving.

[0118] Figure 4 This is a workflow diagram of the skeleton key point extraction module. Figure 4 Part (a) is a schematic diagram of the workflow of the skeleton key point extraction module for extracting skeleton key points from a single frame image. Figure 4 Part (b) is a schematic diagram of the workflow of the skeleton key point extraction module to extract skeleton key points from the video.

[0119] like Figure 4 As shown in part (a), when extracting skeleton key points from a single-frame image, the single-frame image is first sent to the detector to extract the ROI of the image, and then the ROI is sent to the tracker. The encoder and decoder in the tracker are used to extract the heat map, and the offset is determined by the regression encoder to finally determine the coordinates of each skeleton key point.

[0120] like Figure 4 As shown in part (b), when extracting skeletal key points from a video, the images in the video are first fed into the detector frame by frame for human presence detection. If the detector does not detect a human body, that is, the detection fails, the next frame of the image is fed into the detector; if the detector detects a human body, for example, a human body is detected from the first image, the size of the human body circumscribed circle of the first image is determined, and the human body is aligned to obtain the ROI area. The detector then feeds the ROI area into the tracker for processing. The tracker performs skeletal key point detection based on the ROI, and performs ROI estimation on the next frame of the first image, while performing human presence detection. If the result of the human presence detection of a certain frame is a detection failure, that is, no human body is detected, the next frame of the frame is fed into the detector.

[0121] Optionally, the skeleton vector construction module 12 is used to determine a first human skeleton set based on some first skeleton key points in the first skeleton key point set, where the first human skeleton set includes multiple first bones, and the multiple first bones constitute a first human skeleton map.

[0122] Based on the first human skeleton diagram, when the human body is in the shape of a "big" character, the direction of each first bone can be determined according to the rule that the bone endpoint close to the center of the hip is the starting point and the endpoint away from the center of the hip is the end point. Each first bone with a direction is a first bone vector, thereby obtaining multiple first bone vectors in the first bone vector set, and these multiple first bone vectors constitute the first bone vector diagram.

[0123] In the embodiment of the present disclosure, the difference between each first bone and the corresponding first bone vector is that the first bone vector has one more direction than the first bone, and the rest are the same, for example, the length characteristics of the two are the same. Therefore, the length characteristics of each first bone can be determined first, and the length of the first bone can be used as the length of the first bone vector. Alternatively, multiple first bone vectors can be determined first, and then the length characteristics of each first bone vector can be determined. The embodiment of the present disclosure does not limit this.

[0124] Optionally, the length of each first bone is determined by formula (4).

[0125]

[0126] In formula (4), F1 is the length feature, s is any first bone, S is the first human skeleton set, s i is the i-th first bone in the first human skeleton set, i is a positive integer and i is less than 11; s body The torso length is represented by the length of the line between the midpoint of the line connecting the left hip keypoint and the right hip keypoint in the first bone keypoint set and the line connecting the nose bone keypoint.

[0127] Here, since the torso is typically upright and its length is relatively constant across different gestures, torso length can be used as a benchmark for determining length features. Furthermore, since traffic police gesture standards require the torso to always be upright, using torso length as a benchmark for determining length features when determining traffic police gestures can make the determination of gesture types more accurate.

[0128] In the disclosed embodiment, by using the torso length as the reference data for calculating the length feature, the gesture feature is not affected by changes in viewing angle and human body movement. Furthermore, the torso length is substantially greater than the lengths of other bones. When calculating the length feature, the longer torso length is used as the denominator. When slight errors occur between images of different frames, these errors are almost negligible relative to the torso, having little impact on the calculation results, resulting in more accurate results. However, if other shorter bones are used as the denominator when calculating the length feature, the errors relative to the other shorter bones are non-negligible, resulting in larger errors in the calculation results.

[0129] Optionally, the skeleton vector construction module 12 is further used to determine the angle feature of each first skeleton vector based on the sine value and cosine value of each first skeleton vector and the gravity vector.

[0130] Optionally, formula (5) is used to determine the angle feature of each first bone vector:

[0131]

[0132] In formula (5), F2 is the angle feature, v is any first bone vector in the first bone vector set, V is the first bone vector set, and v i is the i-th first bone vector in the first bone vector set, and g is the unit gravity vector, that is, the unit vector in the direction of gravity. The meanings of the other parameters in formula (5) are the same as those in formula (4), and their detailed description is omitted here.

[0133] in, Represents the cosine value of the first bone vector and the gravity vector, Represents the sine of the first bone vector and the gravity vector.

[0134] In practical applications, since the human body captured by a moving vehicle is usually perpendicular to the ground, the image is also horizontal to the ground. Therefore, the unit gravity vector can be used as the reference for the angle feature.

[0135] Figure 5 is a schematic diagram for determining the length features and angle features of multiple first bone vectors. Figure 5 As shown in part (a), the bone key points used to construct the first bone vector set include: right shoulder key point Y 11 , left shoulder key point Y 12 , right elbow key point Y 13 , left elbow key point Y 14 , right wrist key point Y 15 , left wrist key point Y 16 , right hip key point Y 23 , left hip key point Y 24 , right knee key point Y 25 , left knee key point Y 26 , right ankle key point Y 27 , left ankle key point Y 28 The torso length passes through the right hip key point Y 23 and the left hip key point Y 24 It is determined by the length of the line connecting the midpoint of and the nose key point Y0.

[0136] Based on these 12 first skeleton key points, 10 first skeletons can be determined, including: left upper arm skeleton S1, left forearm skeleton S2, left trunk skeleton S3, left thigh skeleton S4, left calf skeleton S5, right upper arm skeleton S6, right forearm skeleton S7, right trunk skeleton S8, right thigh skeleton S9, right calf skeleton S10, right lower arm skeleton S11, right lower arm skeleton S12, right lower arm skeleton S13, right lower arm skeleton S14, right lower arm skeleton S15, right lower arm skeleton S16, right lower arm skeleton S17, right lower arm skeleton S18, right lower arm skeleton S19, right lower arm skeleton S20, right lower arm skeleton S21, right lower arm skeleton S22 10 .

[0137] like Figure 5 As shown in part (b), the first skeleton vectors in the first skeleton vector set include: a first left upper arm vector V1, a first right upper arm vector V6, a first left forearm vector V2, a first right forearm vector V7, a first left trunk vector V3, a first right trunk vector V8, a first left thigh vector V4, a first right thigh vector V9, a first left calf vector V5 and a first right calf vector V 10 It can be seen that the bone endpoint of each first bone vector close to the hip center O is the starting point, and the endpoint away from the hip center O is the end point.

[0138] The temporal feature extraction module 13 is used to extract the temporal features of the human body's bone vector in the video based on the length features and angle features of the human body's bone vector in each frame of the continuous multi-frame images.

[0139] In the disclosed embodiment, the temporal feature extraction module 13 is a LSTM (Long Short Time Memory) network.

[0140] Optionally, the temporal feature extraction module 14 is configured to extract the temporal features of the human body's bone vector in the video based on the length features and angle features of the human body's bone vector in each frame of the continuous multi-frame image in the following manner:

[0141] For the first and second steps of the following three steps, the temporal feature extraction module 14 performs the same processing actions on each frame of image, so the first and second steps are explained using the first image as an example. For other images in the multiple frames of image, the processing method of the first image can be referred to.

[0142] In the first step, the length feature and the angle feature of the first bone vector are matrix-concatenated to obtain the first feature of the first bone vector.

[0143] Optionally, the first step is implemented based on formula (6).

[0144]

[0145] In formula (6), F is the first feature, Indicates matrix concatenation of length features and angle features.

[0146] In the second step, the first feature is input into the LSTM network.

[0147] The LSTM network is composed of multiple LSTM units. The working principle of any LSTM unit can be expressed by formula (7).

[0148]

[0149] In formula (7), C t is the content saved by the LSTM unit, h t-1 is the output of the LSTM unit at the previous moment, C t-1 is the content saved by the LSTM unit at the previous moment, F t is the vector input to the LSTM unit, for example, it can be the first feature of a first bone vector, σ is the sigmoid activation function; W1, W2 and W3 are weight parameters, b1, b2 and b3 are bias parameters, W1, W2, W3, b1, b2 and b3 are all parameters that can be obtained by training the LSTM network.

[0150] After each frame of image is processed in the above manner, the LSTM network can extract the temporal features of the human body's bone vector in the video based on the length features and angle features of the human body's bone vector in each frame of image.

[0151] The third step is to determine the temporal features of the first skeleton vector based on the LSTM network.

[0152] The temporal features of the human skeleton vector in the video extracted by the LSTM network are the output h of the current layer of LSTM. t . Optionally, h t Calculated by formula (8).

[0153] h t =σ(W4·[h t-1 +F t ]+b4·tanh(C t ) (8)

[0154] In formula (8), W4 is similar to W1, W2, and W3 and is also a weight parameter. b4 is similar to b1, b2, and b3 and is also a bias parameter. W4 and b4 are also parameters that can be obtained by training the LSTM network. The meanings of the other parameters in formula (8) are the same as in formula (7) and are not described in detail here.

[0155] In the disclosed embodiment, the temporal feature extraction module 14 is implemented through an LSTM network, which fully utilizes the spatial feature information of the gesture (i.e., the length feature and the angle feature). On the one hand, this ensures the accuracy of recognition, and on the other hand, it effectively reduces the computing cost and avoids the dependence on high-performance hardware, thereby making the gesture recognition system 100 more suitable for real-time mobile scenarios.

[0156] The temporal feature extraction module 13 determines the temporal features of the skeleton vectors of the human body in the video, and then inputs the temporal features into the fully connected layer 14 to obtain the classification results of the gestures.

[0157] The fully connected layer 14 is used to classify the gestures of the human body in the first image based on the temporal features of the skeleton vectors of the human body in the video. Here, the temporal features of the skeleton vectors of the human body in the video are also h t .

[0158] The fully connected layer 14 can transform h t Mapped to the predefined 10 gesture categories and output h t For the probability distribution of different categories, the category with the largest probability is h t The corresponding human gesture type in the video is also the gesture recognition result.

[0159] Optionally, the parameters in the gesture recognition system 100 can be adjusted using an Adam optimizer. The Adam optimizer helps the model effectively navigate in the parameter space through an adaptive learning rate mechanism, thereby accelerating the convergence process and improving training efficiency.

[0160] To ensure that the gesture recognition system 100 does not overfit the training data during training, the gesture recognition system 100 optionally includes a dropout layer (not shown). The dropout layer randomly drops a certain proportion of neuronal connections with a set probability in each iteration, thereby reducing the model's sensitivity to noise in the training data and thereby enhancing the model's generalization ability when faced with unknown data.

[0161] Optionally, when the training module 15 trains the gesture recognition system 100, the difference between the probability distribution output by the gesture recognition system 100 and the true label can be measured by a cross-entropy loss function. The difference can be expressed as a loss. Generally speaking, the smaller the loss, the better the recognition effect of the gesture recognition system 100. During the training process of the gesture recognition system 100, the loss will continue to decrease.

[0162] There are many methods for implementing the cross entropy loss function in related technologies, so detailed description is omitted here.

[0163] In the embodiment of the present disclosure, during the process of training the gesture recognition system 100, a training session may be performed first to obtain a first number of training rounds (e.g., 100,000 rounds) at which the loss no longer decreases. When the loss no longer decreases, it indicates that the gesture recognition system 100 has been trained.

[0164] When subsequently training the gesture recognition system 100 , the maximum number of training rounds may be set based on the first number of training rounds, for example, the maximum number of training rounds may be set to the first number of training rounds or a value greater than the first number of training rounds.

[0165] It should be noted that when the gesture recognition device provided in the above embodiment performs gesture recognition, the division of the above functional modules is only used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0166] The division of modules in the embodiments of the present disclosure is illustrative and represents only a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present disclosure may be integrated into a single processor, exist physically as separate modules, or be integrated into a single module. The integrated modules may be implemented in either hardware or software functional modules.

[0167] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a terminal device (which can be a personal computer, mobile phone, or communication device, etc.) or a processor (processor) to execute all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0168] The embodiment of the present disclosure further provides a gesture recognition method, including: inputting a video to be recognized into the gesture recognition system 100 to obtain a gesture type of the video to be recognized.

[0169] The above description is merely an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A gesture recognition system, characterized in that: The gesture recognition system includes a skeleton key point extraction module, a skeleton vector construction module, a temporal feature extraction module and a fully connected layer, which are connected in sequence. The skeleton key point extraction module is used to extract a first skeleton key point set of a human body in a first image based on a detector and a tracker, wherein the first skeleton key point set includes a plurality of first skeleton key points, the detector is used to determine a first ROI region of interest in the first image, the first ROI being a region where the first skeleton key points exist, the tracker is used to determine the first skeleton key point set based on the first ROI, and the first image is an image of a human body in the first frame of a video; The skeleton vector construction module is used to determine a first skeleton vector set based on some first skeleton key points in the first skeleton key point set, wherein the first skeleton vector set includes a plurality of first skeleton vectors, and the plurality of first skeleton vectors constitute a first human skeleton vector map, and determine length features of the plurality of first skeleton vectors based on a torso length of the human body in the first image; The temporal feature extraction module is used to extract the temporal features of the human body bone vector in the video based on the length features of the human body bone vector in each frame of a continuous multi-frame image, wherein the continuous multi-frame image is a continuous multi-frame image in the video, and the continuous multi-frame image includes the first image; The fully connected layer is used to classify the gestures of the human body in the first image based on the temporal features of the skeleton vectors of the human body in the video; The following formula is used to determine the length feature of each first bone vector: in, is the length feature of the first bone vector, For any first bone, The first human skeleton collection, is the i-th first bone in the first human skeleton set, where i is a positive integer and i is less than 11; The torso length is represented by the length of the line between the midpoint of the line connecting the left hip keypoint and the right hip keypoint in the first bone keypoint set and the line connecting the nose bone keypoint.

2. The gesture recognition system according to claim 1, wherein: The tracker includes an encoder, a decoder, and a regression encoder. The skeleton key point extraction module is used to implement the extraction of skeleton key points of the human body in the first image based on the detector and the tracker in the following manner: Determining, based on the detector, a size of a first human body circumscribed circle with the center of the human body's hips in the first image as a reference, and rotating the first human body circumscribed circle so that, in the rotated first human body circumscribed circle, a line connecting the center of the human body's shoulder in the first image and the center of the human body's hips is perpendicular to the ground, thereby obtaining a first ROI; determining all first joint features of the human body in the first ROI based on the encoder; Determine a heat map of each first joint feature based on the decoder, wherein the heat map of each first joint feature is used to make a first prediction of the first joint feature; Based on the regression encoder, a second prediction is performed on the basis of the heat map of each of the first joint features to determine the first skeleton key point set.

3. The gesture recognition system according to claim 2, characterized in that: The skeleton key point extraction module is also used for: Determining, based on the tracker and the first set of skeletal key points, a second ROI in a second image, where the second image is a frame image next to the first image, and the second ROI is a region where a second skeletal key point of the human body in the second image is located; determining all second joint features of the human body in the second ROI based on the encoder; Determine a heat map of each second joint feature based on the decoder, wherein the heat map of each second joint feature is used to make a first prediction of the second joint feature; Based on the regression encoder, a second prediction is performed based on the heat map of each second joint feature to determine a second skeletal key point set of the human body in the second image.

4. The gesture recognition system according to any one of claims 1 to 3, characterized in that: The multiple first bone vectors include: a first left upper arm vector, a first right upper arm vector, a first left forearm vector, a first right forearm vector, a first left torso vector, a first right torso vector, a first left thigh vector, a first right thigh vector, a first left calf vector and a first right calf vector.

5. The gesture recognition system according to any one of claims 1 to 3, characterized in that: The gesture recognition system is connected to a training module, and the training module is used to pre-process multiple videos in a training data set, each of the videos including multiple frames of temporally continuous images; The training module is used to implement the preprocessing of the multiple videos in the training data set in the following manner: For a third image in a first video, the third image is divided into a plurality of sub-blocks arranged in an array, each sub-block being different from another, the first video being any one of the plurality of videos, and the third image being any frame image in the first video; Performing grayscale histogram equalization processing on the multiple sub-blocks to obtain a grayscale mapping table for each of the sub-blocks; Based on the grayscale mapping table of each sub-block, a grayscale value of each pixel in the third image is determined.

6. The gesture recognition system according to claim 5, characterized in that: The training module is configured to implement the grayscale mapping table based on each sub-block to determine the grayscale value of each pixel in the third image in the following manner: Determine a first dependent sub-block of a first pixel, where the first pixel is any pixel in a first sub-block, and the first sub-block is any one of the multiple sub-blocks; Based on the first dependent sub-block, determining a second dependent sub-block, a third dependent sub-block, and a fourth dependent sub-block; Determining, based on the grayscale value of the first pixel before the grayscale histogram equalization processing and the grayscale mapping table of each sub-block, the grayscale mapping values ​​of the first pixel in the first dependent sub-block, the second dependent sub-block, the third dependent sub-block, and the fourth dependent sub-block, respectively, to obtain four grayscale mapping values; Based on the four grayscale mapping values, bilinear interpolation is performed on the grayscale value of the first pixel before the grayscale histogram equalization processing to obtain the grayscale value of the first pixel.

7. The gesture recognition system according to claim 5, characterized in that: The training module is further configured to implement the preprocessing of the plurality of videos in the training data set in the following manner: Performing bilateral filtering denoising processing on the pixel value of each pixel in the third image based on a Gaussian filter in the spatial domain and a Gaussian filter in the value range to obtain a corrected pixel value of each pixel; The Gaussian filter in the spatial domain is implemented using the following formula: The Gaussian filter of the range is implemented using the following formula: in, is a Gaussian filter in the spatial domain, is a Gaussian filter with a range of is the natural base, Pixel The coordinates of Pixel The coordinates of the pixel point and the pixel point are any two adjacent pixels in the third image in the spatial domain, Pixel The pixel value of Pixel The pixel value of is the smoothing parameter.

8. The gesture recognition system according to claim 5, characterized in that: Each of the multiple videos in the training data set corresponds to a label in a label set, the label set is synchronized with the time sequence of the multiple videos, and the labels in the label set are used to annotate multiple frames of images containing gestures in the video; The training module is further configured to adjust a timing of the label set and the videos when training the gesture recognition system based on the videos, so that the timing of the label set lags behind the timing of the videos by a first time length.

9. The gesture recognition system according to any one of claims 1 to 3, characterized in that: The skeleton vector construction module is further configured to determine an angle feature of each of the first skeleton vectors based on a sine value and a cosine value of each of the first skeleton vectors and a gravity vector; The temporal feature extraction module is also used to extract the temporal features of the human body's bone vector in the video based on the length features and angle features of the human body's bone vector in each frame of continuous multiple frames of images.

10. A gesture recognition method, characterized in that: The method comprises: The video to be recognized is input into the gesture recognition system according to any one of claims 1 to 9 to obtain the gesture type of the video to be recognized.

Images