Gesture recognition method and device, electronic device, and storage medium

Abstract

The application provides a gesture recognition method and device, electronic equipment and computer readable storage medium, comprising: in the case of obtaining the Nth video frame in the continuous video frame, judging whether the trained joint point marking model recognizes the palm from the (N-1)th video frame; N is an integer greater than 1; if yes, determining a plurality of target key point coordinates based on a plurality of joint point coordinates corresponding to the (N-1)th video frame; performing affine transformation on the Nth video frame based on the plurality of target key point coordinates; taking the Nth video frame after affine transformation as the input of the joint point marking model, obtaining the prediction category information and the plurality of joint point coordinates output by the joint point marking model; if the prediction category information represents that the palm exists in the Nth video frame, inputting the plurality of joint point coordinates corresponding to the Nth video frame into the joint point gesture recognition model, and obtaining the gesture category information output by the joint point gesture recognition model. The scheme can accurately implement the gesture recognition task.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a gesture recognition method and apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Users can interact with devices through gestures, enabling functions such as adjusting volume and switching display pages, thereby enhancing the user experience. Currently, gesture recognition technology based on deep learning can capture gesture images through a camera, and then use an image recognition model to extract features from the gesture images to identify the gestures within them.

[0003] However, in related technologies, the recognition performance of image recognition models is highly dependent on the dataset, resulting in weak robustness of the classification models. If the gesture images collected during the application process differ from the images in the dataset, the classification and recognition performance will be poor. Furthermore, when adding new gesture recognition tasks, the entire model needs to be retrained and updated, which is not only time-consuming but also wastes a significant amount of computational resources on repeatedly learning image features. Summary of the Invention

[0004] The purpose of this application is to provide a gesture recognition method, device, electronic device, and computer-readable storage medium for accurately performing gesture recognition tasks.

[0005] On the one hand, this application provides a gesture recognition method, including:

[0006] Given the Nth video frame in a series of consecutive video frames, determine whether the trained keypoint labeling model has identified a hand in the (N-1)th video frame; where N is an integer greater than 1.

[0007] If so, based on the coordinates of multiple joint points corresponding to the (N-1)th video frame, determine the coordinates of multiple target key points;

[0008] Based on the coordinates of the multiple target key points, an affine transformation is performed on the Nth video frame;

[0009] The Nth video frame after affine transformation is used as the input to the joint marker model to obtain the predicted category information and multiple joint coordinates output by the joint marker model; wherein, the predicted category information indicates whether a hand exists in the Nth video frame;

[0010] If the predicted category information indicates that a hand exists in the Nth video frame, the coordinates of multiple joints corresponding to the Nth video frame are used as input to the trained joint gesture recognition model to obtain the gesture category information output by the joint gesture recognition model.

[0011] In one embodiment, the method further includes:

[0012] If the keypoint labeling model fails to identify a hand in the (N-1)th video frame, or if the predicted category information indicates that there is no hand in the Nth video frame, the Nth video frame is used as the input of the trained hand root detection model to obtain the detection result output by the hand root detection model.

[0013] If the detection result represents the palm root position box of the Nth video frame and the coordinates of multiple key points corresponding to the palm root position box, the coordinates of multiple target key points are determined based on the coordinates of the multiple key points corresponding to the palm root position box.

[0014] Continue executing the step of performing an affine transformation on the Nth video frame based on the coordinates of the multiple target key points.

[0015] In one embodiment, the method further includes:

[0016] If the detection result indicates that there is no hand in the Nth video frame, the recognition process for the Nth video frame ends.

[0017] In one embodiment, before determining whether the trained keypoint labeling model has identified a hand from the (N-1)th video frame after acquiring the Nth video frame, the method further includes:

[0018] If the first video frame of the continuous video frames is obtained, the first video frame is used as the input of the trained palm root detection model to obtain the detection result output by the palm root detection model.

[0019] If the detection result represents the palm root position box of the first video frame and the coordinates of multiple key points corresponding to the palm root position box, the coordinates of multiple target key points are determined based on the coordinates of the multiple key points corresponding to the palm root position box.

[0020] Based on the coordinates of the multiple target key points, an affine transformation is performed on the first video frame;

[0021] The first video frame after affine transformation is used as the input of the joint labeling model to obtain the predicted category information and multiple joint coordinates corresponding to the first video frame output by the joint labeling model.

[0022] If the predicted category information corresponding to the first video frame indicates that a hand exists in the first video frame, the coordinates of multiple joint points corresponding to the first video frame are used as the input of the joint point gesture recognition model to obtain the gesture category information corresponding to the first video frame output by the joint point gesture recognition model.

[0023] In one embodiment, the joint marker model is trained as follows:

[0024] The sample images in the first dataset are used as input to a preset labeling model to obtain the sample predicted category and multiple sample key coordinates output by the labeling model; wherein, the sample images carry category labels; the category labels represent sample images containing a hand and carry multiple labeled key coordinates;

[0025] The model parameters of the labeling model are adjusted based on the differences between the predicted category and the category label of the sample image, and the differences between the coordinates of the multiple sample key points and the coordinates of the multiple label key points.

[0026] Repeat the above process until the labeling model converges to obtain the joint labeling model.

[0027] In one embodiment, the key gesture recognition model is trained in the following manner:

[0028] The sample coordinates in the second dataset are combined and used as input to a preset classification model to obtain the predicted gesture category output by the classification model; wherein each sample coordinate combination includes multiple key point coordinates and carries a gesture category label;

[0029] The model parameters of the classification model are adjusted based on the difference between the predicted gesture category and the gesture category label of the sample coordinate combination.

[0030] Repeat the above process until the classification model converges to obtain the key gesture recognition model.

[0031] In one embodiment, the palm base detection model is trained in the following manner:

[0032] The sample images in the third dataset are used as input to a preset neural network model to obtain the target detection results and coordinates of multiple sample key points output by the neural network model; wherein, the sample images carry the border position of the palm base and the coordinates of multiple label key points;

[0033] The model parameters of the neural network model are adjusted based on the differences between the bounding box positions and the target detection results of the sample images, as well as the differences between the coordinates of the multiple label key points and the coordinates of the multiple sample key points.

[0034] Repeat the above process until the neural network model converges to obtain the palm root detection model.

[0035] On the other hand, this application provides a gesture recognition device, including:

[0036] The judgment module is used to determine whether the trained keypoint labeling model has identified a hand in the (N-1)th video frame when the Nth video frame in a series of video frames has been acquired; where N is an integer greater than 1.

[0037] The determination module is used to determine the coordinates of multiple target key points based on the coordinates of multiple joint points corresponding to the (N-1)th video frame if the condition is met.

[0038] The transformation module is used to perform an affine transformation on the Nth video frame based on the coordinates of the plurality of target key points;

[0039] The prediction module is used to take the Nth video frame after affine transformation as input to the joint marker model to obtain the predicted category information and multiple joint coordinates output by the joint marker model; wherein, the predicted category information indicates whether a hand exists in the Nth video frame;

[0040] The recognition module is used to, if the predicted category information indicates that a hand exists in the Nth video frame, take the coordinates of multiple joint points corresponding to the Nth video frame as input to the trained joint point gesture recognition model, and obtain the gesture category information output by the joint point gesture recognition model.

[0041] Furthermore, this application provides an electronic device, the electronic device comprising:

[0042] processor;

[0043] Memory used to store processor-executable instructions;

[0044] The processor is configured to execute the gesture recognition method described above.

[0045] In addition, this application provides a computer-readable storage medium storing a computer program that can be executed by a processor to perform the above-described gesture recognition method.

[0046] In this application, during the processing of consecutive video frames to recognize gestures, if a hand is identified in the previous video frame by the keypoint marking model, it can be assumed that the position and posture of the hand in the subsequent video frame are basically the same as in the previous video frame. Therefore, after determining the coordinates of multiple target key points using the coordinates of multiple key points corresponding to the previous video frame, an affine transformation is performed on the subsequent video frame using these coordinates to obtain an image with a relatively upright hand posture. This image is then processed by the keypoint marking model to accurately determine whether a hand exists in the subsequent video frame and output multiple keypoint coordinates. Based on these keypoint gesture recognition coordinates, the gesture category information of the subsequent video frame is accurately identified. Since the keypoint gesture recognition model performs the classification task using keypoint coordinates, it is minimally affected by video frames (gesture images) and exhibits stronger robustness compared to existing image recognition models. Furthermore, by decoupling the model used for feature extraction (joint marker model) from the model used for classification (joint gesture recognition model), when a new gesture recognition task is added, it is not necessary to update all models; only the joint gesture recognition model needs to be updated, thus reducing training time and resource waste. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below.

[0048] Figure 1 This is a schematic diagram illustrating an application scenario of the gesture recognition method provided in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0050] Figure 3 A schematic flowchart of a gesture recognition method provided in an embodiment of this application;

[0051] Figure 4 A schematic flowchart of a video frame processing method provided in an embodiment of this application;

[0052] Figure 5 A flowchart illustrating a dynamic switching model provided in an embodiment of this application;

[0053] Figure 6 A comparative schematic diagram showing the dynamic switching before and after an embodiment of this application;

[0054] Figure 7 A schematic diagram illustrating the processing flow of the first video frame provided in an embodiment of this application;

[0055] Figure 8 A flowchart illustrating a training method for a joint marker model provided in an embodiment of this application;

[0056] Figure 9 A flowchart illustrating a training method for a joint gesture recognition model provided in an embodiment of this application;

[0057] Figure 10 A flowchart illustrating the training method for a palm root detection model provided in an embodiment of this application;

[0058] Figure 11 A schematic diagram illustrating the overall concept of the solution provided for an embodiment of this application;

[0059] Figure 12 This is a block diagram of a gesture recognition device provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0061] Similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0062] Figure 1 This is a schematic diagram illustrating an application scenario of the gesture recognition method provided in this application. For example... Figure 1 As shown, the application scenario includes a client 20 and a server 30. The client 20 can be an electronic device such as a mobile phone, tablet, or webcam, used to send video stream data to the server 30. The server 30 can be an edge computing device, a server, a server cluster, or a cloud computing center, which can process each video frame in the video stream data to identify gestures in the video frames.

[0063] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12. Figure 2 Taking a processor 11 as an example, the processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 to enable the electronic device 1 to perform all or part of the process of the method in the following embodiments. In one embodiment, the electronic device 1 may be the aforementioned server 30, used to execute the gesture recognition method.

[0064] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0065] This application also provides a computer-readable storage medium storing a computer program that can be executed by a processor 11 to perform the gesture recognition method provided in this application.

[0066] See Figure 3 This is a flowchart illustrating a gesture recognition method provided in an embodiment of this application, as shown below. Figure 3 As shown, the method may include steps 310-350.

[0067] Step 310: Given the Nth video frame in a series of consecutive video frames, determine whether the trained keypoint labeling model has identified the hand from the (N-1)th video frame; where N is an integer greater than 1.

[0068] The server can acquire continuous video frames from the client and perform gesture recognition processing on each acquired video frame. In one embodiment, the continuous video frames are the raw video stream data captured by the client. For example, if the client captures 30 video frames per second, all 30 video frames per second are sent to the server for processing. In another embodiment, to reduce computational load, the continuous video frames are video frames extracted by the client from the acquired raw video stream. For example, if the client captures 30 video frames per second, it can select one video frame every other frame, thus sending 15 video frames out of the 30 video frames captured per second to the server for processing.

[0069] The Nth video frame in a series of video frames represents any video frame other than the first video frame. After obtaining the Nth video frame, the server can determine whether the keypoint marking model has identified the hand from the (N-1)th video frame.

[0070] Here, the joint marker model is used to identify the coordinates of each joint point of the hand in a video frame (image). The joint point coordinates are the coordinate positions of the joint points after establishing an image coordinate system based on the video frames. The number of joint points that the joint marker model can identify can be set as needed. Joint points can include the joints at the phalangeal joints of each finger, the fingertips, the joints at the joints where the phalanges and metacarpals connect, the center point of the base of the hand, the midpoint of the line connecting the base of the thumb and the midpoint of the base of the hand, etc. If all the aforementioned joint points are selected, the joint marker model can identify the coordinates of 21 joint points of a hand.

[0071] The joint labeling model can predict the joint coordinates of each joint of a hand in a video frame by performing a regression task. Furthermore, the joint labeling model can also include a classifier that outputs a predicted category indicating whether a hand exists in the video frame. This predicted category can be a confidence level indicating the presence of a hand; if the confidence level is greater than a preset confidence threshold (e.g., 0.7), then the predicted category indicates the presence of a hand in the video frame.

[0072] Since the server continuously processes each acquired video frame, it can determine whether a hand exists in the (N-1)th video frame based on the predicted category information corresponding to the (N-1)th video frame.

[0073] Step 320: If yes, determine the coordinates of multiple target key points based on the coordinates of multiple joint points corresponding to the (N-1)th video frame.

[0074] If a hand is present in the (N-1)th video frame, then the (N-1)th video frame contains multiple corresponding keypoint coordinates. The server can then calculate multiple target keypoint coordinates based on these keypoint coordinates. Here, target keypoints are keypoints used as the basis for affine transformations. For example, target keypoints could be the center point of the hand, the center point of the base of the hand, or the center point of the left side of the hand; or, target keypoints could be the center point of the hand, the center point of the base of the hand, or the center point of the right side of the hand.

[0075] The coordinates of the center point at the base of the palm are already included in the multiple joint point coordinates and can be selected directly. For the center point of the palm, select the joint coordinates of the four fingers (excluding the thumb) where the phalanges connect to the metacarpal bones (at the base of the fingers). Average the x and y coordinates of these four joint points to obtain the center point coordinates. For the center point of the left or right side of the palm, select the coordinates of the middle fingertip and the center point at the base of the palm to calculate the palm length. Multiply this length by a preset conversion factor to obtain the palm width. With the center point coordinates already obtained, shifting the x-coordinate to the left by half the width yields the center point coordinates of the left side of the palm; similarly, shifting the x-coordinate to the right by half the width yields the center point coordinates of the right side of the palm.

[0076] Step 330: Perform an affine transformation on the Nth video frame based on the coordinates of multiple target key points.

[0077] After obtaining the coordinates of multiple target keypoints, an affine transformation can be performed on the Nth video frame to ensure that the hand is in a more upright position in the transformed frame. Furthermore, before performing the affine transformation on the Nth video frame, it can be preprocessed to ensure that it meets the input requirements of the keypoint labeling model. Preprocessing methods can include padding (adding zeros around the video frame) and normalizing the pixel values ​​of each channel of the video frame.

[0078] Step 340: Take the Nth video frame after affine transformation as the input of the joint labeling model, and obtain the predicted category information and multiple joint coordinates output by the joint labeling model; wherein, the predicted category information represents whether a hand exists in the Nth video frame.

[0079] Step 350: If the predicted category information indicates that a hand exists in the Nth video frame, take the coordinates of multiple joints corresponding to the Nth video frame as the input of the trained joint gesture recognition model to obtain the gesture category information output by the joint gesture recognition model.

[0080] After preprocessing and performing affine transformation on the Nth video frame, it can be input into the joint labeling model, which will output predicted category information and multiple joint coordinates.

[0081] If the predicted category information indicates the presence of a hand in the Nth video frame, it means that the multiple keypoint coordinates are valid. In this case, the input parameters of the keypoint gesture recognition model can be constructed based on the multiple keypoint coordinates. The keypoint gesture recognition model is used to classify gestures based on keypoint coordinate information.

[0082] Since the coordinates of multiple key points are preprocessed and affine transformed coordinates from video frames, before constructing the input parameters, inverse affine transformation and zero-padding errors can be performed on each key point coordinate to obtain the actual key point coordinates corresponding to the Nth video frame. Furthermore, the key point coordinate feature vector can be superimposed from the actual key point coordinates corresponding to the Nth video frame and used as the input parameters. For example, there are a total of 21 key point coordinates, which can be superimposed according to a preset order to obtain a 1*42 key point coordinate feature vector.

[0083] After obtaining the input parameters, they can be fed into the joint gesture recognition model. The joint gesture recognition model then performs a classification task based on the input parameters to obtain gesture category information.

[0084] Through the above measures, in the process of processing consecutive video frames to recognize gestures, if the palm is recognized by the joint point marking model in the previous video frame, it can be assumed that the position and posture of the palm in the subsequent video frame are basically the same as those in the previous video frame. Therefore, after determining the coordinates of multiple target key points by using the coordinates of multiple joint points corresponding to the previous video frame, an affine transformation can be performed on the subsequent video frame using the coordinates of multiple target key points to obtain an image with a relatively upright palm posture. Then, through processing by the joint point marking model, the presence of a palm in the subsequent video frame can be accurately determined and multiple joint point coordinates can be output. Based on the joint point gesture recognition model, the gesture category information of the subsequent video frame can be accurately identified based on the multiple joint point coordinates.

[0085] Because the keypoint gesture recognition model performs classification tasks using keypoint coordinates, it is minimally affected by video frames (gesture images), exhibiting stronger robustness compared to existing image recognition models. Furthermore, by decoupling the model used for feature extraction (keypoint labeling model) from the model used for classification (keypoint gesture recognition model), when a new gesture recognition task is added, it is unnecessary to update all models; only the keypoint gesture recognition model needs to be updated, reducing training time and resource waste.

[0086] In one embodiment, see Figure 4 This is a flowchart illustrating a video frame processing method provided in an embodiment of this application, as shown below. Figure 4 As shown, the method may include steps 410 to 430.

[0087] Step 410: If the keypoint labeling model fails to identify the palm in the (N-1)th video frame, or if the predicted category information indicates that there is no palm in the Nth video frame, use the Nth video frame as the input of the trained palm root detection model to obtain the detection result output by the palm root detection model.

[0088] When the server executes step 310, if it determines that the keypoint labeling model has not identified a hand in the (N-1)th video frame (in other words, a hand does not exist in the (N-1)th video frame), then step 410 can continue. When the server executes step 340, if the predicted category information corresponding to the Nth video frame indicates that a hand does not exist in the Nth video frame, this could be because a hand indeed does not exist in the Nth video frame, or it could be that the hand position in the Nth video frame has changed too much compared to the hand position in the (N-1)th video frame, making it impossible for the keypoint labeling model to process the Nth video frame after the affine transformation using the target keypoint coordinates corresponding to the (N-1)th video frame. Therefore, when the predicted category information indicates that a hand does not exist in the Nth video frame, the server can continue executing step 410.

[0089] The server can input the Nth video frame into the palm root detection model, which then processes it to obtain the detection result. Before inputting the Nth video frame into the palm root detection model, it can be preprocessed to ensure that the Nth video frame meets the input requirements of the palm root detection model. Preprocessing methods can include padding, resizing (changing the image size), and normalizing the pixel values ​​of each channel of the video frame.

[0090] Here, the palm base detection model is a target detection model for the palm base. In addition, the palm base detection model can also perform regression tasks through the added regression task output layer, outputting multiple keypoint coordinates. Among them, the keypoint coordinates can include the coordinates of key joint points where the phalanges and metacarpals of the four fingers other than the thumb connect, the coordinates of the center point of the finger base, and the coordinates of the midpoint of the line connecting the base of the thumb and the midpoint of the palm base, etc.

[0091] After obtaining the detection result, it can be checked whether the detection result indicates that a hand is present in the Nth video frame. In one case, if the detection result indicates that a hand is not present in the Nth video frame, the recognition process for the Nth video frame can be terminated. In this case, the server can continue to acquire the (N+1)th video frame and then perform the recognition process using the recognition method of this application. In another case, if the detection result indicates that a hand is present in the Nth video frame, step 420 can be continued.

[0092] Step 420: If the detection result represents the palm root location box of the Nth video frame and the coordinates of multiple key points corresponding to the palm root location box, determine the coordinates of multiple target key points based on the coordinates of the multiple key points corresponding to the palm root location box.

[0093] If the detection result indicates the presence of a hand in the Nth video frame, the detection result can represent the bounding box of the hand's base in the Nth video frame, as well as the coordinates of multiple keypoints corresponding to the bounding box. Here, the bounding box of the hand's base is used to represent the location of the hand's base in the Nth video frame.

[0094] In one embodiment, the hand root detection model can be built based on models such as SSD (Single Shot MultiBoxDetector) and YOLOv2 (You Only Look Once version 2). In this case, when processing the Nth video frame, the hand root detection model can output corresponding predicted boxes and confidence scores for multiple preset anchor boxes. The confidence score represents the probability that the hand root exists within the predicted box. In addition, the hand root detection model can also output the coordinates of multiple key points corresponding to each predicted box.

[0095] By comparing the confidence scores of the predicted bounding boxes with a preset confidence threshold, several predicted bounding boxes with a confidence threshold can be selected. Furthermore, the NMS (Non-Maximum Suppression) algorithm is used to select the bounding box most likely representing the location of the hand's base from these predicted bounding boxes.

[0096] After obtaining the bounding box of the hand's base in the Nth video frame, the server can calculate the coordinates of multiple target keypoints based on the coordinates of multiple keypoints corresponding to this bounding box. Here, the target keypoints can be the same as those selected in step 320. Furthermore, the coordinates of the keypoints corresponding to the bounding box are represented by their relative positions to the anchor frames (e.g., by the offset relative to the center point of the anchor frame). Before calculating the target keypoint coordinates, the coordinates of multiple keypoints can be reconstructed to obtain the actual coordinates of multiple keypoints in the Nth video frame, and the target keypoint coordinates are then calculated using the reconstructed keypoint coordinates.

[0097] For the target key points, the center point of the palm and the center point of the palm base, the method for determining the coordinates of the target key points can be referred to the relevant description above, and will not be repeated here. For the target key point, the center point of the left side of the palm or the center point of the right side of the palm, the distance from the center point of the palm to the center point of the palm base can be calculated based on the coordinates of the center point of the palm and the center point of the palm base. Multiplying this distance by a preset conversion factor yields the palm width. With the coordinates of the center point of the palm already obtained, shifting the horizontal coordinate to the left by half the width yields the coordinates of the center point of the left side of the palm. With the coordinates of the center point of the palm already obtained, shifting the horizontal coordinate to the right by half the width yields the coordinates of the center point of the right side of the palm.

[0098] Step 430: Continue to perform the step of performing an affine transformation on the Nth video frame based on the coordinates of multiple target key points.

[0099] After obtaining the coordinates of multiple target key points, the server can proceed to step 330, perform an affine transformation on the Nth video frame based on the coordinates of the multiple target key points, and continue to execute subsequent steps after the affine transformation, thereby completing the gesture recognition process for the Nth video frame.

[0100] By employing the above measures, when it is impossible to use the keypoint coordinates corresponding to the previous video frame to perform an affine transformation on the subsequent video frame in order to identify the hand in the subsequent video frame, a hand base detection model can be used to perform target detection on the subsequent video frame to determine the position of the hand and the coordinates of multiple keypoints in the subsequent video frame. Then, if the hand is present in the subsequent video frame, the coordinates of multiple target keypoints can be determined using the coordinates of multiple keypoints, enabling the correct affine transformation to be performed on the subsequent video frame for subsequent keypoint detection and gesture recognition.

[0101] The proposed solution dynamically switches between the aforementioned palm root detection model and joint marker model when processing continuous video frames, which reduces the number of model inference runs during the detection process and greatly reduces the computational load.

[0102] See Figure 5 This is a flowchart illustrating a dynamic switching model provided in an embodiment of this application, as shown below. Figure 5 As shown, in the process of processing continuous video frames, the video frames can first be detected by the palm base detection model, and the presence of a palm in the current video frame can be determined based on the detection results of the palm base detection model.

[0103] In one scenario, where the hand is not present, a new video frame can be acquired, and the new video frame can be re-detected using the hand base detection model.

[0104] In another scenario, if a hand is present, multiple target keypoint coordinates can be determined based on the coordinates of multiple keypoints output by the hand base detection model. Then, an affine transformation is performed on the current video frame based on these target keypoint coordinates. The video frame after the affine transformation is processed using a keypoint labeling model to obtain predicted category information and multiple keypoint coordinates.

[0105] The predicted category information indicates the presence of a hand in the video frame after affine transformation (if the hand root detection model has determined the presence of a hand, the predicted category information will necessarily indicate the presence of a hand). The obtained coordinates of multiple key points can be used as the latest key point coordinates for subsequent hand category recognition.

[0106] Furthermore, multiple target keypoint coordinates are determined using the coordinates of multiple joint points, and an affine transformation is performed on the new video frame. The new video frame after the affine transformation is processed using a joint point labeling model to obtain predicted category information and multiple joint point coordinates.

[0107] On one hand, the predicted category information indicates the presence of a hand in the new video frame after affine transformation. Multiple keypoints corresponding to this video frame can be used as new keypoint coordinates for subsequent hand category recognition. Furthermore, the coordinates of multiple target keypoints are determined using these keypoint coordinates, and the affine transformation is then performed on the next video frame.

[0108] On the other hand, during the prediction of keypoint coordinates in each video frame using a single keypoint labeling model, if the predicted category information for any video frame does not represent a hand, then the hand root detection model needs to be used to detect that video frame. Further, subsequent processes are executed based on the detection results.

[0109] In this process, if the hand movement is small, the relatively simple joint marker model can continuously detect the joint coordinates in each video frame, requiring less computational power. However, when the hand movement is large, and the joint marker model cannot track the hand, a more complex hand root detection model is used to process the video frames. Once the presence of a hand is confirmed, the keypoint coordinates output by the hand root detection model are used to perform an affine transformation on the video frames, which are then processed by the joint marker model. By dynamically switching between the two models, the number of calls to the hand root detection model can be reduced when the hand movement is small, thus lowering the computational load and time. Furthermore, if this solution is applied to mobile devices, dynamically switching between the two models can also reduce device power consumption and enhance battery life.

[0110] See Figure 6 This is a comparative diagram showing the dynamic switching process before and after an embodiment of this application. Figure 6 The left side shows the joint detection status before the dynamic switching model, which can process 17 video frames per second; Figure 6 The right side shows the joint detection results after dynamically switching models, which can process 33 video frames per second. Clearly, dynamic switching can significantly improve the processing efficiency of video frames during gesture recognition.

[0111] In one embodiment, before performing identification processing on the Nth video frame in a continuous video stream, it is first necessary to perform identification processing on the first video frame of the continuous video stream. See [link to documentation]. Figure 7 This is a schematic diagram of the processing flow of the first video frame provided in an embodiment of this application, as shown below. Figure 7 As shown, the method may include steps 710 to 750.

[0112] Step 710: If the first video frame of the continuous video frames is obtained, use the first video frame as the input of the trained palm root detection model to obtain the detection result output by the palm root detection model.

[0113] When the server first obtains a video frame from a series of video frames, it can input the first video frame into the palm root detection model. The palm root detection model processes the first video frame to obtain the detection result. This process is similar to step 410 mentioned above and will not be repeated here. On one hand, if the detection result indicates that there is no palm in the first video frame, the recognition process for the first video frame can be terminated. In this case, the server can continue to obtain the second video frame and then perform the recognition process using the recognition method of this application. On the other hand, if the detection result indicates that there is a palm in the first video frame, step 720 can be executed.

[0114] Step 720: If the detection result represents the palm root location box of the first video frame and the coordinates of multiple key points corresponding to the palm root location box, determine the coordinates of multiple target key points based on the coordinates of the multiple key points corresponding to the palm root location box.

[0115] If the detection result indicates the presence of a hand in the first video frame, the detection result can represent the bounding box of the hand's base in the first video frame, as well as the coordinates of multiple key points corresponding to that bounding box. Here, the bounding box of the hand's base is used to represent the location of the hand's base in the first video frame.

[0116] The server can calculate the coordinates of multiple target key points based on the coordinates of multiple key points corresponding to the location box at the base of the palm. This process is similar to step 420 above and will not be repeated here.

[0117] Step 730: Perform an affine transformation on the first video frame based on the coordinates of multiple target key points.

[0118] After obtaining the coordinates of multiple target keypoints, an affine transformation can be performed on the first video frame to ensure the hand is in a more upright position in the transformed frame. Furthermore, before performing the affine transformation on the first video frame, preprocessing can be performed to ensure it meets the input requirements of the keypoint marking model. Preprocessing methods can include padding and normalizing the pixel values ​​of each channel of the video frame.

[0119] Step 740: Use the first video frame after affine transformation as the input to the joint labeling model to obtain the predicted category information and multiple joint coordinates corresponding to the first video frame output by the joint labeling model.

[0120] Step 750: If the predicted category information corresponding to the first video frame indicates that a hand exists in the first video frame, take the coordinates of multiple joints corresponding to the first video frame as the input of the joint gesture recognition model, and obtain the gesture category information corresponding to the first video frame output by the joint gesture recognition model.

[0121] After preprocessing and affine transforming the first video frame, it can be input into the joint labeling model, which will output predicted category information and multiple joint coordinates.

[0122] If the detection result of the palm base detection model indicates the presence of a palm in the first video frame, the predicted category information can also indicate the presence of a palm in the first video frame. In this case, multiple joint point coordinates are valid. After constructing input parameters from the multiple joint point coordinates, they are processed by the joint point gesture recognition model to obtain the gesture category information of the first video frame. This process is similar to step 350 above and will not be elaborated further.

[0123] Through the above measures, after obtaining the first video frame, multiple joints in the video frame can be accurately identified by using the palm root detection model and the joint mark model. Then, the hand gesture recognition model can be used to process multiple joints to obtain gesture category information.

[0124] In one embodiment, a keypoint marking model can be trained before executing the gesture recognition method of this application. See also Figure 8 This is a flowchart illustrating a training method for a joint marker model provided in an embodiment of this application. Figure 8 As shown, the method may include steps 810 to 830.

[0125] Step 810: Use the sample images in the first dataset as input to the preset labeling model to obtain the sample predicted category and multiple sample key coordinates output by the labeling model; wherein, the sample images carry category labels; the category labels represent sample images containing a hand, and carry multiple labeled key coordinates.

[0126] Here, the first dataset can include a large number of sample images, each carrying a category label indicating whether a hand is present. For sample images containing a hand, multiple labeled joint coordinates can be pre-labeled, representing the coordinate positions of joints in the sample image. For sample images without a hand, their corresponding labeled joint coordinates can be considered empty. To achieve good training results, images with relatively upright hand postures can be selected as sample images.

[0127] Labeling models need to perform classification and regression tasks and can include feature extraction networks, classifiers, and regression layers.

[0128] The sample images in the first dataset are processed by a labeling model to obtain the predicted category and multiple keypoint coordinates for each sample image. The predicted category is the predicted category information for the sample image, and the keypoint coordinates are the keypoint coordinates corresponding to the sample image. The number of keypoint coordinates output by the labeling model can be set according to requirements, and the keypoints predicted during the training phase are the keypoints predicted in the application phase after training is completed.

[0129] Step 820: Adjust the model parameters of the labeling model based on the differences between the predicted category and the category label of the sample image, as well as the differences between the coordinates of multiple sample key points and the coordinates of multiple label key points.

[0130] Step 830: Repeat the above process until the labeling model converges to obtain the joint labeling model.

[0131] After obtaining the predicted class of the sample image and the coordinates of multiple sample key points, the difference between the predicted class and the class label of the same sample image, as well as the difference between the coordinates of multiple sample key points and the coordinates of multiple label key points, can be evaluated using the loss function corresponding to the preset labeling model. The function value of the loss function is obtained, and the model parameters of the labeling model are adjusted according to the function value of the loss function.

[0132] After adjustments, we can return to step 810 and proceed to the next round of training. After repeated iterations, when the number of training rounds reaches the preset number or the value of the loss function tends to stabilize, the labeling model can be considered to have converged. At this point, we obtain the trained joint labeling model.

[0133] Through the above measures, a joint marker model can be trained to determine whether a hand exists in an image (video frame) and to predict the coordinates of the joints.

[0134] In one embodiment, a knuckle gesture recognition model can be trained before executing the gesture recognition method of this application. See also Figure 9 This is a flowchart illustrating a training method for a key gesture recognition model provided in an embodiment of this application. Figure 9 As shown, the method may include steps 910 to 930.

[0135] Step 910: Combine the sample coordinates in the second dataset as input to the preset classification model to obtain the predicted gesture category output by the classification model; wherein, each sample coordinate combination includes multiple key point coordinates and carries a gesture category label.

[0136] Here, the second dataset can include a large number of sample coordinate combinations. Each sample coordinate combination includes multiple keypoint coordinates, the number of which is the same as the number of keypoint coordinates that the keypoint labeling model can output, and the multiple keypoint coordinates are arranged in a preset order. In constructing the second dataset, a trained keypoint labeling model can be used to output multiple keypoint coordinates from images (video frames) containing a hand, and these multiple keypoint coordinates are then used to construct sample coordinate combinations. Gesture category labels are then assigned to the sample coordinate combinations based on the category of the hand in the image. Multiple sample coordinate combinations carrying gesture category labels can then be used to construct the second dataset.

[0137] Before the server inputs the combined sample coordinates into the classification model, it can construct multiple keypoint coordinates from the combined sample coordinates as input parameters. For example, multiple keypoint coordinates can be superimposed to obtain a keypoint coordinate feature vector, which can then be used as an input parameter. The server can then process the input parameters using the classification model to perform the classification task, thereby outputting the predicted gesture category. The predicted gesture category is the gesture category information obtained during the training process by classifying the combined sample coordinates.

[0138] Step 920: Adjust the model parameters of the classification model based on the difference between the predicted gesture category and the gesture category label of the sample coordinate combination.

[0139] Step 930: Repeat the above process until the classification model converges to obtain the key gesture recognition model.

[0140] After obtaining the gesture category information corresponding to the sample coordinate combination, the difference between the gesture category information and the gesture category label of the sample coordinate combination can be evaluated by using the loss function corresponding to the preset classification model, and the function value of the loss function can be obtained. Based on the function value of the loss function, the model parameters of the classification model can be adjusted.

[0141] After adjustments, return to step 910 to begin the next round of training. After repeated iterations, when the number of training rounds reaches the preset number or the value of the loss function tends to stabilize, the classification model can be considered to have converged. At this point, the trained key gesture recognition model is obtained.

[0142] Through the above measures, a joint gesture recognition model based on multiple joint gesture categories can be trained.

[0143] In one embodiment, a palm root detection model can be trained before executing the gesture recognition method of this application. See also Figure 10 A schematic flowchart illustrating the training method for a palm root detection model provided in an embodiment of this application is shown below. Figure 10 As shown, the method may include steps 1010 to 1030.

[0144] Step 1010: Use the sample images in the third dataset as input to the preset neural network model to obtain the target detection results and coordinates of multiple sample key points output by the neural network model; wherein, the sample images carry the border position of the palm base and the coordinates of multiple label key points.

[0145] Here, the third data may include a large number of sample images, which may include a hand, and the position of the base of the hand in the image is defined by the marked border position, and the position of the key point is defined by the marked label key point coordinates.

[0146] Neural network models need to perform object detection and regression tasks, and can include feature extraction networks, object detection modules, and regression tasks.

[0147] The neural network model processes sample images from the third dataset to obtain object detection results and multiple keypoint coordinates for each sample image. The object detection result is the predicted bounding box of the hand's base, generated during the object detection task, and the keypoint coordinates are those output during training. The keypoint coordinates output by the neural network model can be set according to requirements, and the keypoints predicted during training are the same as those predicted in the application phase after training is complete.

[0148] Step 1020: Adjust the model parameters of the neural network model based on the differences between the bounding box positions of the sample images and the target detection results, as well as the differences between the coordinates of multiple label key points and the coordinates of multiple sample key points.

[0149] Step 1030: Repeat the above process until the neural network model converges to obtain the palm root detection model.

[0150] After obtaining the target detection results and keypoint coordinates of the sample image, the difference between the target detection results and the bounding box position of the same sample image, as well as the difference between the keypoint coordinates of multiple labels and the keypoint coordinates of multiple samples, can be evaluated using the loss function corresponding to the preset neural network model. The function value of the loss function is obtained, and the model parameters of the neural network model are adjusted according to the function value of the loss function.

[0151] After adjustments, return to step 1010 to begin the next round of training. After repeated iterations, when the training rounds reach the preset number or the loss function value stabilizes, the neural network model can be considered converged. At this point, the trained palm root detection model is obtained.

[0152] Through the above measures, a hand base detection model can be trained to detect targets at the base of the hand and predict key point coordinates in images.

[0153] See Figure 11 The above is a schematic diagram illustrating the overall concept of the solution provided in one embodiment of this application, as shown below. Figure 11 As shown, firstly, a palm root detection model, a joint marker model, and a joint gesture recognition model can be trained using sample data (which could be the first, second, and third datasets mentioned earlier). Then, these models are used to recognize gesture images (video frames), thereby identifying the gesture category information in each image.

[0154] Figure 12 This is a block diagram of a gesture recognition device according to an embodiment of the present invention, such as... Figure 12 As shown, the device may include:

[0155] The judgment module 1210 is used to determine whether the trained keypoint labeling model has identified a hand from the (N-1)th video frame when the Nth video frame in a series of video frames has been obtained; where N is an integer greater than 1.

[0156] The determination module 1220 is used to determine the coordinates of multiple target key points based on the coordinates of multiple joint points corresponding to the (N-1)th video frame if the condition is met.

[0157] Transformation module 1230 is used to perform an affine transformation on the Nth video frame based on the coordinates of the plurality of target key points;

[0158] The prediction module 1240 is used to take the Nth video frame after affine transformation as the input of the joint marker model to obtain the predicted category information and multiple joint coordinates output by the joint marker model; wherein, the predicted category information indicates whether a hand exists in the Nth video frame;

[0159] The recognition module 1250 is used to, if the predicted category information indicates that a hand exists in the Nth video frame, take the coordinates of multiple joint points corresponding to the Nth video frame as input to the trained joint point gesture recognition model, and obtain the gesture category information output by the joint point gesture recognition model.

[0160] The specific implementation process of the functions and roles of each module in the above-mentioned device can be found in the implementation process of the corresponding steps in the above-mentioned gesture recognition method, and will not be repeated here.

[0161] The apparatuses and methods disclosed in the several embodiments provided in this application can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0162] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0163] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A gesture recognition method, characterized in that, include: Given the Nth video frame in a series of consecutive video frames, determine whether the trained keypoint labeling model has identified a hand in the (N-1)th video frame; where N is an integer greater than 1. If so, based on the coordinates of multiple joint points corresponding to the (N-1)th video frame, determine the coordinates of multiple target key points; Based on the coordinates of the multiple target key points, an affine transformation is performed on the Nth video frame; The Nth video frame after affine transformation is used as the input to the joint marker model to obtain the predicted category information and multiple joint coordinates output by the joint marker model; wherein, the predicted category information indicates whether a hand exists in the Nth video frame; If the predicted category information indicates that a hand exists in the Nth video frame, the coordinates of multiple joints corresponding to the Nth video frame are used as input to the trained joint gesture recognition model to obtain the gesture category information output by the joint gesture recognition model.

2. The method according to claim 1, characterized in that, The method further includes: If the keypoint labeling model fails to identify a hand in the (N-1)th video frame, or if the predicted category information indicates that there is no hand in the Nth video frame, the Nth video frame is used as the input of the trained hand root detection model to obtain the detection result output by the hand root detection model. If the detection result represents the palm root position box of the Nth video frame and the coordinates of multiple key points corresponding to the palm root position box, the coordinates of multiple target key points are determined based on the coordinates of the multiple key points corresponding to the palm root position box. Continue executing the step of performing an affine transformation on the Nth video frame based on the coordinates of the multiple target key points.

3. The method according to claim 2, characterized in that, The method further includes: If the detection result indicates that there is no hand in the Nth video frame, the recognition process for the Nth video frame ends.

4. The method according to claim 1, characterized in that, Before determining whether the trained keypoint labeling model has identified the hand from the (N-1)th video frame after acquiring the Nth video frame in a series of video frames, the method further includes: If the first video frame of the continuous video frames is obtained, the first video frame is used as the input of the trained palm root detection model to obtain the detection result output by the palm root detection model. If the detection result represents the palm root position box of the first video frame and the coordinates of multiple key points corresponding to the palm root position box, the coordinates of multiple target key points are determined based on the coordinates of the multiple key points corresponding to the palm root position box. Based on the coordinates of the multiple target key points, an affine transformation is performed on the first video frame; The first video frame after affine transformation is used as the input of the joint labeling model to obtain the predicted category information and multiple joint coordinates corresponding to the first video frame output by the joint labeling model. If the predicted category information corresponding to the first video frame indicates that a hand exists in the first video frame, the coordinates of multiple joint points corresponding to the first video frame are used as the input of the joint point gesture recognition model to obtain the gesture category information corresponding to the first video frame output by the joint point gesture recognition model.

5. The method according to claim 1, characterized in that, The joint marking model It was obtained through training in the following manner: The sample images in the first dataset are used as input to a preset labeling model to obtain the sample predicted category and multiple sample key coordinates output by the labeling model; wherein, the sample images carry category labels; the category labels represent sample images containing a hand and carry multiple labeled key coordinates; The model parameters of the labeling model are adjusted based on the differences between the predicted category and the category label of the sample image, and the differences between the coordinates of the multiple sample key points and the coordinates of the multiple label key points. Repeat the above process until the labeling model converges to obtain the joint labeling model.

6. The method according to claim 1, characterized in that, The keypoint gesture recognition model is trained in the following manner: The sample coordinates in the second dataset are combined and used as input to a preset classification model to obtain the predicted gesture category output by the classification model; wherein each sample coordinate combination includes multiple key point coordinates and carries a gesture category label; The model parameters of the classification model are adjusted based on the difference between the predicted gesture category and the gesture category label of the sample coordinate combination. Repeat the above process until the classification model converges to obtain the key gesture recognition model.

7. The method according to claim 2, characterized in that, The palm heel detection model was trained in the following manner: The sample images in the third dataset are used as input to a preset neural network model to obtain the target detection results and coordinates of multiple sample key points output by the neural network model; wherein, the sample images carry the border position of the palm base and the coordinates of multiple label key points; The model parameters of the neural network model are adjusted based on the differences between the bounding box positions and the target detection results of the sample images, as well as the differences between the coordinates of the multiple label key points and the coordinates of the multiple sample key points. Repeat the above process until the neural network model converges to obtain the palm root detection model.

8. A gesture recognition device, characterized in that, include: The judgment module is used to determine whether the trained keypoint labeling model has identified a hand in the (N-1)th video frame when the Nth video frame in a series of video frames has been acquired; where N is an integer greater than 1. The determination module is used to determine the coordinates of multiple target key points based on the coordinates of multiple joint points corresponding to the (N-1)th video frame if the condition is met. The transformation module is used to perform an affine transformation on the Nth video frame based on the coordinates of the plurality of target key points; The prediction module is used to take the Nth video frame after affine transformation as input to the joint marker model to obtain the predicted category information and multiple joint coordinates output by the joint marker model; wherein, the predicted category information indicates whether a hand exists in the Nth video frame; The recognition module is used to, if the predicted category information indicates that a hand exists in the Nth video frame, take the coordinates of multiple joint points corresponding to the Nth video frame as input to the trained joint point gesture recognition model, and obtain the gesture category information output by the joint point gesture recognition model.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to execute the gesture recognition method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that can be executed by a processor to perform the gesture recognition method according to any one of claims 1-7.

Images