Gesture recognition method, apparatus, electronic device, medium, and program product

Abstract

The application relates to a gesture recognition method and device, electronic equipment, medium and program product, and applies to the technical field of image processing. The method comprises the following steps: acquiring one or more to-be-processed images of a user gesture, wherein a plurality of gesture images are collected from different directions; determining position information of a plurality of hand key points in the to-be-processed image and a confidence degree of the position information based on a pre-trained position detection model; performing posture calculation according to the position information to obtain posture information of the user gesture; and determining whether to track the user gesture based on the posture information according to the confidence degree of the position information. The application can improve the accuracy of gesture recognition.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a gesture recognition method, device, electronic device, medium, and program product. Background Technology

[0002] In the VR / AR (Virtual Reality / Augmented Reality) field, gesture recognition is a way to achieve interaction. The more accurate the gesture recognition, the better the user's interactive experience. Among related technologies, gesture recognition can be based on computer vision tracking and recognition technology. However, when the user's gestures are complex, there may be a significant residual between the gesture recognition result and the user's actual gesture, resulting in lower gesture recognition accuracy. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a gesture recognition method, apparatus, electronic device, medium, and program product.

[0004] According to a first aspect of this application, a gesture recognition method is provided, comprising: Acquire one or more images to be processed for a user's gesture, wherein the multiple gesture images are acquired from multiple different orientations; Based on a pre-trained position detection model, the position information of multiple hand key points in the image to be processed and the confidence level of the position information are determined. The user's gesture posture information is obtained by performing posture calculation based on the location information. Based on the confidence level of the location information, determine whether to track and predict the user's gestures based on the posture information.

[0005] Optionally, the user gesture's gesture gesture information is obtained by performing gesture calculation based on the location information, including: Based on the location information and the hand model, an optimization algorithm is used to calculate the pose, thereby obtaining the pose information and residual value of the user's gesture; wherein, the hand model includes the feature information of the hand; The step of determining whether to track and predict the user's gesture based on the pose information according to the confidence level of the location information includes: Based on the confidence level of the location information and the residual value, determine whether to track and predict the user's gesture based on the posture information.

[0006] Optionally, determining whether to track and predict the user's gesture based on the pose information according to the confidence level of the location information and the residual value includes: The residual confidence level is determined based on the residual value and the preset residual threshold. The tracking confidence level is determined based on the confidence level of the location information and the confidence level of the residual. If the tracking confidence is greater than or equal to a preset confidence threshold, then the user's gesture is tracked and predicted based on the posture information; If the tracking confidence is less than the preset confidence threshold, then the user's gesture will not be tracked or predicted based on the posture information.

[0007] Optionally, the method further includes: If the user gesture is tracked and predicted based on the posture information, the user gesture in the next frame of the image to be processed is predicted based on the posture information and the posture information of the user gesture in the previous frame or multiple previous frames of the image to be processed, so as to obtain the predicted posture information of the user gesture in the next frame. The next frame of the image to be processed is obtained, and the position information of multiple hand key points in the next frame and the confidence level of the position information are determined by the position detection model. The positional information of multiple hand key points in the next frame image and the confidence level of the positional information are corrected based on the predicted pose information.

[0008] Optionally, the position information of multiple hand key points in the next frame image is corrected based on the predicted pose information, including: Based on the predicted posture information, determine the first predicted position coordinates of the multiple hand key points in the world coordinate system; The first predicted position coordinates are converted into second predicted position coordinates in the multiple camera coordinate systems of different orientations; Convert the second predicted position coordinates into the third predicted position coordinates in the image coordinate system; Based on the third predicted position coordinates, the position information of multiple key hand points is corrected.

[0009] Optionally, the location detection model includes: a hand region recognition sub-model and a key point detection sub-model; The method based on a pre-trained position detection model determines the position information of multiple hand key points in the image to be processed and the confidence level of the position information, including: The multiple images to be processed are respectively input into the hand region recognition sub-model to obtain the first hand region image; Based on the first hand region image, a second hand region image is obtained; The second hand region image is input into the key point detection sub-model to obtain the position information of the hand key points.

[0010] Optionally, the training method for the hand region recognition sub-model includes: Acquire multiple first sample images containing hand information, hand region images in the first sample images, and hand types in the first sample images, wherein the hand type includes left hand and / or right hand; The first sample image is used as input data, and the hand region image and the type of hand in the first sample image are used as label data to train and generate the hand region recognition sub-model. The training method for the keypoint detection sub-model includes: Acquire multiple second sample images containing hand information, and the location information of key hand points in the second sample images; The second sample image is used as input data, and the location information of the key points of the hand in the second sample image is used as label data to train and generate the key point detection sub-model.

[0011] Optionally, the method further includes: After training and generating the hand region recognition sub-model, the parameter values ​​of the network parameters in the hand region recognition sub-model are quantized; and / or; When training the keypoint detection sub-model, the parameter values ​​of the network parameters in the keypoint detection sub-model are quantized.

[0012] According to a second aspect of this application, a gesture recognition device is provided, comprising: The image acquisition module is used to acquire one or more images to be processed for user gestures, wherein the multiple gesture images are acquired from multiple different positions; The location information determination module is used to determine the location information of multiple hand key points in the image to be processed and the confidence level of the location information based on a pre-trained location detection model. The posture information determination module is used to perform posture calculation based on the position information to obtain the posture information of the user's gesture; The gesture tracking and judgment module is used to determine whether to track and predict the user's gesture based on the confidence level of the location information.

[0013] Optionally, the posture information determination module is specifically used to perform posture calculation using an optimization algorithm based on the position information and the hand model to obtain the posture information and residual value of the user's gesture; wherein, the hand model includes the feature information of the hand; The gesture tracking and judgment module is specifically used to determine whether to track and predict the user's gesture based on the posture information, according to the confidence level of the location information and the residual value.

[0014] Optionally, the gesture tracking judgment module is specifically used to determine the residual confidence level based on the residual value and a preset residual threshold; determine the tracking confidence level based on the confidence level of the location information and the residual confidence level; if the tracking confidence level is greater than or equal to the preset confidence threshold, then the user gesture is tracked and predicted based on the posture information; if the tracking confidence level is less than the preset confidence threshold, then the user gesture is not tracked and predicted based on the posture information.

[0015] Optionally, the gesture recognition device further includes: The posture information prediction module is used to predict the user gesture in the next frame of the image to be processed based on the posture information and the posture information of the user gesture in the previous frame or multiple previous frames of the image to be processed, so as to obtain the predicted posture information of the user gesture in the next frame. The image acquisition module is also used to acquire the next frame image of the image to be processed; The location information determination module is further configured to determine the location information of multiple hand key points in the next frame image and the confidence level of the location information through the location detection model; The position information correction module is used to correct the position information of multiple hand key points in the next frame image and the confidence level of the position information based on the predicted pose information.

[0016] Optionally, the position information correction module is specifically used to determine the first predicted position coordinates of the plurality of hand key points in the world coordinate system based on the predicted pose information; convert the first predicted position coordinates into the second predicted position coordinates in the camera coordinate systems of the plurality of different orientations; convert the second predicted position coordinates into the third predicted position coordinates in the image coordinate system; and correct the position information of the plurality of hand key points based on the third predicted position coordinates.

[0017] Optionally, the location detection model includes: a hand region recognition sub-model and a key point detection sub-model; The location information determination module is specifically used to input the plurality of images to be processed into the hand region recognition sub-model to obtain a first hand region image; based on the first hand region image, obtain a second hand region image; and input the second hand region image into the key point detection sub-model to obtain the location information of the hand key points.

[0018] Optionally, the gesture recognition device further includes: The first training data acquisition module is used to acquire multiple first sample images containing hand information, hand region images in the first sample images, and the type of hand in the first sample images, wherein the type of hand includes left hand and / or right hand; The hand region recognition sub-model generation module is used to take the first sample image as input data, and the hand region image in the first sample image and the type of hand in the first sample image as label data to train and generate the hand region recognition sub-model. The second training data acquisition module is used to acquire multiple second sample images containing hand information and the location information of key hand points in the second sample images; The key point detection sub-model generation module is used to take the second sample image as input data and the position information of the hand key points in the second sample image as label data to train and generate the key point detection sub-model.

[0019] Optionally, the gesture recognition device further includes: The first quantization module is used to quantize the parameter values ​​of the network parameters in the hand region recognition sub-model after training and generating the sub-model; and / or; The second quantization module is used to quantize the parameter values ​​of the network parameters in the keypoint detection sub-model during training.

[0020] According to a third aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first aspect.

[0021] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0022] According to a fifth aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first aspect.

[0023] The technical solution provided in this application has the following advantages compared with the prior art: By acquiring images of user gestures from one or more different orientations, and obtaining the positional information and confidence levels of multiple hand key points within these images, the user's gesture pose information can be obtained through pose calculation based on the positional information. Since the confidence level represents the reliability of the positional information, and the reliability of the positional information determines the reliability of the pose information, the confidence level of the positional information can be used to determine whether to track the user's gesture based on the pose information, thereby improving the accuracy of gesture recognition. Acquiring images from multiple different orientations can compensate for the limitations of visual occlusion from a single orientation, resulting in pose information that better matches the actual gesture. Using this pose information for gesture tracking can further improve the accuracy of gesture recognition. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of an application scenario of the gesture recognition method that can be applied to the embodiments of this application; Figure 2 This is a flowchart of a gesture recognition method in an embodiment of this application; Figure 3A A schematic diagram of key points of the hand; Figure 3B A schematic diagram of a world coordinate system for a key point on the index finger of the hand; Figure 4A This is a flowchart of a training method for a hand region recognition sub-model in an embodiment of this application; Figure 4B This is a flowchart of a training method for a key point detection sub-model in an embodiment of this application; Figure 5 This is another flowchart of the gesture recognition method in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a gesture recognition device in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation

[0027] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0028] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0029] The gesture recognition method of this application can be applied to VR / AR scenarios or other scenarios where users can interact with devices through gestures. The executing entity of the gesture recognition method of this application can be a device that interacts with user gestures, such as a VR / AR device.

[0030] Figure 1 This is a schematic diagram illustrating an application scenario of the gesture recognition method that can be applied to the embodiments of this application. For example... Figure 1 As shown, users can interact with VR devices through gestures. One or more cameras are positioned at different locations on the VR device. With multiple cameras, these cameras can capture the user's gestures from different angles, obtaining gesture images. By extracting the positional information and confidence level of key hand points from the gesture images, pose calculation can be performed based on the positional information to obtain the user's gesture pose information. The reliability of the pose information can be evaluated based on the confidence level of the positional information, thereby determining whether to perform gesture tracking based on the pose information to improve the accuracy of gesture recognition. This application combines the positional information of key hand points from multiple different angles for pose calculation, which can compensate for the deficiencies of visual occlusion from a single angle, obtaining more realistic and accurate pose information. Furthermore, when tracking user gestures based on this pose information, the pose information of the user's gesture in the next frame image can be predicted more accurately.

[0031] The gesture recognition method of this application embodiment will be described below first.

[0032] See Figure 2 , Figure 2 This is a flowchart of a gesture recognition method in an embodiment of this application, which may include the following steps: Step S210: Obtain one or more images to be processed for the user's gestures, wherein multiple gesture images are acquired from multiple different positions.

[0033] In this embodiment, the device that interacts with user gestures may be equipped with one or more cameras for capturing image data of the user's gestures. When multiple cameras are configured, they can be located at different positions on the device to capture image data from different angles. The cameras can capture image data in real time; the image to be processed refers to the image data captured at the current moment.

[0034] Step S220: Based on the pre-trained position detection model, determine the position information and confidence level of multiple hand key points in the image to be processed.

[0035] Position detection models are used to detect the positional information of key hand points (such as joints) in an image. See also Figure 3A , Figure 3A This is a schematic diagram of hand joints, showing a total of 21 joints. Using a position detection model, the position information and confidence level of each hand joint can be obtained. For example, if three images are acquired from three different locations, the position detection model can process each image, determining the position information of 21 hand joints and their corresponding confidence levels from each image. Ultimately, this yields the position information of 63 hand joints and 63 confidence levels.

[0036] The location detection model can be a single model, where the image to be processed is input into the location detection model, and the model outputs the location information of multiple hand keypoints in the image, along with the confidence level of that location information. Alternatively, the location detection model can consist of two sub-models. Optionally, the location detection model can include: a hand region recognition sub-model and a keypoint detection sub-model; multiple images to be processed can be input into the hand region recognition sub-model to obtain a first hand region image; based on the first hand region image, a second hand region image is obtained; the second hand region image is input into the keypoint detection sub-model to obtain the location information of the hand keypoints. That is, the image to be processed is sequentially processed through the two sub-models to obtain the location information of multiple hand keypoints in the image, along with the confidence level of that location information.

[0037] The hand region recognition sub-model is used to detect the position of the hand in the image to be detected. The first hand region image is the portion of the image to be detected that contains the hand. For example, the first hand region image can be a rectangular box containing the hand. The keypoint detection sub-model is used to detect the position information of the hand's keypoints. After the hand region recognition sub-model outputs the first hand region image, it can be directly input into the keypoint detection sub-model to obtain the position information of multiple hand keypoints and the confidence level of the position information.

[0038] Alternatively, to ensure that the image input to the keypoint detection sub-model completely includes the hand and avoids cropping, the first hand region image can be expanded (e.g., expanded by 20%) to obtain a second hand region image. This ensures the integrity of the hand keypoint location information and prevents insufficient keypoint location information from affecting the accuracy of the subsequent pose calculation.

[0039] Step S230: Perform pose calculation based on the location information to obtain the pose information of the user's gesture.

[0040] Based on the positional information of multiple hand keypoints, the pose information of the user's gesture can be obtained through a pose calculation algorithm. For example, in a scene with multiple images to be processed, pose calculation can be performed through coordinate system transformation. That is, multiple images to be processed can be combined in pairs. Based on the positional information of each hand keypoint in each pair of images, depth information is determined. The depth information corresponding to various combinations is then weighted and averaged to obtain the final depth information. Next, based on this final depth information and the camera's intrinsic parameters, the positional information in the 2D image coordinate system of any image to be processed is transformed to 3D position coordinates in the camera coordinate system. Further, based on the camera's extrinsic parameters, the 3D position coordinates are transformed to the world coordinate system to obtain the 3D position coordinates in the world coordinate system. Thus, the positional information of a single hand keypoint in multiple images to be processed is transformed into a 3D position coordinate of that single hand keypoint in the world coordinate system.

[0041] Finally, combining Figure 3A The diagram showing the hand joints illustrates how the user's gesture posture information is calculated based on the positional relationships and coordinates of each key hand point. This posture information can include the degrees of freedom of each key hand point, among other things.

[0042] like Figure 3B As shown, taking the index finger as an example, assume the four key points of the hand, from the inside out, are: key point 301, key point 302, key point 303, and key point 304. Calculate the projection vectors Vz1 and Vy1 of vector V12 (the vector from key point 301 to key point 302) onto the z-axis and y-axis respectively. Then calculate the angles between Vz1 and Vy1 and the x-axis and z-axis respectively; these are the degrees of freedom information for key point 301. Calculate the projection vectors of V12 and V23 (the vectors from key point 302 to key point 303) onto the y-axis; the angle between these two projection vectors is the degree of freedom information for key point 302. The calculation method for the degrees of freedom information of key point 303 is similar to that of key point 302. Key point 304 has no degrees of freedom information.

[0043] Step S240: Based on the confidence level of the location information, determine whether to track and predict the user's gestures based on the pose information.

[0044] The confidence level of location information indicates its reliability; a higher confidence level indicates more reliable location information. Correspondingly, the posture information obtained from location information is also more accurate. Therefore, the confidence level of location information can be used to determine whether to track user gestures based on posture information. Optionally, the confidence levels of multiple hand keypoints can be weighted and averaged to obtain a first target confidence level, which can then be used to determine whether to track user gestures based on posture information. Alternatively, the minimum confidence level of multiple hand keypoints can be determined as a second target confidence level, which can then be used to determine whether to track user gestures based on posture information.

[0045] For example, for three images to be processed, the confidence scores corresponding to the location information of 63 hand key points can be obtained. The 63 confidence scores can be weighted and averaged to obtain the first target confidence score. The first target confidence score is used to determine whether to track user gestures based on pose information. Alternatively, the minimum value among the 63 confidence scores can be determined as the second target confidence score. The second target confidence score is used to determine whether to track user gestures based on pose information.

[0046] The gesture recognition method of this application acquires images of user gestures from one or more different orientations, and obtains the position information and confidence level of multiple hand key points in the images. Pose calculation is performed based on the position information to obtain the pose information of the user gesture. Since the confidence level represents the reliability of the position information, and the reliability of the position information determines the reliability of the pose information, the confidence level of the position information can be used to determine whether to track the user gesture based on the pose information, thereby improving the accuracy of gesture recognition. When acquiring images from multiple different orientations, the method can compensate for the defects of visual occlusion from a single orientation, and obtain pose information that better matches the actual gesture. When performing gesture tracking based on this pose information, the accuracy of gesture recognition can be further improved.

[0047] See Figure 4A , Figure 4A This is a flowchart of a training method for a hand region recognition sub-model in an embodiment of this application, which may include the following steps: Step S402: Obtain multiple first sample images containing hand information, hand region images in the first sample images, and hand types in the first sample images, wherein the hand type includes left hand and / or right hand.

[0048] The hand region recognition sub-model is used to identify hand regions in images. Therefore, during training, the first sample image can be an image containing hand information. The first sample image is localized and labeled, for example, from the fingertips to the wrist, using truncated rectangles, circles, etc., as annotations to obtain the hand region image. To improve the model's accuracy, the first sample image can be as rich as possible; for example, it can include hand information from various angles, skin tones, sizes, and poses.

[0049] Understandably, hands are divided into left and right hands. During gesture recognition, it's possible to recognize only one hand (left or right) or both hands simultaneously. Therefore, the first sample image can contain information about only one hand or both hands (left and right, not two left hands or two right hands). The hand region recognition sub-model, besides recognizing the hand region image in the image, can also determine whether the recognized hand region image is a left-hand or right-hand image. Correspondingly, when labeling the first sample image, the type of hand in the first sample image can also be labeled. In this way, corresponding label data can be constructed based on the hand region image and the type of hand in the first sample image.

[0050] Step S404: Using the first sample image as input data, and the hand region image and the type of hand in the first sample image as label data, a hand region recognition sub-model is trained and generated.

[0051] After constructing the input and label data, neural network training can begin. During training, gradient descent and backpropagation principles are used to continuously calculate the loss function, and the network parameters are updated based on this loss function value. The network parameters are related to the network structure of the hand region recognition sub-model; after setting the network structure, the corresponding network parameters can be configured. The loss function value is used to estimate the degree of discrepancy between predicted and actual values. During training, if the loss function value meets requirements (e.g., is less than a preset threshold), training ends, resulting in the hand region recognition sub-model; that is, the parameter values ​​of the network parameters in the hand region recognition sub-model are determined. The preset threshold can be set according to the actual application and is not limited here.

[0052] After training is complete, the accuracy of the hand region recognition sub-model can be verified using test data. If the accuracy is high, the hand region recognition sub-model can be used; if the accuracy is low, it can be retrained until the accuracy of the hand region recognition sub-model meets the requirements.

[0053] It should be noted that the gesture recognition method of this application embodiment can be applied to VR scenarios. Since the model has a large number of network parameters, computational load, and memory consumption, quantization processing can be performed on the hand region recognition sub-model to improve model running efficiency on mobile devices. Optionally, after training and generating the hand region recognition sub-model, the parameter values ​​of the network parameters in the hand region recognition sub-model are quantized. For example, the parameter values ​​of the network parameters can be quantized to 8 bits. That is, the parameter values ​​of the network parameters are converted into integers between 0 and 255, with numbers between 0 and 255 occupying 8 bits. If the parameter values ​​of the network parameters are of type float, occupying 4 bytes (32 bits), after quantization, they occupy only 8 bits, resulting in less storage space and thus improving the model's running efficiency on mobile devices.

[0054] See Figure 4B , Figure 4B This is a flowchart of a training method for a keypoint detection sub-model in an embodiment of this application, which may include the following steps: Step S410: Obtain multiple second sample images containing hand information and the location information of key points of the hand in the second sample images.

[0055] The second sample image can be a hand region image obtained by annotating the first sample image. Alternatively, the second sample image can also be an image obtained by inputting other sample images containing hand information into the aforementioned trained hand region recognition sub-model.

[0056] Similarly, the location information of hand keypoints in the second sample image can be labeled to construct corresponding label data. To ensure the accuracy of model training, labeling can be performed in a 3D scene, and hand keypoint labeling can accurately indicate the location of each hand keypoint. For unoccluded hand keypoints, the location information of the hand keypoint can be accurately determined; for occluded hand keypoints, the accurate location coordinates can be obtained by changing the angle.

[0057] Step S420: Use the second sample image as input data and the position information of the hand key points in the second sample image as label data to train and generate a key point detection sub-model.

[0058] Similarly, after constructing the input and label data, neural network training can be performed. After training, the accuracy of the keypoint detection sub-model can be verified using test data. If the accuracy is high, the keypoint detection sub-model can be used; if the accuracy is low, it can be retrained until the accuracy of the keypoint detection sub-model meets the requirements.

[0059] To improve model efficiency on mobile devices, the keypoint detection sub-model can be quantized. Unlike the aforementioned hand region recognition sub-model, the keypoint detection sub-model requires higher accuracy. Therefore, the parameter values ​​of the network parameters in the keypoint detection sub-model can be quantized during training. This allows the quantization residuals to be iteratively converged during training, improving the accuracy of the keypoint detection sub-model.

[0060] See Figure 5 , Figure 5 This is another flowchart of the gesture recognition method in the embodiments of this application, which may include the following steps: Step S510: Obtain one or more images to be processed for the user's gestures, wherein the multiple gesture images are acquired from multiple different positions.

[0061] Step S520: Based on the pre-trained position detection model, determine the position information and confidence level of multiple hand key points in the image to be processed.

[0062] Steps S510 to S520 above and Figure 2 Steps S210 to S220 are the same in the embodiment; see details below. Figure 2 The descriptions in the embodiments are sufficient and will not be repeated here.

[0063] Step S530: Based on the location information and hand model, use an optimization algorithm to calculate the pose, and obtain the pose information and residual value of the user's gesture. The hand model contains the feature information of the hand.

[0064] In this embodiment, for a scene with one or more images to be processed, the user's gesture posture information can be determined starting from a hand model using an optimization algorithm (such as the least squares optimization algorithm). This avoids the problem of the user's fingers twisting randomly and improves the accuracy of determining the user's gesture posture information. The optimization algorithm's solution process involves giving an assumed initial value, calculating the residual, correcting the initial value, until the corrected value meets a preset condition.

[0065] The specific method can be as follows: Initial pose information is set, such as initial degrees of freedom (DOF). This initial DDF can define the DDF of a hand in a zero state, such as the state with fingers spread as shown in Figure 3. Then, the 3D position coordinates of each hand keypoint in the world coordinate system can be calculated based on the initial DDF. These 3D position coordinates are then transformed to different camera coordinate systems based on the camera's extrinsic parameters, resulting in the 3D position coordinates for each camera orientation. Further, using the camera's intrinsic parameters, the 3D position coordinates in the camera coordinate system are converted into 2D position coordinates in the image coordinate system.

[0066] The difference between the calculated 2D position coordinates and the 2D position coordinates detected by the model is defined as the residual value. An optimization algorithm is used to optimize the degrees of freedom information so that the target residual value (e.g., the sum of residual values ​​in each image coordinate system) is less than a preset residual value, such as 10. In other words, if the target residual value is less than 10, the pose matching requirement is met, and the resulting degrees of freedom information is the final degree of freedom information. If the target residual value cannot be less than the preset residual value, the optimization ends after 10 iterations, regardless of whether the value is less than the preset residual value, and the final degree of freedom information and residual value are returned. The final degree of freedom information is the calculated pose information of the user gesture, and the target residual is the residual value corresponding to the pose information of the user gesture.

[0067] Step S540: Based on the confidence level and residual value of the location information, determine whether to track and predict the user's gestures based on the pose information.

[0068] Since the residual value can represent the accuracy of the user's gesture posture information, and the confidence level of the location information can also determine the accuracy of the user's gesture posture information, this application can determine whether to track and predict the user's gesture based on the confidence level of the location information and the residual value.

[0069] In one optional implementation, the residual confidence level can be determined based on the residual value and a preset residual threshold. For example, if a = the ratio of the residual value to the preset residual threshold, then when a is greater than or equal to 1 (i.e., the residual value is greater than or equal to the preset residual threshold), the residual confidence level is 0; otherwise, the residual confidence level is 1 - a. Of course, the method for determining the residual confidence level is not limited to this.

[0070] Next, the tracking confidence level is determined based on the confidence levels of the location information and the residuals. For example, the weighted average of the confidence levels of the location information and the residuals can be used as the tracking confidence level. Alternatively, the minimum of the confidence levels of the location information and the residuals can be used as the tracking confidence level.

[0071] If the tracking confidence is greater than or equal to a preset confidence threshold, then the user's gesture is tracked and predicted based on the pose information, and step S550 is executed. If the tracking confidence is less than the preset confidence threshold, then the user's gesture is not tracked and predicted based on the pose information. It should be noted that not tracking the user's gesture based on the pose information means that the accuracy of the pose information is low, and the pose information may not be used during tracking. However, the pose information of the user's gesture in the previous frame or several previous frames of the image to be processed can be used for tracking.

[0072] Step S550: Based on the pose information and the pose information of the user's gesture in the previous frame or multiple previous frames of the image to be processed, predict the user's gesture in the next frame of the image to be processed to obtain the predicted pose information of the user's gesture in the next frame.

[0073] The prediction is made based on the results of the opponent's stable tracking. The tracking records and analyzes the results and trends of each stable attitude calculation, and then makes predictions.

[0074] This application can predict the user's gesture posture information in the next frame image by using velocity difference, taking the difference between the first two frames of the image to be processed as the velocity, or taking the average rate of change of the first n frames of the image to be processed as the velocity, and predicting the gesture posture information in the next frame image based on the velocity.

[0075] Alternatively, a pre-trained CNN (Convolutional Neural Network) model can be used for prediction. The input is the pose information of the user's gesture in the image to be processed and the pose information of the user's gesture in the previous frame or multiple previous frames of the image to be processed, and the pose information of the next frame of the image to be processed can be predicted.

[0076] It should be noted that predicting the pose information of the next frame image can not only estimate the pose of the user's gesture in the next frame image for easier tracking, but also more stably detect the position coordinates of the hand region and key hand points. That is, steps S560 to S570 can be performed to correct the position information and confidence level of multiple key hand points in the next frame image.

[0077] Step S560: Obtain the next frame image of the image to be processed, and determine the position information and confidence level of multiple hand key points in the next frame image through the position detection model.

[0078] In this embodiment, since the position detection model can include two sub-models, the position information of the hand key points in the next frame of the actually acquired image to be processed can be directly corrected based on the predicted pose information of the user's gesture in the next frame image. Alternatively, the hand region in the next frame of the actually acquired image to be processed can be corrected first based on the predicted pose information of the user's gesture in the next frame image, and then the position information of the hand key points in the next frame of the actually acquired image to be processed can be corrected.

[0079] Step S570: Correct the position information and confidence level of multiple hand key points in the next frame image based on the predicted pose information.

[0080] Similar to the aforementioned optimization algorithm, coordinate system transformation can be used to correct the positional information and confidence level of multiple hand keypoints in the next frame image. Specifically, based on the predicted pose information, the first predicted position coordinates of multiple hand keypoints in the world coordinate system are determined; these first predicted position coordinates are then converted to second predicted position coordinates in multiple camera coordinate systems with different orientations; these second predicted position coordinates are then converted to third predicted position coordinates in the image coordinate system; and based on these third predicted position coordinates, the positional information of the multiple hand keypoints is corrected. While correcting the positional information, the confidence level of the positional information can also be corrected simultaneously. This ensures the stability of the detection and provides a corrective effect for hand localization, avoiding false positives and false negatives during the localization process.

[0081] The gesture recognition method of this application acquires images of user gestures from one or more different orientations, and obtains the position information and confidence level of multiple hand key points in the images. Based on the position information, an optimization algorithm is used to calculate the pose information and residual value of the user gesture. Since the confidence level represents the reliability of the position information, which determines the reliability of the pose information, and the residual value represents the accuracy of the pose information, the confidence level and residual value of the position information can be used to determine whether to track the user gesture based on the pose information, thereby improving the accuracy of gesture recognition. When acquiring images from multiple different orientations, the method can compensate for the defects of visual occlusion from a single orientation, and obtain pose information that better matches the real gesture. When performing gesture tracking based on this pose information, the predicted pose information of the user gesture in the next frame image can be predicted, and the position information and confidence level of multiple hand key points in the actually acquired next frame image can be corrected to ensure the accuracy of position detection.

[0082] Corresponding to the above method embodiments, this application also provides a gesture recognition device, see [link to relevant documentation]. Figure 6 The gesture recognition device 600 includes: The image acquisition module 610 is used to acquire one or more images to be processed for user gestures, wherein multiple gesture images are acquired from multiple different positions; The location information determination module 620 is used to determine the location information and confidence level of multiple hand key points in the image to be processed based on a pre-trained location detection model. The posture information determination module 630 is used to perform posture calculation based on the position information to obtain the posture information of the user's gesture; The gesture tracking and judgment module 640 is used to determine whether to track and predict user gestures based on the confidence level of the location information.

[0083] Optionally, the pose information determination module 630 is specifically used to perform pose calculation using an optimization algorithm based on the position information and the hand model to obtain the pose information and residual value of the user's gesture; wherein, the hand model contains the feature information of the hand; The gesture tracking and judgment module 640 is specifically used to determine whether to track and predict user gestures based on posture information, according to the confidence level and residual value of the position information.

[0084] Optionally, the gesture tracking judgment module 640 is specifically used to determine the residual confidence based on the residual value and a preset residual threshold; determine the tracking confidence based on the confidence of the location information and the residual confidence; if the tracking confidence is greater than or equal to the preset confidence threshold, then the user's gesture is tracked and predicted based on the posture information; if the tracking confidence is less than the preset confidence threshold, then the user's gesture is not tracked and predicted based on the posture information.

[0085] Optionally, the gesture recognition device 600 also includes: The pose information prediction module is used to track and predict user gestures based on pose information. Based on the pose information and the pose information of the user gestures in the previous frame or multiple previous frames of the image to be processed, the module predicts the user gestures in the next frame of the image to be processed, and obtains the predicted pose information of the user gestures in the next frame. The image acquisition module 610 is also used to acquire the next frame of the image to be processed; The position information determination module 620 is also used to determine the position information and confidence level of multiple hand key points in the next frame image through a position detection model; The position information correction module is used to correct the position information and confidence level of multiple hand key points in the next frame image based on the predicted pose information.

[0086] Optionally, the position information correction module is specifically used to determine the first predicted position coordinates of multiple hand key points in the world coordinate system based on the predicted pose information; convert the first predicted position coordinates into second predicted position coordinates in multiple camera coordinate systems with different orientations; convert the second predicted position coordinates into third predicted position coordinates in the image coordinate system; and correct the position information of multiple hand key points based on the third predicted position coordinates.

[0087] Optionally, the location detection model includes: a hand region recognition sub-model and a key point detection sub-model; The location information determination module 620 is specifically used to input multiple images to be processed into the hand region recognition sub-model to obtain a first hand region image; based on the first hand region image, obtain a second hand region image; and input the second hand region image into the key point detection sub-model to obtain the location information of the hand key points.

[0088] Optionally, the gesture recognition device 600 also includes: The first training data acquisition module is used to acquire multiple first sample images containing hand information, hand region images in the first sample images, and hand types in the first sample images, wherein the hand types include left hand and / or right hand; The hand region recognition sub-model generation module is used to take the first sample image as input data, and the hand region image and the type of hand in the first sample image as label data to train and generate the hand region recognition sub-model. The second training data acquisition module is used to acquire multiple second sample images containing hand information and the location information of key hand points in the second sample images; The keypoint detection sub-model generation module is used to train and generate a keypoint detection sub-model by taking the second sample image as input data and the position information of the hand keypoints in the second sample image as label data.

[0089] Optionally, the gesture recognition device 600 also includes: The first quantization module is used to quantize the parameter values ​​of the network parameters in the hand region recognition sub-model after training and generating the sub-model; and / or; The second quantization module is used to quantize the parameter values ​​of the network parameters in the keypoint detection sub-model during training.

[0090] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.

[0091] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0092] In an exemplary embodiment of this application, an electronic device is also provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the gesture recognition method described in this exemplary embodiment.

[0093] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. It should be noted that... Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0094] like Figure 7 As shown, the electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0095] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a local area network (LAN) card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 710 as needed so that computer programs read from it can be installed into storage section 708 as needed.

[0096] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit 701, it performs the various functions defined in the apparatus of this application.

[0097] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the above-described gesture recognition method.

[0098] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0099] In this embodiment of the application, a computer program product is also provided, which, when run on a computer, causes the computer to execute the above-described gesture recognition method.

[0100] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition method, characterized in that, The method includes: Acquire one or more images to be processed in response to a user gesture, wherein the multiple images to be processed are acquired from multiple different orientations when acquiring the multiple images to be processed; Based on a pre-trained position detection model, the position information of multiple hand key points in the image to be processed and the confidence level of the position information are determined. The posture is calculated based on the position information to obtain the posture information of the user gesture, wherein the posture information of the user gesture is the degree of freedom information of each hand key point; Based on the confidence level of the location information, determine whether to track and predict the user's gestures based on the posture information; The step of calculating the pose based on the location information to obtain the pose information of the user's gesture includes: Based on the location information and the hand model, an optimization algorithm is used to calculate the pose, thereby obtaining the pose information and residual value of the user's gesture; wherein, the hand model includes the feature information of the hand; The step of determining whether to track and predict the user's gesture based on the pose information according to the confidence level of the location information includes: Based on the confidence level of the location information and the residual value, determine whether to track and predict the user's gesture based on the posture information.

2. The method according to claim 1, characterized in that, The step of determining whether to track and predict the user's gesture based on the pose information, based on the confidence level of the location information and the residual value, includes: The residual confidence level is determined based on the residual value and the preset residual threshold. The tracking confidence level is determined based on the confidence level of the location information and the confidence level of the residual. If the tracking confidence is greater than or equal to a preset confidence threshold, then the user's gesture is tracked and predicted based on the posture information; If the tracking confidence is less than the preset confidence threshold, then the user's gesture will not be tracked or predicted based on the posture information.

3. The method according to claim 1, characterized in that, The method further includes: If the user gesture is tracked and predicted based on the posture information, the user gesture in the next frame of the image to be processed is predicted based on the posture information and the posture information of the user gesture in the previous frame or multiple previous frames of the image to be processed, so as to obtain the predicted posture information of the user gesture in the next frame. The next frame of the image to be processed is obtained, and the position information of multiple hand key points in the next frame and the confidence level of the position information are determined by the position detection model. The positional information of multiple hand key points in the next frame image and the confidence level of the positional information are corrected based on the predicted pose information.

4. The method according to claim 3, characterized in that, The positional information of multiple hand key points in the next frame image is corrected based on the predicted pose information, including: Based on the predicted posture information, determine the first predicted position coordinates of the multiple hand key points in the world coordinate system; The first predicted position coordinates are converted into second predicted position coordinates in the multiple camera coordinate systems of different orientations; Convert the second predicted position coordinates into the third predicted position coordinates in the image coordinate system; Based on the third predicted position coordinates, the position information of multiple key hand points is corrected.

5. The method according to claim 1, characterized in that, The location detection model includes: a hand region recognition sub-model and a key point detection sub-model; The method based on a pre-trained position detection model determines the position information of multiple hand key points in the image to be processed and the confidence level of the position information, including: The multiple images to be processed are respectively input into the hand region recognition sub-model to obtain the first hand region image; Based on the first hand region image, a second hand region image is obtained; The second hand region image is input into the key point detection sub-model to obtain the position information of the hand key points.

6. The method according to claim 5, characterized in that, The training method for the hand region recognition sub-model includes: Acquire multiple first sample images containing hand information, hand region images in the first sample images, and hand types in the first sample images, wherein the hand type includes left hand and / or right hand; The first sample image is used as input data, and the hand region image and the type of hand in the first sample image are used as label data to train and generate the hand region recognition sub-model. The training method for the keypoint detection sub-model includes: Acquire multiple second sample images containing hand information, and the location information of key hand points in the second sample images; The second sample image is used as input data, and the location information of the key points of the hand in the second sample image is used as label data to train and generate the key point detection sub-model.

7. The method according to claim 6, characterized in that, The method further includes: After training and generating the hand region recognition sub-model, the parameter values ​​of the network parameters in the hand region recognition sub-model are quantized; and / or; When training the keypoint detection sub-model, the parameter values ​​of the network parameters in the keypoint detection sub-model are quantized.

8. A gesture recognition device, characterized in that, The device includes: The image acquisition module is used to acquire one or more images to be processed based on the user's gesture, wherein the multiple images to be processed are acquired from multiple different positions when acquiring the multiple images to be processed; The location information determination module is used to determine the location information of multiple hand key points in the image to be processed and the confidence level of the location information based on a pre-trained location detection model. The posture information determination module is used to perform posture calculation based on the position information to obtain the posture information of the user gesture, wherein the posture information of the user gesture is the degree of freedom information of each hand key point; The gesture tracking and judgment module is used to determine whether to track and predict the user's gesture based on the confidence level of the location information. The step of calculating the pose based on the location information to obtain the pose information of the user's gesture includes: Based on the location information and the hand model, an optimization algorithm is used to calculate the pose, thereby obtaining the pose information and residual value of the user's gesture; wherein, the hand model includes the feature information of the hand; The step of determining whether to track and predict the user's gesture based on the pose information according to the confidence level of the location information includes: Based on the confidence level of the location information and the residual value, determine whether to track and predict the user's gesture based on the posture information.

9. An electronic device, characterized in that, include: A processor for executing a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1-7.

11. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the steps of the method according to any one of claims 1-7.

Images