A two-handed three-dimensional gesture estimation method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a two-hand three-dimensional gesture estimation method, device, electronic equipment and storage medium, the method comprising: performing two-hand key point detection on an acquired two-hand image to obtain two-dimensional position information of each key point in a two-hand key point set; determining three-dimensional feature information of each key point from a feature image set of the two-hand image based on the two-dimensional position information of each key point; using an attention mechanism to perform associated key point feature fusion on the three-dimensional feature information of each key point to obtain target feature information of each key point; and performing gesture estimation on the target feature information of each key point to obtain three-dimensional position information of each key point. The two-hand three-dimensional gesture estimation method of the present disclosure can simultaneously predict the three-dimensional gestures of both hands, which is simple and convenient; meanwhile, the three-dimensional gestures of both hands can be accurately predicted, and the effect is obviously improved when facing the prediction of a two-hand gesture such as a two-hand fist hug with a large interaction area.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet, and particularly relates to a two-hand three-dimensional gesture estimation method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the research of gesture estimation, hand-based gesture estimation research is favored. Among all human gestures, hand gestures account for 90%, which is the most important human-computer interaction gesture. Future life scenarios are developing towards more and more intelligent direction. Smart home, automatic driving, intelligent medical treatment and first-person immersive interaction application scenarios cannot do without the presence of hand gesture interaction.

[0003] In the related art, hand gesture estimation is mainly designed for single-hand scenarios. When applied to predict two-hand three-dimensional gestures, two hands are respectively cropped from the image as two single hands for processing, the method of single-hand three-dimensional gesture estimation is used to predict the gesture of each hand, and finally the prediction results of the two hands are combined together as the final two-hand gesture estimation result. However, in this processing mode, the two-hand three-dimensional gesture cannot be accurately predicted. SUMMARY

[0004] The present disclosure provides a two-hand three-dimensional gesture estimation method, device, electronic equipment and storage medium. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of the embodiment of the present disclosure, a two-hand three-dimensional gesture estimation method is provided, comprising:

[0006] Performing two-hand key point detection on the obtained two-hand image to obtain two-dimensional position information of each key point in a two-hand key point set;

[0007] Based on the two-dimensional position information of each key point, determining three-dimensional feature information of each key point from a feature image set of the two-hand image;

[0008] Using an attention mechanism, performing associated key point feature fusion on the three-dimensional feature information of each key point to obtain target feature information of each key point;

[0009] Performing gesture estimation on the target feature information of each key point to obtain three-dimensional position information of each key point; the three-dimensional position information of each key point is used to determine a three-dimensional gesture of the two hands in the two-hand image.

[0010] In some possible embodiments, performing two-hand key point detection on the obtained two-hand image to obtain two-dimensional position information of each key point in a two-hand key point set, comprising:

[0011] The multi-scale feature extraction is performed on the two-hand image to obtain a feature image set; the feature image set includes a plurality of feature images of different scales;

[0012] The target feature image is determined from the plurality of feature images of different scales; the scale of the target feature image is the same as the scale of the two-hand image;

[0013] The two-hand key point detection is performed on the target feature image to obtain the two-dimensional position information of each key point.

[0014] In some possible embodiments, the two-hand key point detection is performed on the target feature image to obtain the two-dimensional position information of each key point, including:

[0015] The key point thermal analysis is performed on the target feature image to obtain the thermal map corresponding to each key point; the scale of the thermal map is the same as the scale of the target feature image;

[0016] According to the thermal map corresponding to each key point, the confidence of each key point at each pixel position in the target feature image is determined;

[0017] For each key point, the pixel position with the maximum confidence in the target feature image is taken as the pixel position of each key point;

[0018] According to the pixel position of each key point, the two-dimensional position information of each key point is obtained.

[0019] In some possible embodiments, based on the two-dimensional position information of each key point, the three-dimensional feature information of each key point is determined from the feature image set of the two-hand image, including:

[0020] In each feature image of the plurality of feature images, the key pixel position corresponding to the two-dimensional position information of each key point is determined;

[0021] The feature information at the key pixel position corresponding to each key point in each feature image is fused to obtain the three-dimensional feature information of each key point.

[0022] In some possible embodiments, the attention mechanism is used to perform the associated key point feature fusion on the three-dimensional feature information of each key point to obtain the target feature information of each key point, including:

[0023] The three-dimensional feature information of each key point in the two-hand key point set is input into the attention module to output the attention value between any two key points in the two-hand key point set; the attention value between any two key points represents the correlation between any two key points;

[0024] The three-dimensional feature information of each key point is associated and fused by an attention value between any two key points to obtain target feature information of each key point.

[0025] In some possible embodiments, the method further includes:

[0026] obtaining an initial image;

[0027] performing double-hand detection on the initial image to obtain position information of a double-hand region in the initial image;

[0028] cropping the initial image according to the position information of the double-hand region to obtain a double-hand image.

[0029] In some possible embodiments, the method further includes:

[0030] obtaining a training double-hand image; the training double-hand image carries labeled information; the labeled information includes actual three-dimensional position information of each key point in a double-hand key point set and actual two-dimensional position information of each key point;

[0031] performing double-hand key point detection on the training double-hand image by using a preset machine learning model to obtain predicted two-dimensional position information of each key point; determining predicted three-dimensional feature information of each key point from a feature image set of the training double-hand image based on the predicted two-dimensional position information of each key point; performing associated key point feature fusion on the predicted three-dimensional feature information of each key point by using an attention mechanism to obtain predicted target feature information of each key point; and performing pose estimation on the predicted target feature information of each key point to obtain predicted three-dimensional position information of each key point;

[0032] training the preset machine learning model based on a first loss value between the predicted three-dimensional position information of each key point and actual three-dimensional position information of the key point and a second loss value between the predicted two-dimensional position information of each key point and the actual two-dimensional position information of each key point until a preset end condition is met to obtain a double-hand three-dimensional pose estimation model;

[0033] The double-hand three-dimensional pose estimation model is configured to perform double-hand key point detection on an obtained double-hand image until three-dimensional position information of each key point is obtained.

[0034] According to a second aspect of the embodiments of the present disclosure, a double-hand three-dimensional pose estimation apparatus is provided, including:

[0035] a detection module configured to perform double-hand key point detection on an obtained double-hand image to obtain two-dimensional position information of each key point in a double-hand key point set;

[0036] The determining module is configured to determine three-dimensional feature information of each key point from the feature image set of the double-hand image based on two-dimensional position information of each key point.

[0037] The fusing module is configured to perform associated key point feature fusion on the three-dimensional feature information of each key point by using an attention mechanism to obtain target feature information of each key point.

[0038] The estimating module is configured to perform pose estimation on the target feature information of each key point to obtain three-dimensional position information of each key point, and the three-dimensional position information of each key point is used to determine a three-dimensional pose of the double hands in the double-hand image.

[0039] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising:

[0040] a processor;

[0041] a memory for storing processor-executable instructions;

[0042] The processor is configured to execute the instructions to implement the double-hand three-dimensional pose estimation method according to the first aspect of the present disclosure.

[0043] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the double-hand three-dimensional pose estimation method according to the first aspect of the present disclosure.

[0044] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0045] The double-hand three-dimensional pose estimation method according to the embodiments of the present disclosure can directly predict the three-dimensional poses of the two hands as a whole, which is more concise and convenient than the existing double-hand prediction scheme based on single-hand three-dimensional pose estimation, and no longer needs to use a hand detection module to position and crop the two hands. At the same time, by using an attention mechanism to perform associated key point feature fusion on the three-dimensional feature information of each key point, the influence of key points with strong correlation can be enhanced, the influence of key points with weak correlation can be weakened, and the mutual interference of the two hands can be reduced, so that the three-dimensional poses of the two hands can be more accurately predicted compared with the existing method, and the effect is obviously improved when facing a double-hand gesture prediction with a large interaction area such as a double-hand fist.

[0046] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and, do not limit the present disclosure.

[0048] Figure 1 is a schematic diagram of an application environment according to an example embodiment;

[0049] Figure 2 is a flowchart of a method for estimating a three-dimensional pose of two hands according to an example embodiment;

[0050] Figure 3 is a flowchart of obtaining a two-hand image according to an example embodiment;

[0051] Figure 4 is a schematic diagram of two-hand detection according to an example embodiment;

[0052] Figure 5 is a flowchart of two-dimensional key point detection according to an example embodiment;

[0053] Figure 6 is a schematic diagram of two-dimensional key point detection according to an example embodiment;

[0054] Figure 7 is a flowchart of two-dimensional key point detection according to an example embodiment;

[0055] Figure 8 is a flowchart of obtaining three-dimensional feature information of each key point according to an example embodiment;

[0056] Figure 9 is a schematic diagram of three-dimensional feature information of a key point according to an example embodiment;

[0057] Figure 10 is a flowchart of an attention mechanism according to an example embodiment;

[0058] Figure 11 is a schematic diagram of an attention module according to an example embodiment;

[0059] Figure 12 is a schematic diagram of matrix calculation according to an example embodiment;

[0060] Figure 13 is a flowchart of a training method of a two-hand three-dimensional pose estimation model according to an example embodiment;

[0061] Figure 14is a structural block diagram of a preset machine learning model according to an exemplary embodiment;

[0062] Figure 15 is a block diagram of a dual-hand three-dimensional pose estimation device according to an exemplary embodiment;

[0063] Figure 16 is a block diagram of an electronic device for dual-hand three-dimensional pose estimation according to an exemplary embodiment. DETAILED DESCRIPTION

[0064] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0066] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) involved in the present disclosure is all information authorized by the user or authorized by all parties.

[0067] Please refer to Figure 1 , Figure 1 According to an exemplary embodiment, a schematic diagram of an application environment of a dual-hand three-dimensional pose estimation method is shown in FIG. 1, which includes a server 01 and a terminal device 02. Optionally, the server 01 and the terminal device 02 can be connected through a wireless link, or can be connected through a wired link, which is not limited in the present disclosure. Figure 1

[0068] Dual-hand three-dimensional pose estimation is to predict the coordinate position of the key points of the camera system from the image, or to calculate the spatial position of other key points relative to the zero point with a certain dual-hand key point as the zero point. In Figure 1 the application scenario, the terminal device 02 can acquire a dual-hand image, which can be an RGB image, and then send the acquired dual-hand image to the server 01, so that the server 01 performs a series of image processing operations on the acquired dual-hand image to obtain the three-dimensional position information of each key point, thereby realizing dual-hand three-dimensional pose estimation. ​

[0069] In some possible embodiments, the server 01 can include a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), and big data and artificial intelligence platform. The operating system running on the server can include but is not limited to Android system, IOS system, Linux, Windows, Unix, etc.

[0070] In some possible embodiments, the terminal device 02 described above can include but is not limited to a smart phone, a desktop computer, a tablet computer, a notebook computer, a smart speaker, a digital assistant, an augmented reality (AR) / virtual reality (VR) device, a smart wearable device, and the like. It can also be a software running on the client, such as an application, a mini program, etc. Optionally, the operating system running on the client can include but is not limited to Android system, IOS system, Linux, Windows, Unix, etc.

[0071] In addition, it should be noted that, Figure 1 The application environment shown is only an example. In actual application, the terminal device and the server can cooperate to perform the double-hand three-dimensional pose estimation method of the embodiments of the present disclosure, or the terminal device or the server can independently perform the double-hand three-dimensional pose estimation method of the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit the specific application environment.

[0072] Figure 2 is a flow chart of a double-hand three-dimensional pose estimation method according to an example embodiment, as shown in Figure 2 As shown, the double-hand three-dimensional pose estimation method can be applied to a server, including the following steps:

[0073] In step S201, the obtained double-hand image is subjected to double-hand key point detection to obtain two-dimensional position information of each key point in the double-hand key point set.

[0074] The embodiments of the present disclosure are mainly aimed at three-dimensional pose estimation of double hands in a double-hand scene, and therefore the double-hand image is an image containing double hands. The double-hand image can be collected by a camera module of a terminal device, and the server receives the double-hand image sent by the terminal device and performs a series of image processing operations on the double-hand image. In other embodiments, if the double-hand three-dimensional pose estimation method is performed by the terminal device alone, the terminal device can perform real-time processing locally after collecting the double-hand image.

[0075] Here, the proportion of the double hands in the double-hand image needs to exceed a certain proportion, and an exemplary proportion can be 80%; in some application scenarios, such as a first-view immersive interaction scenario, three-dimensional pose estimation of a hand in a picture is required, and usually a hand only occupies a small part of the entire picture, so for such application scenarios, the server needs to first perform hand recognition on the entire picture, crop the hand region from the picture, and obtain a hand image.

[0076] Correspondingly, in some possible embodiments, before step S201, the method of the embodiments of the present disclosure can further include the following steps as shown in Figure 3

[0077] S301: Obtain an initial image.

[0078] Here, the initial image refers to an original image collected by a camera module in different application scenarios, and the original image includes at least one double-hand region.

[0079] S303: Perform double-hand detection on the initial image to obtain position information of the double-hand region in the initial image.

[0080] In this step, as shown in Figure 4 the server performs double-hand detection on the initial image, identifies the region and position of the double hands in the initial image, and obtains the double-hand region and the position information of the double-hand region. Here, the position information refers to the two-dimensional coordinates of the four vertices of the bounding box of the double-hand gesture in the plane initial image.

[0081] In one specific embodiment, the server can use a lightweight regression model to predict the double-hand region in the initial image. Specifically, the server can pre-collect a large number of training images containing double-hand gestures, and use manual or automatic labeling to label the region of the double-hand gesture and the meaning of the gesture in the training image. In order to improve the detection ability and application range of the model, double-hand gestures indicating different meanings can be collected, including but not limited to clasped hands, prayer, applause, handshaking, and other self-defined double-hand gestures; then, the model is trained using the training images carrying the labeling information. The model framework can refer to the framework of detection models such as YOLO, and after meeting certain model precision or other training conditions, a trained double-hand detection model is obtained.

[0082] S305: Crop the initial image according to the position information of the double-hand region to obtain a double-hand image.

[0083] In this step, after the server detects the position information of the double-hand region from the initial image, as shown in Figure 4 ​As shown, based on the position information of the double-hand region, the initial image is cropped to obtain a double-hand image. Further, during the cropping, a certain proportion can be expanded based on the position information of the double-hand region, and then the cropping is performed; for example, the expansion proportion can be 20%, so that the double-hand region can be completely cropped out, and the double-hand information can be avoided from being lost.

[0084] In the above embodiment, the server performs double-hand detection on the initial image, completely and accurately segments the double-hand region from the initial image, and separates the double-hand from the mixed background, so that the subsequent positioning of the double-hand key points is more accurate, and the reduction of the pixels of the obtained double-hand image after processing also reduces the subsequent calculation amount, which helps to improve the efficiency of the subsequent three-dimensional pose estimation.

[0085] Before introducing how to detect the double-hand key points in the double-hand image, first, it is explained that each key point in the double-hand key point set of the embodiment of the present disclosure is defined in the commonly used hand model in pose estimation, and each key point represents a joint of the hand; the number of key points defined in different hand models is different, for example, in the hand 21 key point model, the key point numbered 0 is the wrist, and each finger has 4 key points: the knuckle, the proximal knuckle, the distal knuckle, and the fingertip. The difference is that the embodiment of the present disclosure further distinguishes the left-hand key point and the right-hand key point, so that the double-hand key set of the embodiment of the present disclosure contains 42 key points, and the definition of the single-hand key point can refer to the hand 21 key point model. In the embodiment of the present disclosure, taking the double-hand 42 key point model as an example, the prediction of the key points in the two-dimensional space and the three-dimensional space is based on this model; in actual application, it can also be based on other key point models.

[0086] In some possible embodiments, the server obtains the double-hand key point set by performing double-hand key point detection on the obtained double-hand image through the key point two-dimensional position detection module; the key point two-dimensional position detection module includes a feature extraction submodule and a prediction submodule, and correspondingly, step S201 can include the following steps as shown in Figure 5

[0087] In step S501, multi-scale feature extraction is performed on the double-hand image to obtain a feature image set; the feature image set includes a plurality of feature images of different scales.

[0088] ​Specifically, the server can perform multi-scale feature extraction on the hand images through the feature extraction submodule, outputting multiple feature images at different scales to form a feature image set. Multi-scale feature extraction is a commonly used method in the field of image feature extraction, which can effectively integrate features from receptive fields at different scales to obtain different semantic information. Here, the feature extraction submodule is designed to extract two-dimensional and three-dimensional spatial feature information from the image from multiple dimensions as much as possible, including but not limited to edge information, texture information, and depth information.

[0089] In step S503, a target feature image is determined from multiple feature images at different scales; the scale of the target feature image is the same as the scale of the hand images.

[0090] In this step, in order to predict the two-dimensional position information of each key point, a target feature image is determined from multiple feature images of different scales. The scale of the target feature image is the same as that of the hand images; here, the same scale means that the length and width are the same.

[0091] In step S505, the target feature image is subjected to two-hand key point detection to obtain the two-dimensional position information of each key point.

[0092] In this step, the server performs regression prediction on each of the 42 key points of the hands through the prediction submodule based on the target feature image, and obtains the two-dimensional position information of each key point, that is, the two-dimensional coordinates in the hands image.

[0093] In a specific example, such as Figure 6 As shown, Figure 6 The following example illustrates a simple two-handed gesture. The server performs multi-scale feature extraction on the two-handed image using a feature extraction submodule, resulting in a feature image set. The last layer of the feature image set is the target feature image, which has the same dimensions as the two-handed image. Then, the prediction submodule predicts the target feature image to obtain the two-dimensional coordinates of each of the 42 key points in the two-handed image (the figure exemplarily shows the key points of the left wrist, the fingertips of the left hand, the right wrist, and the fingertips of the right hand; not all 42 key points are shown).

[0094] In the above embodiments, by performing multi-scale feature extraction on the images of both hands, and then performing regression prediction on the key points of both hands based on the target feature image that integrates features of each scale, the accuracy of the position prediction of the key points of both hands can be improved. This is beneficial for accurately obtaining the three-dimensional feature information of each key point from the feature image set and improving the accuracy of the three-dimensional pose estimation of both hands.

[0095] In one specific embodiment, step S505 may specifically include, as follows:Figure 7 The following steps are shown:

[0096] In step S701, key point heat analysis is performed on the target feature image to obtain a heat map corresponding to each key point.

[0097] In this step, the server further performs key point heat analysis on the target feature image in the prediction submodule to determine a heat map for each key point. It can be understood that the scale of the heat map is the same as that of the target feature image; each point on the heat map represents the likelihood of the corresponding key point appearing at this position.

[0098] In step S703, the confidence of each key point at each pixel position in the target feature image is determined according to the heat map corresponding to each key point.

[0099] In this step, the server calculates the heat map corresponding to each key point to obtain the confidence of the key point at each pixel position in the target feature image. Confidence refers to reliability; the greater the confidence, the greater the likelihood of the key point appearing at the corresponding pixel position.

[0100] In step S705, the pixel position with the highest confidence in the target feature image is determined as the pixel position of each key point.

[0101] Optionally, the server determines the pixel position of each key point in the target feature image as the pixel position of the key point with a confidence greater than or equal to a preset value.

[0102] The preset value can be adaptively adjusted according to actual application, which can be a fixed value. For example, the confidence can be 0.98 or 0.99. The preset value can also be dynamically changed according to the calculation result each time. For example, the maximum confidence value in the target feature image is used as the preset value. In order to ensure that the pixel position of the key point can be determined, the latter method is usually used in actual application, i.e., the preset value is the maximum confidence value in the target feature image. Accordingly, this step can be simplified to: for each key point, the pixel position with the maximum confidence value in the target feature image is determined as the pixel position of the key point.

[0103] In step S707, two-dimensional position information of each key point is obtained according to the pixel position of each key point.

[0104] In this step, the server determines the two-dimensional coordinates of the pixel position as the two-dimensional position information of the key point after determining the pixel position of the key point.

[0105] In the above embodiments, the server predicts the two-dimensional position of each key point in the image based on the heat map method, which can improve the accuracy of the key point two-dimensional position information prediction compared with directly regressing the position of the key point, and can improve the accuracy of the three-dimensional feature information of the key point based on the high-accuracy key point two-dimensional position, thereby improving the accuracy of the final three-dimensional pose estimation.

[0106] In step S203, the three-dimensional feature information of each key point is determined from the feature image set of the double-hand image based on the two-dimensional position information of each key point.

[0107] In the embodiments of the present disclosure, the server determines the three-dimensional feature information of each key point from the feature image set of the double-hand image based on the two-dimensional position information of each key point, and the three-dimensional feature information is a three-dimensional feature vector.

[0108] In some possible embodiments, the determination of the three-dimensional feature information of each key point from the feature image set of the double-hand image based on the two-dimensional position information of each key point can include the following steps as shown in Figure 8

[0109] In step S801, the two-dimensional position information of each key point is determined in each feature image in the plurality of feature images.

[0110] In this step, the server determines the key pixel position of each key point in each feature image in the plurality of feature images based on the two-dimensional position information of the key point.

[0111] Specifically, the two-dimensional position information of each key point can be normalized two-dimensional coordinates, that is, the server maps the two-dimensional coordinates of the key point based on the scale of the double-hand image to the interval [0, 1]; for example, Figure 6 In the example in Table 1, the double-hand image is 64*64, and the two-dimensional coordinates of the left-hand key point 1 are (16, 16), and the normalized two-dimensional coordinates are (16 / 64, 16 / 64), that is, (0.25, 0.25). Then, the normalized two-dimensional coordinates are mapped to each feature image to obtain the two-dimensional coordinates of each key point in each feature image and the key pixel position corresponding to the two-dimensional coordinates; for example, Figure 6 In the example in Table 1, the scale of the feature image at the second-to-last layer of the feature image set is 32*32, and the normalized two-dimensional coordinates (0.25, 0.25) of the left-hand key point 1 are mapped to the feature image to obtain the two-dimensional coordinates (0.25*32, 0.25*32), that is, (8, 8) of the left-hand key point 1 in the feature image and the key pixel position corresponding to the position. ​

[0112] In step S803, the feature information at the key pixel positions corresponding to each key point in each feature image is fused to obtain three-dimensional feature information of each key point.

[0113] In this step, after determining the key pixel positions corresponding to each key point in each feature image, the server fuses the feature information at the key pixel positions corresponding to each feature image. Specifically, the feature information after splicing can be used as the three-dimensional feature information of the key point.

[0114] As shown in FIG. 8, taking the left hand key point 1 as an example, after determining the corresponding key pixel positions in each feature image, the feature information at the key pixel positions is spliced to obtain the three-dimensional feature information of the left hand key point 1. Figure 9

[0115] In the above embodiment, the server uses the two-dimensional position information of each key point obtained in the previous step to obtain the key pixel positions in the global feature for each key point, so as to filter out the most important key point features. In this way, the most useful features of the key points are extracted as much as possible instead of all the features in the image for prediction of three-dimensional pose estimation. This can reduce the influence of redundant features on the three-dimensional pose estimation result. Moreover, the three-dimensional feature information of each key point is obtained by fusing the features extracted from multiple feature images. The three-dimensional feature information of the key point obtained in this way covers different scale feature information and has strong feature expression ability, which is very beneficial to the subsequent three-dimensional position prediction of the key point.

[0116] In step S205, the three-dimensional feature information of each key point is associated and fused by using an attention mechanism to obtain target feature information of each key point.

[0117] In the embodiment of the present disclosure, after determining the three-dimensional feature information of each key point, the server considers that, in a two-hand pose, not only the relative positions between the key points of a single hand are associated, but also the relative positions between the key points of the left hand and the right hand have certain spatial features. For example, in a fist pose, the key points on the fingers of the left hand and the key points on the corresponding fingers of the right hand are staggered in the three-dimensional space. Therefore, the present disclosure uses the spatial features between the key points of the left hand and the key points of the right hand in a specific two-hand pose to associate and fuse the three-dimensional feature information of each key point by using an attention mechanism to obtain target feature information of each key point. That is, the associated key points of each key point are determined, and the three-dimensional feature information of the associated key points is fused on the basis of the three-dimensional feature information of each key point. The fused feature is used as the target feature information of each key point.

[0118] ​In some possible embodiments, the server implements the associated key point feature fusion on the three-dimensional feature information of each key point through the attention module to obtain target feature information of each key point; correspondingly, the above step S205 can include the following steps as shown in Figure 10

[0119] In step S1001, the three-dimensional feature information of each key point in the double-hand key point set is input into the attention module, and the attention value between any two key points in the double-hand key point set is output; the attention value between any two key points represents the correlation between any two key points.

[0120] In this step, the server inputs the three-dimensional feature information of each key point into the attention module, and automatically calculates the attention value between any two key points in the 42 key points of the double hands in the attention module; the attention value between any two key points represents the correlation between any two key points. At present, there are various methods for calculating the attention value, and exemplarily, the dot product can be used to calculate the attention value.

[0121] In one specific example, as shown in Figure 11 In the attention module, the three-dimensional feature information of the 42 key points of the double hands is first linearly calculated with three weight matrix parameters Wq, Wk and Wv, which are obtained through learning; the three-dimensional feature information of each key point in the 42 key points of the double hands is multiplied by the first weight matrix parameter Wq to obtain a first input matrix Q; the three-dimensional feature information of each key point in the 42 key points of the double hands is multiplied by the second weight matrix parameter Wk to obtain a second input matrix K; the three-dimensional feature information of each key point in the 42 key points of the double hands is multiplied by the third weight matrix parameter Wv to obtain a third input matrix V; then, the first input matrix Q and the second input matrix K are used to calculate the correlation information between each two key points, that is, to calculate the attention value. At present, there are various methods for calculating the attention value, and here the dot product can be used to obtain an attention matrix A; each element in the attention matrix A is the attention value between any two key points; then, optionally, each element in the attention matrix A can be normalized by softmax to map the attention value to the interval [0, 1] to obtain a normalized attention matrix A'.

[0122] In step S1003, based on the attention value between any two key points, the three-dimensional feature information of each key point is subjected to the associated key point feature fusion to obtain the target feature information of each key point.

[0123] ​In this step, the server can perform corresponding operations based on the correlation information between any two key points, which can be normalized correlation information, combined with the three-dimensional feature information of each key point, to realize the fusion of associated key point features and obtain the target feature information of each key point.

[0124] Specifically, in combination with Figure 11 and Figure 12 As shown in the attention module, the server multiplies the attention matrix A' and the third input matrix V, where in the attention matrix A', in terms of the dimension of the column, each column represents the attention value between a key point and other key points including itself. When multiplying each column element and the corresponding element in the third input matrix V, the result of multiplication is the target feature information of the key point. Taking the first column in the attention matrix A' as an example, the element [α1,1] in the column represents the attention value between the left hand key point 1 and the left hand key point 1, the element [α1,2] represents the attention value between the left hand key point 1 and the left hand key point 2, and so on, and the element [α1,42] represents the attention value between the left hand key point 1 and the right hand key point 21. The attention value between the left hand key point 1 and all other key points, i.e., the first column element, is multiplied by the element v1 corresponding to the left hand key point in the third input matrix V to obtain b1, which is the target feature information of the left hand key point. The other columns are the same, and thus the target feature information [b1, b2, …, b42] of each key point in the 42 key points of the two hands is obtained, and the target feature matrix B is obtained.

[0125] In the above embodiment, the server uses the attention mechanism to search for associated key points that have correlations with each key point in the global range, adaptively obtains the degree of influence of each key point on other key points by calculating the correlation between different key points, and fuses the features of each key point and the features of the associated key points through weighted fusion, so that the target feature information after fusion carries the spatial features of the real pose, which can improve the accuracy of the three-dimensional position prediction of each key point.

[0126] Compared with the method of using a single-hand pose estimation model to estimate the pose of the left hand and the right hand respectively in the related art when facing the estimation of the three-dimensional pose of the two hands, since there is a large area of overlap between the left hand and the right hand when interacting, the key points of the two hands have a large interference, for example, when predicting the key points of the left hand in a fist pose, the key points of the right hand are easily misdetected, thereby causing the accuracy of the final estimation of the three-dimensional pose of the two hands to be poor. In the embodiments of the present disclosure, the features of each key point are fused with the features of the associated key points, the influence of the key points with strong correlation is increased, the influence of the key points with weak correlation is weakened, the mutual interference between the key points of the two hands can be reduced, and thus the three-dimensional feature information extracted from the feature image in the previous step can be corrected, and the accuracy of the estimation of the three-dimensional pose of the two hands can be greatly improved.

[0127] In step S207, the target feature information of each key point is subjected to pose estimation to obtain three-dimensional position information of each key point; and the three-dimensional position information of each key point is used to determine the three-dimensional pose of the two hands in the two-hand image.

[0128] In the embodiments of the present disclosure, the server can use the pose estimation module, which can optionally include a multi-layer perception (MLP), to implement the pose estimation of the target feature information of each key point to obtain the three-dimensional position information of each key point; wherein the three-dimensional position information of each key point includes three-dimensional coordinates based on a camera coordinate system; subsequently, the server can draw a three-dimensional model of the two hands corresponding to the pose based on the three-dimensional position information of each key point to determine the three-dimensional pose of the two hands in the two-hand image.

[0129] In some possible embodiments, the method for estimating the three-dimensional pose of the two hands according to the embodiments of the present disclosure can implement end-to-end prediction by training a three-dimensional pose estimation model of the two hands, that is, a three-dimensional position information of each key point can be output by inputting a two-hand image. Therefore, the method according to the embodiments of the present disclosure further includes a training step of the three-dimensional pose estimation model of the two hands, and specifically, as shown in Figure 13 The training method of the three-dimensional pose estimation model of the two hands includes the following steps:

[0130] In step S1301, a training two-hand image is obtained.

[0131] The training two-hand image carries annotation information; the annotation information can be obtained by manual or automatic labeling; the annotation information includes actual three-dimensional position information of each key point in a set of key points of the two hands and actual two-dimensional position information of each key point. It can be understood that the more the number of training two-hand images is, the higher the accuracy of the finally trained model is.

[0132] In step S1303, the training two-hand image is subjected to two-hand key point detection by a preset machine learning model to obtain predicted two-dimensional position information of each key point; based on the predicted two-dimensional position information of each key point, predicted three-dimensional feature information of each key point is determined from a feature image set of the training two-hand image; through an attention mechanism, the predicted three-dimensional feature information of each key point is subjected to associated key point feature fusion to obtain predicted target feature information of each key point; and the predicted target feature information of each key point is subjected to pose estimation to obtain predicted three-dimensional position information of each key point.

[0133] The preset machine learning model can include a key point two-dimensional position detection module, an attention module and a pose estimation module.

[0134] As shown in FIG. 13B, corresponding to each step above, when the preset machine learning model is built, the key point two-dimensional position detection module, the attention module and the pose estimation module are designed to realize the functions corresponding to each step respectively. Figure 14

[0135] Specifically, during training, the server inputs the training two-hand image into the preset machine learning model. First, the training two-hand image is subjected to multi-scale feature extraction by the key point two-dimensional position detection module. The key point two-dimensional position detection module can include a feature extraction sub-module and a prediction sub-module. The feature extraction sub-module extracts multi-scale features from the training two-hand image, and the prediction sub-module determines predicted two-dimensional position information of each key point. Then, based on the predicted two-dimensional position information of each key point, the server determines predicted three-dimensional feature information of each key point from a feature image set obtained after the training two-hand image is subjected to multi-scale feature extraction in the previous step. Then, the server subjects the predicted three-dimensional feature information of each key point to associated key point feature fusion by the attention module to obtain predicted target feature information of each key point. Finally, the server subjects the predicted target feature information of each key point to pose estimation by the pose estimation module to obtain predicted three-dimensional position information of each key point.

[0136] In step S1305, the preset machine learning model is trained based on a first loss value between the predicted three-dimensional position information of each key point and actual three-dimensional position information of the key point, and a second loss value between the predicted two-dimensional position information of each key point and actual two-dimensional position information of each key point, until a preset end condition is met, to obtain a two-hand three-dimensional pose estimation model.

[0137] ​In this step, after obtaining the predicted three-dimensional position information of each key point, the server calculates a first loss value between the predicted three-dimensional position information of each key point and the actual three-dimensional position information of the key point, and a second loss value between the predicted two-dimensional position information of each key point and the actual two-dimensional position information of each key point, trains the preset machine learning model based on the first loss value and the second loss value. Optionally, the first loss value and the second loss value are weighted and summed to obtain a total loss value, and the preset machine learning model is trained using the total loss value. In the case of meeting the preset end condition, for example, the preset end condition can include that the total loss value is less than or equal to a preset value, or the number of iterations reaches a preset number; for example, if the current total loss value is still greater than the preset value, the model parameters are updated using the loss value, and the steps S1301-S1305 are repeatedly executed until the total loss value is less than or equal to the preset value, and a trained two-hand three-dimensional pose estimation model is obtained.

[0138] Therefore, in actual application, the two-hand three-dimensional pose estimation model can be used to realize end-to-end prediction. Only the two-hand image needs to be input, and the three-dimensional position information of each key point of the two hands can be obtained, which is easy to use and can play a key role in different application scenarios.

[0139] In summary, the two-hand three-dimensional pose estimation method of the embodiments of the present disclosure can directly predict the three-dimensional poses of the two hands as a whole, which is more concise and convenient than the existing two-hand prediction scheme based on single-hand three-dimensional pose estimation, and no longer needs to use a hand detection module to locate and crop the two hands. At the same time, by using the attention mechanism, the three-dimensional feature information of each key point is associated and fused with the key point feature, which can enhance the influence of key points with strong correlation and weaken the influence of key points with weak correlation, thereby reducing the mutual interference of the two hands and enabling the two hands to be more accurately predicted compared with the existing method. When facing a large interactive area such as a two-hand fist gesture, the effect is obviously improved.

[0140] Figure 15 is a block diagram of a two-hand three-dimensional pose estimation device according to an example embodiment. Referring to Figure 15 The device includes a detection module 1501, a determination module 1502, a fusion module 1503, and an estimation module 1504.

[0141] The detection module 1501 is configured to perform two-hand key point detection on the obtained two-hand image to obtain two-dimensional position information of each key point in the two-hand key point set.

[0142] The determination module 1502 is configured to determine three-dimensional feature information of each key point from the feature image set of the two-hand image based on the two-dimensional position information of each key point.

[0143] The fusion module 1503 is configured to perform associated key point feature fusion on the three-dimensional feature information of each key point by using an attention mechanism to obtain target feature information of each key point.

[0144] The estimation module 1504 is configured to perform pose estimation on the target feature information of each key point to obtain three-dimensional position information of each key point; and the three-dimensional position information of each key point is used to determine a three-dimensional pose of the two hands in the two-hand image.

[0145] In some possible embodiments, the detection module 1501 is further configured to perform multi-scale feature extraction on the two-hand image to obtain a feature image set; the feature image set includes a plurality of feature images of different scales; a target feature image is determined from the plurality of feature images of different scales; the scale of the target feature image is the same as the scale of the two-hand image; and two-hand key point detection is performed on the target feature image to obtain two-dimensional position information of each key point.

[0146] In some possible embodiments, the detection module 1501 is further configured to perform key point heat analysis on the target feature image to obtain a heat map corresponding to each key point; the scale of the heat map is the same as the scale of the target feature image; the confidence of each key point at each pixel position in the target feature image is determined according to the heat map corresponding to each key point; for each key point, the pixel position with the highest confidence in the target feature image is taken as the pixel position of each key point; and the two-dimensional position information of each key point is obtained according to the pixel position of each key point.

[0147] In some possible embodiments, the determination module 1502 is further configured to perform, in each feature image in the plurality of feature images, determination of a key pixel position corresponding to the two-dimensional position information of each key point; and feature information at the key pixel position corresponding to each key point in each feature image is fused to obtain three-dimensional feature information of each key point.

[0148] In some possible embodiments, the fusion module 1503 is further configured to input the three-dimensional feature information of each key point in the two-hand key point set into an attention module to output an attention value between any two key points in the two-hand key point set; the attention value between any two key points represents the correlation between any two key points; and the three-dimensional feature information of each key point is fused by using the attention value between any two key points to obtain target feature information of each key point.

[0149] In some possible embodiments, the apparatus further includes an acquisition module configured to perform acquiring an initial image; performing hand detection on the initial image to obtain position information of a hand region in the initial image; and performing cropping on the initial image according to the position information of the hand region to obtain a hand image.

[0150] In some possible embodiments, the method further includes a training module configured to perform acquiring a training hand image; the training hand image carrying labeled information; the labeled information including actual three-dimensional position information of each key point in a hand key point set and actual two-dimensional position information of each key point; performing hand key point detection on the training hand image through a preset machine learning model to obtain predicted two-dimensional position information of each key point; determining predicted three-dimensional feature information of each key point from a feature image set of the training hand image based on the predicted two-dimensional position information of each key point; performing associated key point feature fusion on the predicted three-dimensional feature information of each key point through an attention mechanism to obtain predicted target feature information of each key point; performing pose estimation on the predicted target feature information of each key point to obtain predicted three-dimensional position information of each key point; training the preset machine learning model based on a first loss value between the predicted three-dimensional position information of each key point and actual three-dimensional position information of the key point, and a second loss value between the predicted two-dimensional position information of each key point and actual two-dimensional position information of each key point, until a preset end condition is met, to obtain a hand three-dimensional pose estimation model; and the hand three-dimensional pose estimation model being used for performing hand key point detection on an acquired hand image until three-dimensional position information of each key point is obtained.

[0151] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described here in detail.

[0152] Figure 16 is a block diagram of an electronic device for hand three-dimensional pose estimation according to an example embodiment. The electronic device can be a terminal, and its internal structure diagram can be as shown in Figure 16As shown in the figure. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the electronic device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a two-hand three-dimensional pose estimation method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0153] Those skilled in the art can understand that, Figure 16 The skilled in the art can understand that,

[0154] In an example embodiment, an electronic device is also provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the two-hand three-dimensional pose estimation method in the embodiments of the present disclosure.

[0155] In an example embodiment, a computer readable storage medium is also provided, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the two-hand three-dimensional pose estimation method in the embodiments of the present disclosure.

[0156] In an example embodiment, a computer program product containing instructions is also provided, the computer program product includes a computer program, the computer program is stored in a readable storage medium, at least one processor of the computer device reads and executes the computer program from the readable storage medium, so that the computer device executes the two-hand three-dimensional pose estimation method in the embodiments of the present disclosure.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0158] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure that are deemed to fall within the general principles of the disclosure and include commonly known or customary practice in the art. The specification and examples are to be considered exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0159] It should be understood that the present disclosure is not limited to the precise structures as herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A method of two-hand three-dimensional pose estimation, the method comprising: The method comprises the following steps: performing double-hand key point detection on the obtained double-hand image to obtain two-dimensional position information of each key point in a double-hand key point set; determining three-dimensional feature information of each key point from the feature image set of the double-hand image based on the two-dimensional position information of each key point; inputting the three-dimensional feature information of each key point into an attention module to output an attention value between any two key points in the double-hand key point set; the attention value between any two key points represents the correlation between the two key points; performing associated key point feature fusion on the three-dimensional feature information of each key point based on the attention value between any two key points to obtain target feature information of each key point; performing pose estimation on the target feature information of each key point to obtain three-dimensional position information of each key point; the three-dimensional position information of each key point is used to determine the three-dimensional pose of the double hands in the double-hand image.

2. The dual-hand three-dimensional pose estimation method of claim 1, wherein, The method comprises the following steps: performing multi-scale feature extraction on the double-hand image to obtain the feature image set; the feature image set comprises a plurality of feature images of different scales; determining a target feature image from the plurality of feature images of different scales; the scale of the target feature image is the same as that of the double-hand image; performing double-hand key point detection on the target feature image to obtain two-dimensional position information of each key point.

3. The dual-hand three-dimensional pose estimation method of claim 2, wherein, The method comprises the following steps: performing key point heat analysis on the target feature image to obtain a heat map corresponding to each key point; the scale of the heat map is the same as that of the target feature image; determining the confidence of each key point at each pixel position in the target feature image according to the heat map corresponding to each key point; for each key point, the pixel position with the highest confidence in the target feature image is taken as the pixel position of the key point; obtaining two-dimensional position information of each key point according to the pixel position of each key point.

4. The dual-hand three-dimensional pose estimation method of claim 2, wherein, The method comprises the following steps: in each feature image in the plurality of feature images, determining a key pixel position corresponding to the two-dimensional position information of each key point; fusing feature information at the key pixel position corresponding to each key point in each feature image to obtain three-dimensional feature information of each key point.

5. The dual-hand three-dimensional pose estimation method of claim 1, wherein, The method further comprises the following steps: obtaining an initial image; performing double-hand detection on the initial image to obtain position information of a double-hand region in the initial image; cropping the initial image according to the position information of the double-hand region to obtain the double-hand image.

6. The dual-hand three-dimensional pose estimation method of claim 1, wherein, The method further comprises the following steps: obtain a training double-hand image; the training double-hand image carries annotation information; the annotation information includes actual three-dimensional position information of each key point in a double-hand key point set and actual two-dimensional position information of the each key point; perform double-hand key point detection on the training double-hand image through a preset machine learning model to obtain predicted two-dimensional position information of the each key point; determine predicted three-dimensional feature information of the each key point from a feature image set of the training double-hand image based on the predicted two-dimensional position information of the each key point; perform associated key point feature fusion on the predicted three-dimensional feature information of the each key point through an attention mechanism to obtain predicted target feature information of the each key point; and perform pose estimation on the predicted target feature information of the each key point to obtain predicted three-dimensional position information of the each key point; train the preset machine learning model based on a first loss value between the predicted three-dimensional position information of the each key point and the actual three-dimensional position information of the key point and a second loss value between the predicted two-dimensional position information of the each key point and the actual two-dimensional position information of the each key point until a preset end condition is met, to obtain a double-hand three-dimensional pose estimation model; the double-hand three-dimensional pose estimation model is configured to perform double-hand key point detection on an obtained double-hand image until three-dimensional position information of each key point in a double-hand key point set is obtained.

7. A dual-hand three-dimensional pose estimation apparatus, characterized by comprising: comprise: a detection module configured to perform double-hand key point detection on an obtained double-hand image to obtain two-dimensional position information of each key point in a double-hand key point set; a determination module configured to determine three-dimensional feature information of the each key point from a feature image set of the double-hand image based on the two-dimensional position information of the each key point; a fusion module configured to input the three-dimensional feature information of the each key point into an attention module and output an attention value between any two key points in the double-hand key point set; the attention value between the any two key points represents a correlation between the any two key points; perform associated key point feature fusion on the three-dimensional feature information of the each key point based on the attention value between the any two key points to obtain target feature information of the each key point; an estimation module configured to perform pose estimation on the target feature information of the each key point to obtain three-dimensional position information of the each key point.

8. An electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the double-hand three-dimensional pose estimation method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can perform the double-hand three-dimensional pose estimation method of any one of claims 1-6.

Images