Gesture data labeling optimization method and electronic equipment

Abstract

The invention provides an annotation optimization method of gesture data and electronic equipment, which are used for improving the annotation quality of the gesture data. Comprising the steps that camera coordinates of all hand joint points in all hand images in any gesture sample are utilized to obtain all target hand images of all hand joint points; for any hand joint point, according to the image position coordinates of the hand joint point, obtaining the optimized world position coordinates of the hand joint point and the re-projection error of the hand joint point; updating an initial world coordinate of each hand joint point in the plurality of gesture samples by using the optimized world position coordinate of each hand joint point, and training a gesture estimation model by using a target gesture sample obtained by screening the plurality of gesture samples by using a re-projection error of each hand joint point and an initial training data set; and returning to use the camera coordinates of the hand articulation points in the hand images in any gesture sample until the training frequency is greater than the specified frequency, and determining the plurality of gesture samples as the optimized labeled gesture data.

Description

Background Art

[0002] Gesture interaction technology is one of the core interaction technologies in fields including virtual reality (VR), augmented reality (AR) and mixed reality (MR). At present, gesture interaction is mainly implemented through deep learning algorithms. As a data-driven algorithm, deep learning is highly dependent on rich and high-quality data, which makes how to produce rich and high-quality gesture data the top priority for implementing gesture interaction technology.

[0003] In the process of producing gesture data, gesture data is mainly divided into synthetic data and real data. The method of producing synthetic data mainly uses motion capture data to drive virtual models and cooperates with different maps to generate rich and high-quality data. However, in practical applications, synthetic data has domain differences. Deep learning models trained only on synthetic data generally have poor generalization in actual scenes. Real data can avoid domain differences. However, it is very difficult to produce real data that is both rich and high-quality. On the one hand, public data sets such as CMU Panoptic and InterHand2.6M use multi-view settings to ensure high-quality annotation. However, it cannot guarantee sample diversity in terms of background and other aspects. On the other hand, portable acquisition equipment using 1 depth camera and a small number of industrial cameras can ensure sample diversity, but it is difficult to ensure the quality of annotation based on optimized automatic / semi-automatic annotation methods. Therefore, the annotation quality of gesture data is low. Summary of the invention

[0004] The present application provides a gesture data annotation optimization method and electronic device for improving the annotation quality of gesture data.

[0005] In a first aspect, an embodiment of the present application provides a method for optimizing the annotation of gesture data, the method comprising:

[0006] For any gesture sample in the gesture data set to be optimized, using the camera position coordinates of each hand joint point in each hand image in the gesture sample, obtain each target hand image corresponding to the each hand joint point, wherein the camera position coordinates of each hand joint point in any hand image are obtained based on the initial world position coordinates of each hand joint point in the gesture sample; and,

[0007] For any hand joint point, each target hand image corresponding to the hand joint point is input into the gesture estimation model to perform gesture estimation, and the image position coordinates of the hand joint point in each target hand image are obtained; and,

[0008] According to the image position coordinates of the hand joint points in the target hand images, obtaining the optimized world position coordinates of the hand joint points in the gesture samples and the reprojection errors of the hand joint points in the gesture samples corresponding to the hand images; and

[0009] The plurality of gesture samples are screened based on the reprojection errors of the hand joint points corresponding to the hand images in the plurality of gesture samples to obtain a target gesture sample;

[0010] The initial world position coordinates of each hand joint point in the multiple gesture samples are updated using the optimized world position coordinates of each hand joint point in the multiple gesture samples, and after training the posture estimation model using the target gesture samples and the initial gesture training data set, return to the step of obtaining each target hand image corresponding to each hand joint point using the camera position coordinates of each hand joint point in each hand image in the gesture samples, until the number of training times for the posture estimation model is greater than the specified number, then the multiple gesture samples in the gesture data set to be optimized are determined as optimized labeled gesture data.

[0011] A second aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0012] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0013] For any gesture sample in the gesture data set to be optimized, using the camera position coordinates of each hand joint point in each hand image in the gesture sample, obtain each target hand image corresponding to the each hand joint point, wherein the camera position coordinates of each hand joint point in any hand image are obtained based on the initial world position coordinates of each hand joint point in the gesture sample; and,

[0014] For any hand joint point, each target hand image corresponding to the hand joint point is input into the gesture estimation model to perform gesture estimation, and the image position coordinates of the hand joint point in each target hand image are obtained; and,

[0015] According to the image position coordinates of the hand joint points in the target hand images, obtaining the optimized world position coordinates of the hand joint points in the gesture samples and the reprojection errors of the hand joint points in the gesture samples corresponding to the hand images; and

[0016] The plurality of gesture samples are screened based on the reprojection errors of the hand joint points corresponding to the hand images in the plurality of gesture samples to obtain a target gesture sample;

[0017] The initial world position coordinates of each hand joint point in the multiple gesture samples are updated using the optimized world position coordinates of each hand joint point in the multiple gesture samples, and after training the posture estimation model using the target gesture samples and the initial gesture training data set, return to the step of obtaining each target hand image corresponding to each hand joint point using the camera position coordinates of each hand joint point in each hand image in the gesture samples, until the number of training times for the posture estimation model is greater than the specified number, then the multiple gesture samples in the gesture data set to be optimized are determined as optimized labeled gesture data.

[0018] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.

[0019] In the above-mentioned embodiment of the present application, each target hand image corresponding to each hand joint point is obtained by using the camera position coordinates of each hand joint point in each hand image in any gesture sample in the gesture data set. Then, for any hand joint point, each target hand image corresponding to the hand joint point is input into the gesture estimation model for gesture posture estimation to obtain the image position coordinates of the hand joint point in each target hand image, and then, based on the image position coordinates of the hand joint point in each target hand image, each hand joint point obtained is respectively selected from the multiple gesture samples according to the reprojection errors corresponding to each hand image to obtain the target gesture sample. Finally, the initial world position coordinates of each hand joint point in the multiple gesture samples are updated using the optimized world position coordinates of each hand joint point in the multiple gesture samples, and after training the posture estimation model using the target gesture samples and the initial gesture training data set, the step of using the camera position coordinates of each hand joint point in each hand image in the gesture samples is returned to until the number of training times of the posture estimation model is greater than the specified number, and the multiple gesture samples in the gesture data set to be optimized are determined as optimized labeled gesture data, completing the annotation optimization of the gesture data, and improving the annotation quality of the gesture data while ensuring the diversity of the samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 One of the flowcharts of gesture data annotation optimization provided in the embodiment of the present application is exemplarily shown;

[0022] Figure 2 The flowchart of determining each target hand image corresponding to each hand joint point provided in the embodiment of the present application is exemplarily shown;

[0023] Figure 3 The schematic diagram of the process of determining hand voxels provided in the embodiment of the present application is exemplarily shown;

[0024] Figure 4 One of the schematic diagrams of various hand joint points and corresponding edges provided in an embodiment of the present application is exemplarily shown;

[0025] Figure 5 The second schematic diagram exemplarily shows various hand joint points and corresponding edges provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a hand voxel provided by an embodiment of the present application is exemplarily shown;

[0027] Figure 7 The schematic diagram of the process of determining the reprojection error provided by the embodiment of the present application is exemplarily shown;

[0028] Figure 8 The following is a schematic diagram of a process for determining a target gesture sample provided by an embodiment of the present application;

[0029] Fig. 9 The second flowchart of the optimization of gesture data annotation provided by the embodiment of the present application is exemplarily shown;

[0030] Fig.10 The structural diagram of the device for optimizing the annotation of gesture data provided by the embodiment of the present application is exemplarily shown;

[0031] Fig.11 The hardware structure diagram of the electronic device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0032] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0033] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0034] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0035] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0036] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0037] The idea of ​​the embodiments of the present application is summarized below.

[0038] Based on the problem of low annotation quality of gesture data in the prior art, an embodiment of the present application provides a method for optimizing annotation of gesture data, by using the camera position coordinates of each hand joint point in each hand image in any gesture sample in the gesture data set, to obtain each target hand image corresponding to the hand joint point. Then, for any hand joint point, each target hand image corresponding to the hand joint point is input into the posture estimation model for gesture posture estimation, to obtain the image position coordinates of the hand joint point in each target hand image, and then, based on the image position coordinates of the hand joint point in each target hand image, the reprojection errors corresponding to each hand image in the multiple gesture samples are respectively obtained to screen the multiple gesture samples, to obtain the target gesture sample. Finally, the initial world position coordinates of each hand joint point in the multiple gesture samples are updated using the optimized world position coordinates of each hand joint point in the multiple gesture samples, and after training the posture estimation model using the target gesture samples and the initial gesture training data set, the step of using the camera position coordinates of each hand joint point in each hand image in the gesture samples is returned to until the number of training times of the posture estimation model is greater than the specified number, and the multiple gesture samples in the gesture data set to be optimized are determined as optimized labeled gesture data, completing the annotation optimization of the gesture data, and improving the annotation quality of the gesture data while ensuring the diversity of the samples.

[0039] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0040] like Figure 1 FIG. 1 is a flow chart of a method for optimizing the annotation of gesture data, which may include the following steps:

[0041] Step 101: for any one gesture sample in the gesture data set to be optimized, using the camera position coordinates of each hand joint point in each hand image in the gesture sample, obtain each target hand image corresponding to the each hand joint point, wherein the camera position coordinates of each hand joint point in any one hand image are obtained based on the initial world position coordinates of each hand joint point in the gesture sample;

[0042] In the embodiment of the present application, the gesture data set to be optimized includes multiple gesture samples, each gesture sample includes a small number of hand images, and each hand image in the same gesture sample is taken at different camera angles, and any gesture sample corresponds to a set of pre-marked initial world position coordinates of each hand joint point. And each hand image in any gesture sample corresponds to a rotation and translation matrix of a camera, and for any hand image, the initial world position coordinates of each hand joint point are transformed based on the rotation and translation matrix to obtain the camera position coordinates corresponding to each hand joint point in the hand image.

[0043] In order to ensure the diversity of gesture data samples, the placement positions of the cameras corresponding to each hand image in the few hand image settings are quite different. In this case, the proportion of self-occluded images in each hand image in any gesture sample will increase significantly. Therefore, in the embodiments of the present application, it is necessary to perform self-occlusion identification on each hand image, that is, the target hand image corresponding to any hand joint point is a hand image that does not occlude the hand joint point in the image. Below, the method for determining each target hand image corresponding to each hand joint point is described in detail:

[0044] like Figure 2 As shown, it is a schematic diagram of a process of determining each target hand image corresponding to each hand joint point, comprising the following steps:

[0045] Step 201: for any hand image in the gesture sample, based on the camera position coordinates of each hand joint point in the hand image, construct a voxel map corresponding to the hand image;

[0046] In one embodiment, a voxel map corresponding to the hand image is constructed in the following manner: the lengths of each side in the voxel map are obtained according to the camera position coordinates of each hand joint point in the hand image, and a voxel map corresponding to the hand image is constructed based on the lengths of each side with the optical center of the camera corresponding to the hand image as the origin, wherein the voxel map includes a plurality of voxel grids.

[0047] In one embodiment, the length of each side in the voxel map is determined in the following manner: the length of the voxel map is obtained according to the horizontal coordinate of the camera position with the largest value among the camera position coordinates of the hand joint points; the width of the voxel map is obtained according to the vertical coordinate of the camera position with the largest value among the camera position coordinates of the hand joint points; the height v of the voxel map is obtained according to the vertical coordinate of the camera position with the largest value among the camera position coordinates of the hand joint points. The length of each side in the voxel map is obtained according to the length, width and height of the voxel map.

[0048] In the embodiment of the present application, the horizontal coordinate of the camera position with the largest value among the camera position coordinates of the hand joint points is enlarged by a specified multiple to obtain the length of the voxel map; and the vertical coordinate of the camera position with the largest value among the camera position coordinates of the hand joint points is enlarged by the specified multiple to obtain the width of the voxel map; and the vertical coordinate of the camera position with the largest value among the camera position coordinates of the hand joint points is enlarged by a specified multiple to obtain the height of the voxel map.

[0049] Among them, the specified multiple in the embodiment of the present application is 2, but the embodiment of the present application does not limit the specified multiple, and the specified multiple in the embodiment of the present application can be set according to the actual situation. And the shape of the voxel map in the embodiment of the present application is a rectangle, so the length, width and height of the voxel map are the length, width and height of the rectangle. Therefore, the length of each side in the voxel map can be directly obtained according to the length, width and height in the voxel map.

[0050] Step 202: Obtain hand voxels corresponding to the hand image using the camera position coordinates of each hand joint point in the hand image and the voxel map;

[0051] like Figure 3 As shown, it is a schematic diagram of the process of determining the hand voxel, which specifically includes the following steps:

[0052] Step 301: obtaining first target voxel grids of each hand joint point in the voxel map according to the camera position coordinates of each hand joint point in the hand image, and setting the identifiers of each first target voxel grid as first target identifiers, wherein the first target identifier of any first target voxel grid is determined based on the identifier of the hand joint point corresponding to the first target voxel grid;

[0053] In the embodiment of the present application, based on the length of each side of the voxel map and the number of voxel grids in the voxel map, the position coordinates corresponding to each voxel grid can be determined. The number of voxel grids in the embodiment of the present application can be set according to actual conditions, but the embodiment of the present application does not limit the number of each voxel grid in the voxel map. And the method for determining the position coordinates of each voxel grid in the voxel map belongs to the prior art, and the embodiment of the present application will not be repeated here.

[0054] In one embodiment, the first target voxel grid of any hand joint point in the voxel map is determined in the following manner: for any hand joint point, the horizontal coordinate of the camera position of the hand joint point is subtracted from the horizontal coordinate of the smallest value among the horizontal position coordinates of each voxel grid in the voxel map to obtain a horizontal position difference, and the horizontal position difference is divided by a first number to obtain a target horizontal position coordinate, wherein the first number is the number of each voxel grid on the horizontal coordinate axis in the voxel map; and the vertical coordinate of the camera position of the hand joint point is subtracted from the horizontal coordinate of each voxel grid in the voxel map to obtain a target horizontal position coordinate. The vertical position difference is obtained by subtracting the vertical coordinate with the smallest value from the vertical coordinate of the vertical position coordinates of each voxel in the voxel map, and the vertical position difference is obtained by dividing the vertical position difference by a second number to obtain the target vertical position coordinate, wherein the second number is the number of each voxel grid on the vertical coordinate axis in the voxel map; and the vertical coordinate of the camera position of the hand joint point is subtracted from the vertical coordinate with the smallest value from the vertical position coordinates of each voxel grid in the voxel map to obtain the vertical position difference, and the vertical position difference is divided by a third number to obtain the target vertical position coordinate, wherein the third number is the number of each voxel grid on the vertical coordinate axis in the voxel map.

[0055] It should be noted that the first target identifier of the first target voxel of any hand joint point in the embodiment of the present application is set to be the same as the index of the hand joint point. For example, if the index of hand joint point 1 is thumb, the first target identifier of the first target voxel of hand joint point 1 is set to thumb. However, the specific setting method of the first target identifier can be set according to the actual situation, and the embodiment of the present application does not limit the setting method of the first target identifier.

[0056] Step 302: for any two adjacent hand joint points among the hand joint points, perform equidistant sampling on the edges corresponding to the any two adjacent hand joint points to obtain a plurality of first target key points on the edges, obtain second target voxel grids of the plurality of first target key points in the voxel map according to the camera position coordinates of the plurality of first target key points, and set the identifier of each second target voxel grid as the second target identifier;

[0057] For example, Figure 4 As shown in FIG. 1 , it is a schematic diagram of each hand joint point and the corresponding edge, and the edges corresponding to hand joint points 0 to 17 are used as an example to illustrate. Figure 4 It can be seen that the first target key points after equidistant sampling in the edges corresponding to hand joint points 0 to hand joint points 17 are: key point a, key point b, key point c, key point d and key point e.

[0058] It should be noted that the distance of equidistant sampling in the embodiment of the present application is set according to the actual situation, and the distance of equidistant sampling is not limited in the embodiment of the present application. And the method of determining the second target voxel of the first target key point in the embodiment of the present application is the same as the method of determining the first target voxel corresponding to each hand joint point described above, and the embodiment of the present application will not be repeated here. And the second target identifier in the embodiment of the present application is pre-set, which is different from the first target identifier in the embodiment of the present application.

[0059] Step 303: for any set of preset hand joint points among the hand joint points, determine a plurality of second target key points in a plane formed by the hand joint points in the hand joint point set, obtain third target voxel grids of the plurality of second target key points in the voxel map based on the camera position coordinates of the plurality of second target key points, and set the identifier of each third target voxel grid as the second target identifier;

[0060] Any hand joint point set in the embodiment of the present application is three hand joint points. The following is a method for determining multiple second target key points: connect the midpoints of each side of the plane formed by each hand joint point in the hand joint point set to obtain a target plane, perform equidistant sampling on each side of the target plane, and obtain multiple second target key points. Figure 5 As shown, taking the plane composed of hand joint point 0, hand joint point 1 and hand joint point 5 as an example, the midpoints of each edge in the plane are connected, that is, point m, point n and point p are connected to obtain target plane 1, and then equidistant sampling is performed on each edge of the target plane to obtain each second target key point.

[0061] It should be noted that the distances for equidistant sampling on each side of the target plane in the embodiment of the present application may be the same as or different from the distances for equidistant sampling on the sides corresponding to any two adjacent hand joint points as described above. The specific distances may be set according to the actual situation, and the implementation of the present application is not limited thereto. Furthermore, the union of the preset hand joint point sets in the embodiment of the present application does not include all hand joint points, but only includes the hand joint points corresponding to the palm, and the hand joint points included in each hand joint point set are preset. Figure 4For example, the hand joint points included in the union of the sets of hand joint points are: hand joint point 0, hand joint point 1, hand joint point 5, hand joint point 9, hand joint point 13, hand joint point 17, hand joint point 2. And the corresponding hand joint point sets are (hand joint point 0, hand joint point 1, hand joint point 5), (hand joint point 0, hand joint point 5, hand joint point 9), (hand joint point 0, hand joint point 9, hand joint point 13), (hand joint point 0, hand joint point 13, hand joint point 17), (hand joint point 1, hand joint point 2, hand joint point 5).

[0062] Step 304: Obtain hand voxels corresponding to the hand image according to the voxel grid identified as the first target identifier or the second target identifier.

[0063] In one embodiment, step 304 may be specifically implemented as follows: each voxel grid identified as the first target identifier or the second target identifier constitutes the hand voxel.

[0064] like Figure 6 As shown, it is a schematic diagram of obtaining hand voxels. It can be seen from the figure that the plane corresponding to each hand joint point is first obtained based on each hand joint point, and then the hand voxel is obtained based on the plane corresponding to each hand joint point.

[0065] Step 203: using a ray casting algorithm to identify the hand voxels, and obtaining the unobstructed hand joint points in the hand image;

[0066] In one embodiment, step 203 can be specifically implemented as follows: for any hand joint point, if the identifier of the target voxel through which the line between the camera optical center and the hand joint point passes determined by the ray casting algorithm meets a specified condition, then it is determined that the hand joint point is not obscured in the hand image, wherein the specified condition is that the identifier is the first target identifier corresponding to the hand joint point or there is no identifier.

[0067] Step 204: For any hand joint point, determine a hand image that does not block the hand joint point as a target hand image of the hand joint point.

[0068] For example, if the hand joint point a in the hand image 1 and the hand image 2 is not blocked, the target hand images corresponding to the hand joint point a are determined to be the hand image 1 and the hand image 2.

[0069] Step 102: for any hand joint point, input each target hand image corresponding to the hand joint point into a posture estimation model to perform hand gesture estimation, and obtain the image position coordinates of the hand joint point in each target hand image;

[0070] It should be noted that the posture estimation model in the embodiment of the present application can be set according to the specific actual situation, and the embodiment of the present application does not limit the posture estimation model.

[0071] Step 103: obtaining the optimized world position coordinates of the hand joint points in the gesture samples and the reprojection errors of the hand joint points in the gesture samples corresponding to the hand images according to the image position coordinates of the hand joint points in the target hand images;

[0072] Next, the method for determining the reprojection error in the embodiment of the present application is described. Figure 7 As shown, a flow chart of determining the reprojection error may include the following steps:

[0073] Step 701: Obtain the intermediate world position coordinates of the hand joint points according to the image position coordinates of the hand joint points in the target hand images; wherein the intermediate world position coordinates of any hand joint point can be obtained by formula (1):

[0074]

[0075] Among them, p i is the image position coordinate of the hand joint point in the i-th target hand image, P is the intermediate world position coordinate of the hand joint point, π i () is the preset projection equation corresponding to the i-th target hand image, π i (P) is the projection position coordinate of the hand joint point in the i-th target hand image, and n is the total number of target hand images.

[0076] Step 702: Based on the intermediate world position coordinates of the hand joint points, the hand joint points are reprojected in each hand image in the gesture sample to obtain the reprojected position coordinates of the hand joint points in each hand image;

[0077] In one embodiment, step 702 can be specifically implemented as follows: for any hand joint point, the intermediate world position coordinates of the hand joint point are respectively input into the pre-set projection equation corresponding to each hand image to obtain the reprojected position coordinates of the hand joint point in the hand image.

[0078] Step 703: using the reprojection position coordinates of the hand joint points in the hand images and the image position coordinates of the hand joint points in the hand images, obtain the reprojection errors of the hand joint points corresponding to the hand images; wherein the reprojection errors of the hand joint points corresponding to any hand image can be obtained by formula (2):

[0079]

[0080] Among them, d is the reprojection error corresponding to the hand joint point in the hand image, x1 is the horizontal coordinate of the reprojection position of the hand joint point in the hand image, x2 is the horizontal coordinate of the image position of the hand joint point, y1 is the vertical coordinate of the reprojection position of the hand joint point in the hand image, and y2 is the vertical coordinate of the image position of the hand joint point.

[0081] Step 704: Obtain the confidence of the intermediate world position coordinates of the hand joint points through the reprojection errors corresponding to the hand joint points and the hand images respectively;

[0082] In one embodiment, step 704 may be specifically implemented as: determining the number of each reprojection error greater than a specified threshold as the confidence level of the intermediate world position coordinates of the hand joint point.

[0083] It should be noted that the specified threshold in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the specified threshold.

[0084] Step 705: Determine whether the confidence level is greater than the target threshold. If so, execute step 706; if not, execute step 707v.

[0085] Step 706: Determine the intermediate world position coordinates of the hand joint points as optimized world position coordinates, wherein the target threshold is obtained based on the number of target hand images corresponding to the hand joint points;

[0086] Step 707: Determine the initial world position coordinates of the hand joint points as the optimized world position coordinates.

[0087] The target threshold in the embodiment of the present application is obtained by adding the number of target hand images corresponding to the hand joint points to a preset value. The preset value in the embodiment of the present application is 1, but the embodiment of the present application does not limit the preset value, and the preset value in the embodiment of the present application can be set according to actual conditions.

[0088] Step 104: Screening the plurality of gesture samples based on the reprojection errors of the hand joint points corresponding to the hand images in the plurality of gesture samples to obtain a target gesture sample;

[0089] like Figure 8 As shown, it is a flowchart of determining a target gesture sample, which includes the following steps:

[0090] Step 801: For any one of the plurality of gesture samples, a first ranking value of the gesture sample is obtained according to an inverse document probability of the gesture sample and the number of target hand joint points of the gesture sample, wherein the target hand joint point is a hand joint point whose confidence is greater than a specified threshold, and the inverse document probability of the gesture sample is determined based on the number of times the gesture sample is determined as a target gesture sample when the posture estimation model is previously trained.

[0091] In one embodiment, the inverse document probability of the gesture sample is determined by:

[0092] For any gesture sample, the inverse document probability of the gesture sample is obtained according to the current training times of the gesture estimation model and the times when the gesture sample was determined as the target gesture sample when the gesture estimation model was previously trained. The inverse document probability of any gesture sample can be determined by formula (3):

[0093] ω j =1+log(k / c)......(3);

[0094] Among them, ω j is the inverse document probability of gesture sample j, k is the current training times of the gesture estimation model, and c is the number of times the gesture sample is determined as a target gesture sample when the gesture estimation model is previously trained.

[0095] In one embodiment, the first ranking value of the gesture sample is obtained by multiplying the inverse document probability of the gesture sample and the number of target hand joint points of the gesture sample.

[0096] Step 802: Obtain a second ranking value of the gesture sample through the confidence corresponding to each target joint point in the gesture sample;

[0097] In one embodiment, step 802 may be specifically implemented as: determining the confidence corresponding to each target joint point in the gesture sample as the second ranking value of the gesture sample.

[0098] Step 803: obtaining a third ranking value of the gesture sample based on the reprojection error of each hand image of each target joint point in the gesture sample in the gesture sample;

[0099] In one embodiment, step 803 may be specifically implemented as follows: determining the reprojection error of the hand image of each target joint point in the gesture sample in the gesture sample as the third ranking value of the gesture sample.

[0100] Step 804: Sort the gesture samples according to the target sorting value of each gesture sample, and filter the multiple gesture samples based on the arrangement order of each gesture sample to obtain a target gesture sample, wherein the target sorting value is at least one of the first sorting value, the second sorting value, and the third sorting value.

[0101] In one embodiment, the gesture samples are sorted in the following manner:

[0102] For any two gesture samples among the multiple gesture samples, if the first ranking value of the first gesture sample among the two gesture samples is greater than the first ranking value of the second gesture sample, it is determined that the priority of the first gesture sample is higher than the priority of the second gesture sample, wherein the first gesture sample is the gesture sample with the smaller sample sequence number among the two gesture samples, and the second gesture sample is the gesture sample with the larger sample sequence number among the two gesture samples; if the first ranking value of the first gesture sample is less than the first ranking value of the second gesture sample, and the second ranking value of the first gesture sample is greater than the second ranking value of the second gesture sample, it is determined that the priority of the first gesture sample is higher than the priority of the second gesture sample; if the first ranking value of the first gesture sample is less than the first ranking value of the second gesture sample, and the second ranking value of the first gesture sample is less than the second ranking value of the second gesture sample, and the third ranking value of the first gesture sample is not greater than the third ranking value of the second gesture sample, it is determined that the priority of the first gesture sample is higher than the priority of the second gesture sample; otherwise, it is determined that the priority of the first gesture sample is lower than the priority of the second gesture sample; the multiple gesture samples are sorted according to the priorities between any two gesture samples among the multiple gesture samples.

[0103] In the embodiment of the present application, the plurality of gesture samples are sorted from high to low according to the priority of each gesture sample to obtain a plurality of sorted gesture samples, and a first specified number of gesture samples among the plurality of sorted gesture samples are determined as the target gesture samples.

[0104] It should be noted that the specified number in the embodiments of the present application can be set according to actual conditions, and the embodiments of the present application do not limit the specified number.

[0105] Step 105: using the optimized world position coordinates of each hand joint point in the multiple gesture samples to update the initial world position coordinates of each hand joint point in the multiple gesture samples, and using the target gesture samples and the initial gesture training data set to train the posture estimation model;

[0106] Step 106: Determine whether the number of training times for the posture estimation model is greater than a specified number of times, if so, execute step 107, if not, return to execute step 101;

[0107] Step 107: Determine the plurality of gesture samples in the gesture data set to be optimized as optimized labeled gesture data.

[0108] In the embodiment of the present application, the world position coordinates of each hand joint point after optimization in multiple gesture samples are determined as the initial world position coordinates of each hand joint point after update. And in the embodiment of the present application, the target gesture sample is added to the initial gesture training data to train the posture estimation model. The specified number of times summarized in the embodiment of the present application can be set according to the actual situation, and the embodiment of the present application does not limit the specified number of times.

[0109] In order to further connect the technical solutions in this application, Fig. 9 A detailed description may include the following steps:

[0110] Step 901: for any hand image in the gesture sample, based on the camera position coordinates of each hand joint point in the hand image, construct a voxel map corresponding to the hand image, wherein the camera position coordinates of each hand joint point in any hand image are obtained based on the initial world position coordinates of each hand joint point in the gesture sample;

[0111] Step 902: Obtain hand voxels corresponding to the hand image using the camera position coordinates of the hand joints in the hand image and the voxel map;

[0112] Step 903: using a ray casting algorithm to identify the hand voxels, and obtaining the unobstructed hand joint points in the hand image;

[0113] Step 904: for any hand joint point, determine the hand image that does not block the hand joint point as the target hand image of the hand joint point;

[0114] Step 905: for any hand joint point, input each target hand image corresponding to the hand joint point into the posture estimation model to perform hand gesture estimation, and obtain the image position coordinates of the hand joint point in each target hand image;

[0115] Step 906: Obtaining the intermediate world position coordinates of the hand joint points according to the image position coordinates of the hand joint points in the target hand images;

[0116] Step 907: Based on the intermediate world position coordinates of the hand joint points, the hand joint points are reprojected in each hand image in the gesture sample to obtain the reprojected position coordinates of the hand joint points in each hand image;

[0117] Step 908: using the reprojection position coordinates of the hand joint points in the hand images and the image position coordinates of the hand joint points in the hand images, respectively, to obtain the reprojection errors of the hand joint points corresponding to the hand images;

[0118] Step 909: Obtain the confidence of the intermediate world position coordinates of the hand joint points through the reprojection errors corresponding to the hand joint points and the hand images respectively;

[0119] Step 910: Determine whether the confidence is greater than a target threshold, if so, execute step 911, if not, execute step 912;

[0120] Step 911: determining the intermediate world position coordinates of the hand joint points as optimized world position coordinates, wherein the target threshold is obtained based on the number of target hand images corresponding to the hand joint points;

[0121] Step 912: Determine the initial world position coordinates of the hand joint point as the optimized world position coordinates v

[0122] Step 913: for any one of the multiple gesture samples, obtaining a first ranking value of the gesture sample according to an inverse document probability of the gesture sample and the number of target hand joint points of the gesture sample, wherein the target hand joint point is a hand joint point whose confidence is greater than a specified threshold, and the inverse document probability of the gesture sample is determined based on the number of times the gesture sample is determined as a target gesture sample when the posture estimation model is previously trained;

[0123] Step 914: Obtain a second ranking value of the gesture sample through the confidence corresponding to each target joint point in the gesture sample;

[0124] Step 915: obtaining a third ranking value of the gesture sample based on the reprojection error of each hand image of each target joint point in the gesture sample in the gesture sample;

[0125] Step 916: sorting the gesture samples according to the target sorting value of each gesture sample, and screening the multiple gesture samples based on the arrangement order of each gesture sample to obtain a target gesture sample, wherein the target sorting value is at least one of the first sorting value, the second sorting value, and the third sorting value;

[0126] Step 917: using the optimized world position coordinates of each hand joint point in the multiple gesture samples to update the initial world position coordinates of each hand joint point in the multiple gesture samples, and using the target gesture sample and the initial gesture training data set to train the posture estimation model;

[0127] Step 918: Determine whether the training times of the posture estimation model is greater than the specified times, if not, return to step 901, if yes, execute step 919;

[0128] Step 919: Determine the multiple gesture samples in the gesture data set to be optimized as optimized labeled gesture data.

[0129] Based on the same inventive concept, the gesture data annotation optimization method disclosed above can also be implemented by a gesture data annotation optimization device. The effect of the gesture data annotation optimization device is similar to that of the aforementioned method, which will not be described in detail here.

[0130] Fig.10 The figure is a schematic diagram of the structure of a gesture data annotation optimization device according to an embodiment of the present disclosure.

[0131] like Fig.10 As shown, the gesture data annotation optimization device 1000 of the present disclosure may include a target hand image determination module 1010, a gesture posture estimation module 1020, a reprojection error determination module 1030, a target gesture sample determination module 1040 and a annotation gesture data optimization module 1050.

[0132] The target hand image determination module 1010 is used to obtain target hand images corresponding to each hand joint point using the camera position coordinates of each hand joint point in each hand image in the gesture sample for any gesture sample in the gesture data set to be optimized, wherein the camera position coordinates of each hand joint point in any hand image are obtained based on the initial world position coordinates of each hand joint point in the gesture sample;

[0133] The hand gesture estimation module 1020 is used to input each target hand image corresponding to any hand joint point into the gesture estimation model to perform hand gesture estimation, and obtain the image position coordinates of the hand joint point in each target hand image;

[0134] A reprojection error determination module 1030, configured to obtain, according to the image position coordinates of the hand joint points in the target hand images, the optimized world position coordinates of the hand joint points in the gesture samples and the reprojection errors of the hand joint points in the gesture samples corresponding to the hand images;

[0135] A target gesture sample determination module 1040 is used to screen the multiple gesture samples based on the reprojection errors of the hand joint points corresponding to the hand images in the multiple gesture samples to obtain a target gesture sample;

[0136] The labeled gesture data optimization module 1050 is used to update the initial world position coordinates of each hand joint point in the multiple gesture samples using the optimized world position coordinates of each hand joint point in the multiple gesture samples, and after training the posture estimation model using the target gesture samples and the initial gesture training data set, return to the step of obtaining each target hand image corresponding to each hand joint point using the camera position coordinates of each hand joint point in each hand image in the gesture samples, until the number of training times for the posture estimation model is greater than the specified number, then the multiple gesture samples in the gesture data set to be optimized are determined as optimized labeled gesture data.

[0137] In one embodiment, the target hand image determination module 1010 is specifically configured to:

[0138] For any one hand image in the gesture sample, based on the camera position coordinates of each hand joint point in the hand image, construct a voxel map corresponding to the hand image; and,

[0139] Obtaining hand voxels corresponding to the hand image using the camera position coordinates of the hand joints in the hand image and the voxel map; and,

[0140] Using a ray casting algorithm to identify the hand voxels, and obtain the unobstructed hand joint points in the hand image;

[0141] For any hand joint point, a hand image that does not block the hand joint point is determined as a target hand image of the hand joint point.

[0142] In one embodiment, the target hand image determination module 1010 is specifically configured to:

[0143] According to the camera position coordinates of each hand joint point in the hand image, the length of each side in the voxel map is obtained, and with the camera optical center corresponding to the hand image as the origin, a voxel map corresponding to the hand image is constructed based on the length of each side, wherein the voxel map includes multiple voxel grids.

[0144] In one embodiment, the target hand image determination module 1010 is specifically configured to:

[0145] According to the camera position coordinates of the hand joints in the hand image, first target voxel grids of the hand joints in the voxel map are obtained, and the identifiers of the first target voxels are respectively set as first target identifiers, wherein the first target identifier of any first target voxel grid is determined based on the identifier of the hand joint corresponding to the first target voxel grid; and

[0146] For any two adjacent hand joint points among the hand joint points, perform equidistant sampling on the edges corresponding to the any two adjacent hand joint points to obtain a plurality of first target key points on the edges, obtain the second target voxel grids of the plurality of first target key points in the voxel map according to the camera position coordinates of the plurality of first target key points, and set the identifier of each second target voxel grid as the second target identifier; and

[0147] For any set of preset hand joint points among the hand joint points, a plurality of second target key points are determined in a plane formed by the hand joint points in the hand joint point set, and based on the camera position coordinates of the plurality of second target key points, third target voxel grids of the plurality of second target key points in the voxel map are obtained, and an identifier of each third target voxel grid is set as a second target identifier;

[0148] According to the voxel grid identified as the first target identifier or the second target identifier, a hand voxel corresponding to the hand image is obtained.

[0149] In one embodiment, the reprojection error determination module 1030 is specifically configured to:

[0150] Obtaining the intermediate world position coordinates of the hand joint points according to the image position coordinates of the hand joint points in the target hand images;

[0151] Based on the intermediate world position coordinates of the hand joint points, the hand joint points are reprojected in each hand image in the gesture sample to obtain the reprojected position coordinates of the hand joint points in each hand image;

[0152] Using the reprojection position coordinates of the hand joint points in the hand images and the image position coordinates of the hand joint points in the hand images, reprojection errors of the hand joint points corresponding to the hand images are obtained;

[0153] Obtaining the confidence of the intermediate world position coordinates of the hand joint points through the reprojection errors corresponding to the hand joint points and the hand images respectively;

[0154] If the confidence level is greater than a target threshold, determining the intermediate world position coordinates of the hand joint point as the optimized world position coordinates, wherein the target threshold is obtained based on the number of target hand images corresponding to the hand joint point;

[0155] If the confidence level is not greater than the target threshold, the initial world position coordinates of the hand joint point are determined as the optimized world position coordinates.

[0156] In one embodiment, the reprojection error determination module 1030 is specifically configured to:

[0157] The number of each reprojection error greater than a specified threshold is determined as the confidence level of the intermediate world position coordinates of the hand joint point.

[0158] In one embodiment, the target gesture sample determination module 1040 is specifically configured to:

[0159] For any one of the multiple gesture samples, a first ranking value of the gesture sample is obtained according to an inverse document probability of the gesture sample and the number of target hand joint points of the gesture sample, wherein the target hand joint point is a hand joint point whose confidence is greater than a specified threshold, and the inverse document probability of the gesture sample is determined based on the number of times the gesture sample is determined as a target gesture sample when a posture estimation model is previously trained; and,

[0160] Obtaining a second ranking value of the gesture sample by the confidence corresponding to each target joint point in the gesture sample; and,

[0161] Obtaining a third ranking value of the gesture sample based on a reprojection error of each hand image in the gesture sample at each target joint point in the gesture sample;

[0162] The gesture samples are sorted according to the target sorting value of each gesture sample, and the multiple gesture samples are screened based on the arrangement order of each gesture sample to obtain a target gesture sample, wherein the target sorting value is at least one of the first sorting value, the second sorting value and the third sorting value.

[0163] In one embodiment, the target gesture sample determination module 1040 performs the inverse document probability according to the gesture sample and the number of target hand joint points of the gesture sample to obtain the first ranking value of the gesture sample, which is specifically used for:

[0164] Multiplying the inverse document probability of the gesture sample by the number of target hand joint points of the gesture sample to obtain a first ranking value of the gesture sample;

[0165] The target gesture sample determination module 1040 performs the step of obtaining the second ranking value of the gesture sample by the confidence corresponding to each target joint point in the gesture sample, which is specifically used for:

[0166] Determining an average value of the confidences corresponding to the target joint points in the gesture sample as a second ranking value of the gesture sample;

[0167] The target gesture sample determination module 1040 performs the reprojection error of each hand image in the gesture sample based on each target joint point in the gesture sample to obtain a third ranking value of the gesture sample, which is specifically used for:

[0168] An average value of reprojection errors of each target joint point in the gesture sample in the gesture sample is determined as the third ranking value.

[0169] In one embodiment, the target gesture sample determination module 1040 is specifically configured to:

[0170] For any two gesture samples among the multiple gesture samples, if the first ranking value of a first gesture sample among the two gesture samples is greater than the first ranking value of a second gesture sample, it is determined that the priority of the first gesture sample is higher than the priority of the second gesture sample, wherein the first gesture sample is the gesture sample with a smaller sample sequence number among the two gesture samples, and the second gesture sample is the gesture sample with a larger sample sequence number among the two gesture samples;

[0171] If the first ranking value of the first gesture sample is less than the first ranking value of the second gesture sample, and the second ranking value of the first gesture sample is greater than the second ranking value of the second gesture sample, determining that the priority of the first gesture sample is higher than the priority of the second gesture sample;

[0172] If the first ranking value of the first gesture sample is less than the first ranking value of the second gesture sample, and the second ranking value of the first gesture sample is less than the second ranking value of the second gesture sample, and the third ranking value of the first gesture sample is not greater than the third ranking value of the second gesture sample, then determining that the priority of the first gesture sample is higher than the priority of the second gesture sample;

[0173] Otherwise, determining that the priority of the first gesture sample is lower than the priority of the second gesture sample;

[0174] The multiple gesture samples are sorted according to the priority between any two gesture samples among the multiple gesture samples.

[0175] After introducing a method and device for optimizing gesture data annotation according to an exemplary embodiment of the present invention, an electronic device according to another exemplary embodiment of the present invention is introduced next.

[0176] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0177] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for optimizing the annotation of gesture data according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 1 Steps 101-105 shown in FIG.

[0178] Refer to the following Fig.11 The electronic device 1100 according to this embodiment of the present invention is described. Fig.11 The electronic device 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0179] like Fig.11 As shown, the electronic device 1100 is in the form of a general electronic device. The components of the electronic device 1100 may include but are not limited to: the at least one processor 1101, the at least one computer storage medium 1102, and a bus 1103 connecting different system components (including the computer storage medium 1102 and the processor 1101).

[0180] Bus 1103 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0181] Computer storage media 1102 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1121 and / or cache storage media 1122 , and may further include read-only computer storage media (ROM) 1123 .

[0182] The computer storage medium 1102 may also include a program / utility 1125 having a set (at least one) of program modules 1124, such program modules 1124 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0183] The electronic device 1100 may also communicate with one or more external devices 1104 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1105. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1106. As shown, the network adapter 1106 communicates with other modules for the electronic device 1100 via a bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0184] In some possible implementations, various aspects of a method for optimizing the annotation of gesture data provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the method for optimizing the annotation of gesture data according to various exemplary embodiments of the present invention described above in this specification.

[0185] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access computer storage medium (RAM), a read-only computer storage medium (ROM), an erasable programmable read-only computer storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only computer storage medium (CD-ROM), an optical computer storage medium, a magnetic computer storage medium, or any suitable combination of the above.

[0186] The program product of gesture data annotation optimization of the embodiment of the present invention can adopt a portable compact disk read-only computer storage medium (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present invention is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0187] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0188] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0189] The program code for performing the operation of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages-such as Java, C++, etc., and also conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or completely on the remote electronic device or server. In the case of involving a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0190] It should be noted that although several modules of the device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0191] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0192] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk computer storage media, CD-ROM, optical computer storage media, etc.) that contain computer-usable program code.

[0193] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0194] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable computer storage medium produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0196] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for optimizing the annotation of gesture data, characterized in that: The method comprises: For any gesture sample in the gesture data set to be optimized, using the camera position coordinates of each hand joint point in each hand image in the gesture sample, obtain each target hand image corresponding to the each hand joint point, wherein the camera position coordinates of each hand joint point in any hand image are obtained based on the initial world position coordinates of each hand joint point in the gesture sample; and, For any hand joint point, each target hand image corresponding to the hand joint point is input into the gesture estimation model to perform gesture estimation, and the image position coordinates of the hand joint point in each target hand image are obtained; and, According to the image position coordinates of the hand joint points in the target hand images, obtaining the optimized world position coordinates of the hand joint points in the gesture samples and the reprojection errors of the hand joint points in the gesture samples corresponding to the hand images; and The plurality of gesture samples are screened based on the reprojection errors of the hand joint points corresponding to the hand images in the plurality of gesture samples to obtain a target gesture sample; The initial world position coordinates of each hand joint point in the multiple gesture samples are updated using the optimized world position coordinates of each hand joint point in the multiple gesture samples, and after training the posture estimation model using the target gesture samples and the initial gesture training data set, return to the step of obtaining each target hand image corresponding to each hand joint point using the camera position coordinates of each hand joint point in each hand image in the gesture samples, until the number of training times for the posture estimation model is greater than the specified number, then the multiple gesture samples in the gesture data set to be optimized are determined as optimized labeled gesture data.

2. The method according to claim 1, characterized in that The method of obtaining target hand images corresponding to the hand joint points by using the camera position coordinates of the hand joint points in the hand images in the gesture samples includes: For any one hand image in the gesture sample, based on the camera position coordinates of each hand joint point in the hand image, construct a voxel map corresponding to the hand image; and, Obtaining hand voxels corresponding to the hand image using the camera position coordinates of the hand joints in the hand image and the voxel map; and, Using a ray casting algorithm to identify the hand voxels, and obtain the unobstructed hand joint points in the hand image; For any hand joint point, a hand image that does not block the hand joint point is determined as a target hand image of the hand joint point.

3. The method according to claim 2, characterized in that The step of constructing a voxel map corresponding to the hand image based on the camera position coordinates of each hand joint point in the hand image comprises: According to the camera position coordinates of each hand joint point in the hand image, the length of each side in the voxel map is obtained, and with the camera optical center corresponding to the hand image as the origin, a voxel map corresponding to the hand image is constructed based on the length of each side, wherein the voxel map includes multiple voxel grids.

4. The method according to claim 2, characterized in that: The obtaining of hand voxels corresponding to the hand image by using the camera position coordinates of each hand joint point in the hand image and the voxel map includes: According to the camera position coordinates of the hand joints in the hand image, first target voxel grids of the hand joints in the voxel map are obtained, and the identifiers of the first target voxels are respectively set as first target identifiers, wherein the first target identifier of any first target voxel grid is determined based on the identifier of the hand joint corresponding to the first target voxel grid; and For any two adjacent hand joint points among the hand joint points, perform equidistant sampling on the edges corresponding to the any two adjacent hand joint points to obtain a plurality of first target key points on the edges, obtain the second target voxel grids of the plurality of first target key points in the voxel map according to the camera position coordinates of the plurality of first target key points, and set the identifier of each second target voxel grid as the second target identifier; and For any set of preset hand joint points among the hand joint points, a plurality of second target key points are determined in a plane formed by the hand joint points in the hand joint point set, and based on the camera position coordinates of the plurality of second target key points, third target voxel grids of the plurality of second target key points in the voxel map are obtained, and an identifier of each third target voxel grid is set as a second target identifier; According to the voxel grid identified as the first target identifier or the second target identifier, a hand voxel corresponding to the hand image is obtained.

5. The method according to claim 1, characterized in that The step of obtaining the optimized world position coordinates of the hand joint points in the gesture samples and the reprojection errors of the hand joint points in the gesture samples corresponding to the hand images according to the image position coordinates of the hand joint points in the target hand images includes: Obtaining the intermediate world position coordinates of the hand joint points according to the image position coordinates of the hand joint points in the target hand images; Based on the intermediate world position coordinates of the hand joint points, the hand joint points are reprojected in each hand image in the gesture sample to obtain the reprojected position coordinates of the hand joint points in each hand image; Using the reprojection position coordinates of the hand joint points in the hand images and the image position coordinates of the hand joint points in the hand images, reprojection errors of the hand joint points corresponding to the hand images are obtained; Obtaining the confidence of the intermediate world position coordinates of the hand joint points through the reprojection errors corresponding to the hand joint points and the hand images respectively; If the confidence level is greater than a target threshold, determining the intermediate world position coordinates of the hand joint point as the optimized world position coordinates, wherein the target threshold is obtained based on the number of target hand images corresponding to the hand joint point; If the confidence level is not greater than the target threshold, the initial world position coordinates of the hand joint point are determined as the optimized world position coordinates.

6. The method according to claim 5, characterized in that The obtaining the confidence of the intermediate world position coordinates of the hand joint points by reprojecting the hand joint points respectively corresponding to the hand images comprises: The number of each reprojection error greater than a specified threshold is determined as the confidence level of the intermediate world position coordinates of the hand joint point.

7. The method according to claim 5, characterized in that The plurality of gesture samples are screened based on the reprojection errors of the hand joint points corresponding to the hand images in the plurality of gesture samples to obtain a target gesture sample, including: For any one of the multiple gesture samples, a first ranking value of the gesture sample is obtained according to an inverse document probability of the gesture sample and the number of target hand joint points of the gesture sample, wherein the target hand joint point is a hand joint point whose confidence is greater than a specified threshold, and the inverse document probability of the gesture sample is determined based on the number of times the gesture sample is determined as a target gesture sample when a posture estimation model is previously trained; and, Obtaining a second ranking value of the gesture sample by the confidence corresponding to each target joint point in the gesture sample; and, Obtaining a third ranking value of the gesture sample based on a reprojection error of each hand image in the gesture sample at each target joint point in the gesture sample; The gesture samples are sorted according to the target sorting value of each gesture sample, and the multiple gesture samples are screened based on the arrangement order of each gesture sample to obtain a target gesture sample, wherein the target sorting value is at least one of the first sorting value, the second sorting value and the third sorting value.

8. The method according to claim 7, characterized in that The obtaining, according to the inverse document probability of the gesture sample and the number of target hand joint points of the gesture sample, a first ranking value of the gesture sample comprises: Multiplying the inverse document probability of the gesture sample by the number of target hand joint points of the gesture sample to obtain a first ranking value of the gesture sample; The obtaining a second ranking value of the gesture sample by the confidence corresponding to each target joint point in the gesture sample includes: Determining an average value of the confidences corresponding to the target joint points in the gesture sample as a second ranking value of the gesture sample; The obtaining of a third ranking value of the gesture sample based on the reprojection error of each hand image in the gesture sample at each target joint point in the gesture sample comprises: An average value of reprojection errors of each target joint point in the gesture sample in the gesture sample is determined as the third ranking value.

9. The method according to claim 7, characterized in that: The sorting of the gesture samples according to the target sorting value of each gesture sample includes: For any two gesture samples among the multiple gesture samples, if the first ranking value of a first gesture sample among the two gesture samples is greater than the first ranking value of a second gesture sample, it is determined that the priority of the first gesture sample is higher than the priority of the second gesture sample, wherein the first gesture sample is the gesture sample with a smaller sample sequence number among the two gesture samples, and the second gesture sample is the gesture sample with a larger sample sequence number among the two gesture samples; If the first ranking value of the first gesture sample is less than the first ranking value of the second gesture sample, and the second ranking value of the first gesture sample is greater than the second ranking value of the second gesture sample, determining that the priority of the first gesture sample is higher than the priority of the second gesture sample; If the first ranking value of the first gesture sample is less than the first ranking value of the second gesture sample, and the second ranking value of the first gesture sample is less than the second ranking value of the second gesture sample, and the third ranking value of the first gesture sample is not greater than the third ranking value of the second gesture sample, then determining that the priority of the first gesture sample is higher than the priority of the second gesture sample; Otherwise, determining that the priority of the first gesture sample is lower than the priority of the second gesture sample; The multiple gesture samples are sorted according to the priority between any two gesture samples among the multiple gesture samples.

10. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: For any gesture sample in the gesture data set to be optimized, using the camera position coordinates of each hand joint point in each hand image in the gesture sample, obtain each target hand image corresponding to the each hand joint point, wherein the camera position coordinates of each hand joint point in any hand image are obtained based on the initial world position coordinates of each hand joint point in the gesture sample; and, For any hand joint point, each target hand image corresponding to the hand joint point is input into the gesture estimation model to perform gesture estimation, and the image position coordinates of the hand joint point in each target hand image are obtained; and, According to the image position coordinates of the hand joint points in the target hand images, obtaining the optimized world position coordinates of the hand joint points in the gesture samples and the reprojection errors of the hand joint points in the gesture samples corresponding to the hand images; and The plurality of gesture samples are screened based on the reprojection errors of the hand joint points corresponding to the hand images in the plurality of gesture samples to obtain a target gesture sample; The initial world position coordinates of each hand joint point in the multiple gesture samples are updated using the optimized world position coordinates of each hand joint point in the multiple gesture samples, and after training the posture estimation model using the target gesture samples and the initial gesture training data set, return to the step of obtaining each target hand image corresponding to each hand joint point using the camera position coordinates of each hand joint point in each hand image in the gesture samples, until the number of training times for the posture estimation model is greater than the specified number, then the multiple gesture samples in the gesture data set to be optimized are determined as optimized labeled gesture data.

Images