Hand posture estimation method, device, equipment and computer storage medium

Abstract

The embodiments of the present application disclose a method, apparatus, device and computer storage medium for hand posture estimation, the method comprising: determining a classification logic diagram corresponding to each of a plurality of key points; wherein the plurality of key points represent the skeleton key nodes of the target hand, and the first key point is any one of the plurality of key points; based on a preset classification diagram and the classification logic diagram corresponding to the first key point, determining the coordinate information of the first key point; after determining the coordinate information of each of the plurality of key points, obtaining a posture estimation result of the target hand.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image recognition technology, and in particular to a hand posture estimation method, device, equipment and computer storage medium. Background Art

[0002] The ability to accurately and efficiently reconstruct the motion of human hands from images has exciting new applications in areas such as immersive virtual and augmented reality, robotic control, and sign language recognition. Reconstructing the motion of the hand has also made great progress in recent years, especially with the advent of consumer depth cameras. However, it remains a challenging task due to unconstrained global and local pose variations, frequent occlusions, local self-similarity, and high definition.

[0003] Current hand pose estimation schemes either use a fully connected layer to regress the coordinates of the key points of the hand, or use a classification-based method to predict the spatial position of the key points. Among them, the regression-based method calculates the hand pose in a global way, that is, using all the information of the key point features to predict each key point; on the contrary, the classification-based method tends to be more local, that is, gradually obtaining the features of adjacent key points to predict the key points.

[0004] However, due to the small area of ​​the human hand, the high degree of freedom of the hand, the complex relationship between the joints, and the easy occurrence of self-occlusion, and the high self-similarity of the fingers themselves, current solutions are difficult to obtain high-precision hand posture estimation results. Summary of the invention

[0005] The present application provides a hand posture estimation method, apparatus, device and computer storage medium, which can improve the accuracy of hand posture estimation and obtain high-precision hand posture estimation results.

[0006] The technical solution of this application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for estimating a hand posture, the method comprising:

[0008] Determine a classification logic diagram corresponding to each of the multiple key points; wherein the multiple key points represent the skeleton key nodes of the target hand, and the first key point is any key point among the multiple key points;

[0009] Determine the coordinate information of the first key point based on a preset classification map and a classification logic map corresponding to the first key point;

[0010] After determining the coordinate information of each of the plurality of key points, the posture estimation result of the target hand is obtained.

[0011] In a second aspect, an embodiment of the present application provides a hand posture estimation device, the hand posture estimation device comprising a determination unit, a calculation unit and an estimation unit; wherein,

[0012] The determining unit is configured to determine a classification logic diagram corresponding to each of the plurality of key points; wherein the plurality of key points represent key nodes of the skeleton of the target hand, and the first key point is any key point among the plurality of key points;

[0013] The calculation unit is configured to determine the coordinate information of the first key point based on a preset classification diagram and a classification logic diagram corresponding to the first key point;

[0014] The estimation unit is configured to obtain a posture estimation result of the target hand after determining the coordinate information of each of the multiple key points.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor; wherein:

[0016] The memory is used to store executable instructions that can be run on the processor;

[0017] The processor is used to execute the method as described in the first aspect when running the executable instructions.

[0018] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a hand posture estimation program, and when the hand posture estimation program is executed by a processor, the method described in the first aspect is implemented.

[0019] The embodiment of the present application provides a method, device, equipment and computer storage medium for hand posture estimation, by determining a classification logic diagram corresponding to each of a plurality of key points; wherein the plurality of key points represent the skeleton key nodes of the target hand, and the first key point is any key point among the plurality of key points; based on the preset classification diagram and the classification logic diagram corresponding to the first key point, the coordinate information of the first key point is determined; after the coordinate information of each of the plurality of key points is determined, the posture estimation result of the target hand is obtained. In this way, for each key point, the classification logic diagram can be used for classification first, and then the coordinate information of the key point can be determined by the regression model. After the coordinate information of each of the plurality of key points is determined, the posture estimation result of the target hand can be obtained. In this way, the classification and regression methods are comprehensively considered, so that the accuracy of hand posture estimation can be improved, and a high-precision hand posture estimation result can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A schematic diagram of an image taken by a TOF camera provided in a related technical solution;

[0021] Figure 2 A schematic diagram of a hand bounding box detection result provided for a related technical solution;

[0022] Figure 3 A schematic diagram of key point positions of a hand skeleton provided for a related technical solution;

[0023] Figure 4 A schematic diagram of a two-dimensional hand posture estimation result provided for a related technical solution;

[0024] Figure 5 A schematic diagram of a traditional hand gesture detection process provided for a related technical solution;

[0025] Figure 6 A structural diagram of RoIAlign feature extraction provided for related technical solutions;

[0026] Figure 7 A structural schematic diagram of non-maximum suppression provided for a related technical solution;

[0027] Figure 8 A structural diagram of a union and intersection provided for a related technical solution;

[0028] Fig. 9 A schematic diagram of a hand posture estimation method provided in an embodiment of the present application;

[0029] Fig. 10A A schematic diagram of the Hour-Glass network feature structure of a segmentation model head provided in an embodiment of the present application;

[0030] Fig. 10B A schematic diagram of the convolutional neural network structure of a segmentation model head provided in an embodiment of the present application;

[0031] Fig.11 A schematic diagram of a network architecture provided for an embodiment of the present application;

[0032] Fig.12 A schematic diagram of the structure of a classification diagram provided in an embodiment of the present application;

[0033] Fig.13 A schematic diagram of a flow chart of another hand posture estimation method provided in an embodiment of the present application;

[0034] Fig.14 A flowchart of another hand posture estimation method provided in an embodiment of the present application;

[0035] Fig.15A schematic diagram of the structure of a hand posture estimation device provided in an embodiment of the present application;

[0036] Fig.16 A schematic diagram of a specific hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0038] Hand pose estimation mainly refers to accurately estimating the 3D coordinate positions of the hand skeleton nodes from an image. This is a key issue in the field of computer vision and human-computer interaction, and is of great significance in the fields of virtual reality, augmented reality, contactless interaction, and gesture recognition. With the rise and development of commercial and inexpensive depth cameras, hand pose estimation has made great progress.

[0039] Among them, depth cameras include structured light, laser scanning and TOF, etc., and in most cases refer to TOF cameras. Here, TOF is the abbreviation of Time of Fight, which literally means time of flight. The so-called three-dimensional (3D) imaging of the time of flight method is to continuously send light pulses to the target, and then use the sensor to receive the light returned from the object, and obtain the target distance of the object by detecting the flight (round-trip) time of the light pulse. Specifically, a TOF camera is a distance imaging camera system that uses the time of flight method to measure the round-trip time of the artificial light signal provided by a laser or light emitting diode (Light Emitting Diode, LED), thereby calculating the distance between the TOF camera and the object at each point on the image.

[0040] The TOF camera outputs an image of size H×W. Each pixel value on this two-dimensional (2D) image can represent the depth value of the pixel; wherein the pixel value ranges from 0 to 3000 millimeters (mm). Figure 1 A schematic diagram of an image captured by a TOF camera provided by a related technical solution is shown. In the embodiment of the present application, the image captured by the TOF camera can be referred to as a depth image.

[0041] For example, compared with TOF cameras from other manufacturers, the TOF camera provided by manufacturer O has the following differences: (1) the TOF camera can be installed inside a smartphone instead of being fixed on a static bracket; (2) compared with TOF cameras from other manufacturers (such as Microsoft Kinect or Intel Realsense, etc.), it has lower power consumption; (3) it has a lower image resolution, such as 240x180, while the typical value is 640x480.

[0042] It can be understood that the input of hand detection is a depth image, and then the output is the probability of the existence of the hand (that is, a number between 0 and 1, a larger value indicates a greater confidence in the existence of the hand) and the hand bounding box (that is, a bounding box indicating the position and size of the hand). Figure 2 FIG. 2 shows a schematic diagram of a hand bounding box detection result provided by a related technical solution. Figure 2 As shown, the black rectangular box is the hand bounding box, and the score of the hand bounding box is as high as 0.999884.

[0043] In the embodiment of the present application, a bounding box may also be referred to as a boundary box. Here, the bounding box may be represented as (xmin, ymin, xmax, ymax), where (xmin, ymin) represents the upper left corner of the bounding box, and (xmax, ymax) represents the lower right corner of the bounding box.

[0044] Specifically, in the process of 2D hand posture estimation, the input is a depth image and the output is the 2D key point positions of the hand skeleton. The key point positions of the hand skeleton are shown in the following example: Figure 3 As shown. Figure 3 In the hand skeleton, 20 key points can be set, and the position of each key point is as follows Figure 3 Here, each key point position can be represented by 2D coordinate information (x, y), where x is the coordinate information in the horizontal image axis direction and y is the coordinate information in the vertical image axis direction. For example, after determining the coordinate information of these 20 key points, a 2D hand posture estimation result is as follows: Figure 4 shown.

[0045] In the process of 3D hand posture estimation, the input is still the depth image, and the output is the 3D key point position of the hand skeleton. The key point position example of the hand skeleton is still as follows Figure 3 Here, each key point position can be represented by 3D coordinate information (x, y, z), where x is the coordinate information in the horizontal image axis direction, y is the coordinate information in the vertical image axis direction, and z is the coordinate information in the depth direction. The embodiment of the present application mainly solves the problem of three-dimensional hand posture estimation.

[0046] At present, a typical hand posture detection process may include a hand detection part and a hand posture estimation part, wherein the hand detection part may include a trunk feature extractor and a bounding box detection head module, and the hand posture estimation part may include a trunk feature extractor and a posture estimation head module. Exemplarily, Figure 5 FIG. 1 shows a flow chart of a conventional hand gesture detection process provided by a related technical solution. Figure 5 As shown in the figure, after obtaining an original depth image including a hand, hand detection can be performed first, that is, the backbone feature extractor and bounding box detection head module included in the hand detection part are used for detection processing; at this time, the bounding box boundary can be adjusted, and then the adjusted bounding box is used for image cropping, and the hand posture estimation is performed on the cropped image, that is, the backbone feature extractor and posture estimation head module included in the hand posture estimation part are used for posture estimation processing. It should be noted that the tasks of hand detection and hand posture estimation are completely separate. In order to connect these two tasks, the position of the output bounding box is adjusted to the centroid of the pixels in the bounding box, and the size of the bounding box is slightly enlarged to include all hand pixels. The adjusted bounding box is used to crop the original depth image. The cropped image is input into the task of hand posture estimation. Here, when the backbone feature extractor is used twice to extract image features, repeated calculations will be caused, which increases the amount of calculation.

[0047] At this time, RoIAlign can be introduced. Among them, ROIAlign is a regional feature aggregation method, which can effectively solve the problem of regional mismatch caused by double quantization in ROIPool operation. In the detection task, replacing ROIPool with ROIAlign can improve the accuracy of the detection results. In other words, the RoIAlign layer eliminates the strict quantization of RoIPool and correctly aligns the extracted features with the input. Here, any quantization of the RoI boundary or area can be avoided. For example, x / 16 can be used here instead of [x / 16]. In addition, bilinear interpolation can be used to calculate the exact value of the input features at four regularly sampled positions in each RoI area, and summarize the results (using the maximum value or average), such as Figure 6 As shown. Figure 6 In the figure, the dashed grid represents a feature map, the bold solid line represents an RoI (such as a 2×2 region), and 4 sampling points are dotted in each region. RoIAlign uses bilinear interpolation calculations on adjacent grid points on the feature map to obtain the value of each sampling point. No quantization is performed on any of the coordinates involved for an ROI, ROI region, or multiple sampling points. It is also important to note that as long as quantization is not performed, the detection results are insensitive to the accuracy of the sampling position or the number of sampling points.

[0048] In addition, Non-Maximum Suppression (NMS) has been widely used in several key aspects of computer vision and is an integral part of many detection methods such as edge, corner or object detection. Its necessity stems from the imperfect ability of detection algorithms to localize the concept of interest, resulting in multiple detection groups appearing near the actual location.

[0049] In the context of object detection, sliding window based methods usually produce multiple high-scoring windows that are close to the correct location of the object. This is a result of the generalization ability of the object detector, the smoothness of the response function, and the near-window visual correlation. Such a relatively dense output is usually unsatisfactory for understanding the content of the image. In fact, in this step, the number of window hypotheses is not correlated with the actual number of objects in the image. Therefore, the goal of NMS is to keep only one window per detection group, corresponding to the exact local maximum of the response function, ideally obtaining only one detection per object. A specific example of NMS is Figure 7 As shown, the purpose of NMS is just to keep a window (such as Figure 7 (bold grey rectangle in the figure).

[0050] like Figure 8 As shown, it shows a schematic diagram of a union and intersection provided by a related technical solution. Figure 8 In the example, two bounding boxes are given, denoted by BB1 and BB2. Here, the black area in (a) is the intersection of BB1 and BB2, denoted by BB1∩BB2, i.e., the overlapping area of ​​BB1 and BB2; the black area in (b) is the union of BB1 and BB2, denoted by BB1∪BB2, i.e., the merged area of ​​BB1 and BB2. Specifically, the calculation formula of the intersection over union (denoted by IoU) is as follows:

[0051]

[0052] In addition, the coordinates of each pixel in the image can be expressed in the XYZ coordinate system or the UVD coordinate system. Here, (x, y, z) is the pixel coordinate in the XYZ coordinate system, and (u, v, d) is the pixel coordinate in the UVD coordinate system. Assume that C x and C y represents the coordinates of the principal point, which should ideally be located at the center of the image; f x and f y are the focal lengths in the x-direction and y-direction respectively, then the conversion relationship between the UVD coordinate system and the XYZ coordinate system is shown in the following formula:

[0053]

[0054] It is also important to note that the classification predictive modeling problem is different from the regression predictive modeling problem. Classification is the task of predicting discrete class labels, while regression is the task of predicting continuous quantities. There is also some overlap between classification and regression algorithms; for example, classification algorithms can predict continuous values, but the continuous values ​​exist in the form of probabilities of class labels, and regression algorithms can predict discrete values, but the discrete values ​​are expressed in the form of integer quantities.

[0055] The current hand posture estimation scheme either uses a fully connected layer to regress the coordinates of the key points of the hand, or uses a classification-based method to predict the spatial position of the key points. Among them, the regression-based method calculates the hand posture in a global manner, that is, all the information of the key point features is used to predict each key point; in contrast, the classification-based method tends to be more local, that is, gradually acquiring the features of adjacent key points. Considering the global shape and geometry as well as the local shape and geometry, these two methods are complementary. In order to make better use of the advantages of these two methods, an embodiment of the present application proposes a method for hand posture estimation based on cascade classification and regression to improve the accuracy of hand posture estimation.

[0056] Based on this, the embodiment of the present application provides a method for hand posture estimation, the basic idea of ​​which is: determine the classification logic diagram corresponding to each of the multiple key points; wherein the multiple key points represent the skeleton key nodes of the target hand, and the first key point is any key point among the multiple key points; based on the preset classification diagram and the classification logic diagram corresponding to the first key point, determine the coordinate information of the first key point; after determining the coordinate information of each of the multiple key points, obtain the posture estimation result of the target hand. In this way, for each key point, the classification logic diagram can be used for classification first, and then the coordinate information of the key point can be determined by the regression model. After determining the coordinate information of each of the multiple key points, the posture estimation result of the target hand can be obtained. In this way, the classification and regression methods are comprehensively considered, so that the accuracy of hand posture estimation can be improved, and a high-precision hand posture estimation result can be obtained.

[0057] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0058] In one embodiment of the present application, see Fig. 9 , which shows a flow chart of a hand posture estimation method provided by an embodiment of the present application. Fig. 9 As shown, the method may include:

[0059] S901: Determine a classification logic diagram corresponding to each of the multiple key points; wherein the multiple key points represent the skeleton key nodes of the target hand, and the first key point is any key point among the multiple key points.

[0060] It should be noted that the method of the embodiment of the present application can be applied to a hand posture estimation device, or an electronic device integrated with the device. The electronic device can be a smart phone, a tablet computer, a laptop computer, a PDA, a navigation device, a wearable device, a desktop computer, etc., and the embodiment of the present application does not make any limitation.

[0061] It should also be noted that for a human hand, there can be multiple key nodes or key points in the skeleton of the hand. Usually, the hand includes 20 key points. The specific positions of these 20 key points in the hand are as follows: Figure 3 shown.

[0062] It can be understood that the classification logic map is obtained by convolution operation of the feature map, and the feature map is obtained on the basis of the previous segmentation of the head. Therefore, in some embodiments, the determination of the classification logic map corresponding to each of the multiple key points may include:

[0063] Get the feature map corresponding to the target hand;

[0064] The multiple key points in the feature map are classified to obtain classification logic maps corresponding to the multiple key points.

[0065] Furthermore, in some embodiments, obtaining a feature map corresponding to the target hand may include:

[0066] Acquire a depth image including the target hand;

[0067] Performing hand detection processing on the depth image using a preset feature extractor to obtain an initial feature map including the target hand;

[0068] The initial feature map is convolved using the segmentation model head to obtain a feature map corresponding to the target hand.

[0069] It should be noted that the preset feature extractor may be a backbone feature extractor and a RoIAlign feature extractor. After acquiring a depth image including the target hand, the backbone feature extractor and the bounding box detection head module may be used to perform detection processing, and after selecting a suitable bounding box, the RoIAlign feature extractor may be used to perform feature mapping to obtain an initial feature map including the target hand.

[0070] For the initial feature map, the segmentation model head can be used to perform convolution processing to obtain the feature map corresponding to the target hand. The segmentation model head can include an upsampling block and a downsampling block. The hourglass (HG) network feature map of the segmentation model head is as follows: Fig. 10A As shown in Figure 2, the convolutional neural network structure is as follows: Fig. 10B Here, assuming that the initial feature map after the RoIAlign feature extractor is 8×8×256, the feature map obtained after the convolution processing of the segmentation model head is 16×16×128.

[0071] It should also be noted that after obtaining the feature map corresponding to the target hand, the multiple key points in the feature map may be classified to obtain the classification logic maps corresponding to the multiple key points. Specifically, in some embodiments, the classification of the multiple key points in the feature map to obtain the classification logic maps corresponding to the multiple key points may include:

[0072] The feature map is input into a preset classification neural network model, and the preset classification neural network model outputs the classification logic map corresponding to each of the multiple key points.

[0073] That is to say, by inputting the obtained feature map into the preset classification neural network model, the classification logic map corresponding to each of the 20 key points can be obtained. Among them, the preset classification neural network model is obtained by training the classification neural network model using the training sample set.

[0074] For example, Fig.11 As shown, if the feature map obtained by the segmentation model head in the embodiment of the present application is 16×16×128, then the 16×16×128 feature map is subjected to a 1×1×20 convolution operation, and a 16×16×20 classification logic map can be output. i Here, each of the 20 key points has a 16×16 classification logic graph.

[0075] In this way, after obtaining the classification logic diagrams corresponding to the multiple key points, the coordinate information of the key points can be calculated by taking the classification logic diagram corresponding to a certain key point (such as the first key point) among the multiple key points as an example.

[0076] S902: Determine the coordinate information of the first key point based on a preset classification diagram and a classification logic diagram corresponding to the first key point.

[0077] It should be noted that the first key point is any key point among the multiple key points, and the preset classification diagram is a classification diagram predefined in the embodiment of the present application. Specifically, in some embodiments, for S902, the determination of the coordinate information of the first key point based on the preset classification diagram and the classification logic diagram corresponding to the first key point may include:

[0078] Based on the classification logic diagram corresponding to the first key point, determine the triple information of each grid in the preset classification diagram;

[0079] The coordinate information of the first key point is determined according to the triplet information of each grid in the preset classification map.

[0080] In the embodiment of the present application, the triplet information may include a network probability value (denoted by p i ), the horizontal coordinate value of the center pixel of the grid (indicated by C ix ) and vertical coordinate values ​​(in C iy , where i represents the i-th grid in the preset classification graph, i∈{0, 1.., 15}×{0, 1, ..., 15}.

[0081] It is understandable that before determining the triple information of each grid in the preset classification map, the embodiment of the present application needs to predefine a classification map. Therefore, in some embodiments, the method may also include: establishing a preset classification map; wherein the number of grids in the preset classification map is related to the size parameter of the classification logic map.

[0082] That is to say, the embodiment of the present application predefines a classification diagram S={t i}, i∈{0,1..,15}×{0,1,...,15}. In the classification map, the coordinate information (including horizontal coordinate value and vertical coordinate value) of each network center pixel in the determined classification logic map and the preset classification map can be included. Here, if the classification logic map has a size parameter of 16×16, then the classification map corresponds to a grid number of 16×16. Usually, the shape of the classification map is 112×112 pixels, 16×16 grids, then each network occupies 7×7 pixels, such as Fig.12 An example of the structure of a classification map is shown. In the embodiment of the present application, the preset classification map is pre-set, corresponding to 16×16 grids, and the classification logic map is different according to each input image, and a different 16×16×20 classification logic map will be output, but the 16×16 in the classification logic map and the 16×16 in the preset classification map correspond.

[0083] Further, in some embodiments, determining the triplet information of each grid in the preset classification diagram based on the classification logic diagram corresponding to the first key point may include:

[0084] Based on the classification logic diagram corresponding to the first key point, using the first calculation model to calculate the grid probability value corresponding to each grid;

[0085] Get the horizontal coordinate value and vertical coordinate value of the center pixel of each grid;

[0086] According to the grid probability value corresponding to each grid and the horizontal coordinate value and the vertical coordinate value of the center pixel of the grid, the triplet information of each grid in the preset classification map is composed.

[0087] That is, each network (t i ) and a triplet of information (p i , C ix , C iy ). Among them, p i is the grid probability value of the i-th grid, C ix and C iy are the horizontal and vertical coordinate values ​​of the center pixel of the i-th grid.

[0088] Here, the grid probability value can be calculated using the classification logic diagram and the first calculation model. Among them, the first calculation model can be a softmax function. The softmax function can also be called a normalized exponential function, which is actually the gradient logarithm normalization of a finite discrete probability distribution and a generalization of the logic function. It can "compress" a K-dimensional vector z containing any real number into another K-dimensional real vector δ(z), so that the range of each element is between (0,1), and the sum of all elements is 1. Specifically, the formula of the first calculation model is as follows,

[0089]

[0090] In formula (3), the numerator can map the real number output to zero to positive infinity through the exponential function, and the denominator is normalized by adding all the results. In this way, for the classification logic graph {y i}, the softmax function shown in formula (3) can be used to convert the multi-classification output into a grid probability value.

[0091] Furthermore, after obtaining the network probability value (p i ), then combine the coordinate information of the center pixel of each grid (C ix , C iy ), forming the triplet information of each grid in the preset classification map. According to the triplet information of each grid in the preset classification map, the coordinate information of the first key point can be obtained. Specifically, in some embodiments, the coordinate information of the first key point is determined according to the triplet information of each grid in the preset classification map, which may include: performing regression calculation based on the grid probability value corresponding to each grid in the preset classification map, the horizontal coordinate value and the vertical coordinate value of the grid center pixel, to obtain the coordinate information of the first key point.

[0092] Further, in some embodiments, performing regression calculation based on the grid probability value corresponding to each grid in the preset classification map, the horizontal coordinate value and the vertical coordinate value of the center pixel of the grid to obtain the coordinate information of the first key point may include:

[0093] Based on the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the center pixel of the grid, the horizontal coordinate information of the first key point is obtained by a second calculation model;

[0094] Based on the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the center pixel of the grid, the vertical coordinate information of the first key point is obtained by a third calculation model;

[0095] The coordinate information of the first key point is obtained according to the horizontal coordinate information of the first key point and the vertical coordinate information of the first key point.

[0096] It should be noted that the regression calculation can be implemented by a preset regression model. In other words, based on the grid probability value corresponding to each grid in the preset classification map, the horizontal coordinate value and the vertical coordinate value of the grid center pixel, the coordinate information of the first key point can be obtained by the preset regression model.

[0097] In an embodiment of the present application, the preset regression model is obtained by training the regression model using a training sample set. Here, the preset regression model may include a second calculation model and a third calculation model. The second calculation model is used to calculate the horizontal coordinate value of the first key point, and the third calculation model is used to calculate the vertical coordinate value of the first key point.

[0098] That is to say, for the first key point, the second calculation model can be used in combination with the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the center pixel of the grid to calculate the horizontal coordinate information of the first key point. Specifically, the horizontal coordinate information of the first key point obtained by the second calculation model based on the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the center pixel of the grid can include:

[0099] The second calculation model is used to perform weighted sum calculation on the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the grid center pixel, and the calculated weighted sum value is determined as the horizontal coordinate information of the first key point.

[0100] It should be noted that the calculation formula of the second calculation model is as follows:

[0101] x=∑ i p i *C ix (4)

[0102] Here, i represents the i-th grid in the preset classification graph, i∈{0, 1.., 5}×{0, 1, ..., 15}.

[0103] It should also be noted that, for the first key point, the vertical coordinate information of the first key point can be calculated by using the third calculation model in combination with the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the center pixel of the grid. Specifically, the vertical coordinate information of the first key point obtained by the third calculation model based on the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the center pixel of the grid can include:

[0104] The third calculation model is used to perform weighted sum calculation on the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the grid center pixel, and the calculated weighted sum value is determined as the vertical coordinate information of the first key point.

[0105] It should be noted that the calculation formula of the third calculation model is as follows:

[0106] y=∑ i p i *C iy (5)

[0107] Here, i represents the i-th grid in the preset classification graph, i∈{0, 1.., 15}×{0, 1, ..., 15}.

[0108] It should be noted that equations (4) and (5) are used to achieve normalization, and the coordinate information (x, y) of the first key point can be obtained, where x represents the horizontal coordinate information of the first key point, and y represents the vertical coordinate information of the first key point.

[0109] In this way, since any key point can obtain the coordinate information of the key point by executing S902 and S903, for multiple key points, the coordinate information of each of the multiple key points can be obtained by executing S902 and S903 multiple times.

[0110] S903: After determining the coordinate information of each of the plurality of key points, obtaining a posture estimation result of the target hand.

[0111] It should be noted that after obtaining the coordinate information of multiple key points, it is usually as follows Figure 3 The coordinate information of the 20 key points shown can be used to estimate the posture of the target hand, such as Figure 4 The pose estimation result of a target hand is shown.

[0112] Furthermore, if the coordinate information of each key point is a three-dimensional coordinate, in addition to (x, y), it can also include the coordinate value in the depth direction (expressed by z), that is, the coordinate information of each key point is (x, y, z), then the three-dimensional posture estimation result of the target hand can be obtained.

[0113] This embodiment provides a hand posture estimation method, by determining the classification logic diagram corresponding to each of the multiple key points; wherein the multiple key points represent the skeleton key nodes of the target hand, and the first key point is any key point among the multiple key points; based on the preset classification diagram and the classification logic diagram corresponding to the first key point, the coordinate information of the first key point is determined; after the coordinate information of each of the multiple key points is determined, the posture estimation result of the target hand is obtained. In this way, for each key point, the classification logic diagram can be used for classification first, and then the coordinate information of the key point can be determined by the regression model. After the coordinate information of each of the multiple key points is determined, the posture estimation result of the target hand can be obtained. In this way, the classification and regression methods are comprehensively considered, so that the accuracy of hand posture estimation can be improved, and a high-precision hand posture estimation result can be obtained.

[0114] In another embodiment of the present application, see Fig.13 , which shows a flow chart of another hand posture estimation method provided by an embodiment of the present application. Fig.13 As shown, the method may include:

[0115] S1301: Determine a classification logic diagram corresponding to a first key point; wherein the first key point is any key point among a plurality of key points of a target hand.

[0116] S1302: Based on the classification logic diagram corresponding to the first key point, determine the triplet information of each grid in the preset classification diagram.

[0117] S1303: Determine the coordinate information of the first key point according to the triplet information of each grid in the preset classification map.

[0118] It should be noted that the first key point is any key point among the multiple key points. After obtaining the classification logic diagram corresponding to the first key point, the coordinate information of the first key point can be calculated according to S1302 and S1303. Among them, the triple information can include the network probability value (expressed as p i ), the horizontal coordinate value of the center pixel of the grid (indicated by C ix ) and vertical coordinate values ​​(in C iy , where i represents the i-th grid in the preset classification graph, i∈{0,1..,15}×{0,1,...,15}.

[0119] In this way, according to the triplet information of each grid in the preset classification map, the coordinate information of the first key point can be calculated in combination with the above formula (4) and formula (5).

[0120] S1304: Scale the coordinate information of the first key point, and determine the scaled coordinate information as the coordinate information of the first key point.

[0121] It should be noted that after obtaining the coordinate information of the first key point, the coordinate information of the first key point may be scaled (eg, multiplied), and then the scaled coordinate information is determined as the coordinate information of the first key point.

[0122] In some embodiments, the coordinate information of the first key point includes horizontal coordinate information of the first key point and vertical coordinate information of the first key point. Further, scaling the coordinate information of the first key point and determining the scaled coordinate information as the coordinate information of the first key point may include:

[0123] Scaling the horizontal coordinate information of the first key point using a first preset value to obtain scaled horizontal coordinate information of the first key point;

[0124] Scaling the vertical coordinate information of the first key point using a second preset value to obtain scaled vertical coordinate information of the first key point;

[0125] The coordinate information of the first key point is obtained according to the scaled horizontal coordinate information of the first key point and the scaled vertical coordinate information of the first key point.

[0126] Here, the first preset value and the second preset value may be the same or different. The first preset value is used to scale the horizontal coordinate information of the first key point, that is, multiplying the horizontal coordinate information of the first key point by the first preset value; the second preset value is used to scale the vertical coordinate information of the first key point, that is, multiplying the vertical coordinate information of the first key point by the second preset value.

[0127] In an embodiment of the present application, the first preset value and the second preset value may be determined according to a ratio of a size parameter of a bounding box of the target hand to a size parameter of a preset classification graph. In some embodiments, the method may further include:

[0128] Acquire size parameters of a bounding box including the target hand and size parameters of the preset classification map;

[0129] Calculating a first ratio of the size parameter of the bounding box to the size parameter of the preset classification graph in the horizontal direction, and determining the first ratio as the first preset value;

[0130] A second ratio of the size parameter of the bounding box to the size parameter of the preset classification map in the vertical direction is calculated, and the second ratio is determined as the second preset value.

[0131] It should be noted that, assuming that the bounding box size parameter of the target hand is (W B , H B ) indicates that the size parameter of the preset classification map is (112, 112), so the first preset value is W B / 112, the second preset value is H B / 112. At this time, the horizontal coordinate information of the first key point can be multiplied by W B / 112, multiply the vertical coordinate information of the first key point by H B / 112 to obtain the coordinate information of the first key point after scaling.

[0132] Thus, since any key point can be executed through S1301 to S1304 to obtain the coordinate information of the key point, for multiple key points, by executing S1301 to S1304 multiple times, the coordinate information of each of the multiple key points can be obtained. Thus, after obtaining the coordinate information of each of the multiple key points, the posture estimation of the target hand can be realized, for example Figure 4 The result of the target hand posture estimation is shown in FIG. At this time, if the coordinate information of each key point is a three-dimensional coordinate, in addition to (x, y), it can also include the coordinate value in the depth direction (expressed by z), that is, the coordinate information of each key point is (x, y, z), then the three-dimensional posture estimation result of the target hand can be obtained.

[0133] The present embodiment provides a hand posture estimation method. The specific implementation of the aforementioned embodiment is elaborated in detail through the above embodiment. It can be seen that through the technical scheme of the aforementioned embodiment, for each key point, it is possible to first classify it using a classification logic diagram, and then use a regression model to determine the coordinate information of the key point. After determining the coordinate information of each of these multiple key points, the posture estimation result of the target hand can be obtained. In this way, the two methods of classification and regression are comprehensively considered, so that the accuracy of hand posture estimation can be improved, and a high-precision hand posture estimation result can be obtained.

[0134] In another embodiment of the present application, see Fig.14 , which shows a flow chart of another method for estimating hand posture provided by an embodiment of the present application. Fig.14 As shown, the method may include:

[0135] S1401: Acquire a training sample set; wherein the training sample set includes a plurality of training images;

[0136] S1402: constructing a classification neural network model and a regression model, and based on each training image in the training sample set, performing model training on the classification neural network model using a first preset loss function, and performing model training on the regression model using a second preset loss function;

[0137] S1403: When the loss value of the first preset loss function is within a first preset interval, determining the trained classification neural network model as the preset classification neural network model;

[0138] S1404: When the loss value of the second preset loss function is within a second preset interval, the trained regression model is determined as the preset regression model.

[0139] It should be noted that the first preset loss function is the softmax cross entropy loss function, and the second preset function is the Smooth L1 loss function.

[0140] It should also be noted that Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Moreover, convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure. Therefore, they are also called "Shift-Invariant Artificial Neural Networks (SIANN)".

[0141] It should be understood that a convolutional neural network consists of an input layer and an output layer and multiple hidden layers. The hidden layers of a CNN are usually composed of a series of convolutional layers that are convolved with multiplications or other dot convolutions. The activation function is usually a Rectified Linear Unit (ReLU) layer, followed by additional convolutional layers, such as pooling layers, fully connected layers, and normalization layers. Since their inputs and outputs are masked by the activation function and the final convolutional layer, these layers can also be called hidden layers. In turn, the final convolutional layer often involves backpropagation to more accurately measure the final product. Although these layers are colloquially called convolutions, this is just a convention. Mathematically speaking, it is technically a sliding dot product or cross-correlation. This has important implications for the indicators in the matrix because it affects the way the weights are determined at specific indicator points.

[0142] When programming a CNN, each convolutional layer in the neural network should have the following properties: (1) The input is a tensor with the shape (number of images) × (image width) × (image height) × (image depth). (2) The convolution kernel has width and height as hyperparameters, and its depth must be equal to the image depth. The convolutional layer convolves the input and passes its result to the next layer. This is similar to how neurons in the visual cortex respond to specific stimuli.

[0143] It is also important to note that each convolutional neuron processes only data within its receptive field (or receptive domain). While fully connected feedforward neural networks can be used to learn features and classify data, it is not practical to apply this structure to images. Even in shallow (as opposed to deep) architectures, a very large number of neurons are required due to the very large input size parameter associated with images, where each pixel is a relevant variable. For example, a fully connected layer for a (small) image of size 100×100 has 10,000 weights for each neuron in the second layer. The convolution operation solves this problem because it reduces the number of free parameters, allowing the network to be deepened with fewer parameters. For example, regardless of the image size parameter, a tiled area of ​​size 5×5 (each area has the same shared weights) only requires 25 learnable parameters. That is, this method solves the problem of gradient disappearance or explosion in traditional multi-layer neural network training where multiple layers are implemented using backpropagation.

[0144] Furthermore, convolutional neural networks can also include local or global pooling layers to simplify the underlying calculations. Pooling layers can reduce the dimensionality of data by merging the outputs of a layer of neuron clusters into a single neuron in the next layer. Local pooling combines small clusters, usually 2×2, while global pooling acts on all neurons in the convolutional layer. In addition, pooling can also calculate the average or maximum value. Among them, maximum pooling uses the maximum value of each neuron cluster in the previous layer, and average pooling uses the average value of each neuron cluster in the previous layer.

[0145] It should also be noted that the fully connected layer connects each neuron in one layer to each neuron in another layer. It is the same in principle as the traditional multi-layer perceptron (MLP) neural network. The flat matrix classifies the image through a fully connected layer.

[0146] In model training, a common loss function is the cross entropy loss of softmax, or logarithmic loss, which is used to measure the performance of a classification neural network model whose output is a probability value between 0 and 1. Cross entropy is used to determine how close the actual output is to the expected output, and the cross entropy loss value increases as the predicted probability deviates from the actual label. Therefore, when the actual observed label is 1, a predicted probability of 0.012 indicates that the classification result is very poor and will result in a higher loss value. The logarithmic loss value expected by an ideal model is 0. The specific formula is as follows,

[0147]

[0148] Where M is the number of classes (such as dog, cat, fish, etc.); log is the natural logarithm; y is a binary indicator (0 or 1) if class label c is the correct classification for observation oo; and p is the predicted probability that observation oo is of class c.

[0149] In some embodiments, for training a classification neural network model, the method may include:

[0150] Input each training image in the training sample set into the classification neural network model to obtain a classification logic diagram of each key point in each training image;

[0151] Based on the classification logic map of each key point in each training image, determine the predicted probability value of each key point in each training image;

[0152] According to the predicted probability value of each key point in each training image, the loss value of the classification neural network model is calculated by using the softmax cross entropy loss function;

[0153] When the loss value is not within the first preset interval, updating the model parameters of the classification neural network model according to the loss value, and continuing to perform the step of inputting each training image in the training sample set into the classification neural network model until the loss value is within the first preset interval;

[0154] When the loss value is in a first preset interval, the trained classification neural network model is determined as the preset classification neural network model.

[0155] That is to say, the softmax cross entropy loss function shown in formula (6) can be used to supervise the training process of the classification neural network model to ensure that the loss value of the preset classification neural network model obtained after training is in the first preset interval, which can improve the classification accuracy of each key point.

[0156] It should also be noted that the Smooth L1 loss function is usually used for regression. Specifically, the prediction of the bounding box is a regression problem. Usually the Mean Squared Error (MSE) loss function (L2 loss) can be selected, but this loss has a high penalty for relatively large errors; a slightly milder absolute loss function (L1 loss) can also be used, which grows linearly with the error rather than quadratically, but the derivative of this function does not exist at zero, which may affect convergence. Therefore, a common solution is a piecewise function, which uses the mean squared error square function near the zero point to make it smoother, that is, the Smooth L1 loss function. In other words, compared with MSE, it is less sensitive to outliers because it only considers the error as squared within the interval. The specific formula is as follows,

[0157]

[0158] Among them, L δ represents the loss value, y represents the true value, represents the predicted value; δ represents the preset constant value, which can usually be equal to 1.

[0159] In some embodiments, for training a classification neural network model, the method may include:

[0160] Input each training image in the training sample set into the regression model to obtain predicted coordinate information of each key point in each training image;

[0161] According to the difference between the predicted coordinate information and the real coordinate information of each key point in each training image, the loss value of the regression model is calculated by the Smooth L1 loss function;

[0162] When the loss value is not within the second preset interval, updating the model parameters of the regression model according to the loss value, and continuing to perform the step of inputting each training image in the training sample set into the regression model until the loss value is within the second preset interval;

[0163] When the loss value is in the second preset interval, the trained regression model is determined as the preset regression model.

[0164] That is to say, in the training sample set, each training image includes multiple key points of the hand and the real coordinate information corresponding to the multiple key points. The Smooth L1 loss function shown in formula (7) can be used to supervise the training process of the regression model to ensure that the loss value of the preset regression model obtained after training is in the second preset interval, which can improve the accuracy of the coordinate information of each key point.

[0165] In this way, after training the preset classification neural network model and the preset regression model, the preset classification neural network model can be used to obtain the classification logic map of each key point, and a preset classification map can be defined. Then, for the classification logic map of each key point, the triplet information of each grid in the preset classification map corresponding to the classification logic map is determined. At the same time, the coordinate information of each key point is normalized in combination with the preset regression model to obtain the coordinate information of each key point, thereby realizing the posture estimation of the hand.

[0166] In short, in the embodiment of the present application, the hand posture estimation method mainly includes the following design rules:

[0167] Rule #1: The cascade method of classification and regression is to perform pose estimation based on the previous head segmentation. After the head segmentation, the 16×16×128 features are convolved by 1×1×20 to obtain a 16×16×20 classification logic map Y={y i}, i∈{0, 1.., 15}×{0, 1, ..., 15}, where each of the 20 key points has a 16×16 classification logic map, such as Fig.11 An example of a network architecture provided by an embodiment of the present application is shown.

[0168] Rule #2: This embodiment of the application predefines a classification diagram S = {t i}, i∈{0, 1.., 15}×{0, 1, ..., 15}, to contain the output classification logic map and the center pixel coordinate information of each grid in the classification map. The shape of the classification map is 112×112 pixels, 16×16 grids, and each grid occupies 7×7 pixels. Here, each grid t i With a triple (p i , C ix , C iy ) related, p i is the grid probability value, C ix and C iy are the horizontal and vertical coordinate values ​​of the center pixel of the grid. Fig.12 A structural example of a classification diagram provided in an embodiment of the present application is shown.

[0169] Rule #3: In Rule 2, the grid probability value is calculated as the softmax function of the classification logistic map

[0170] Rule #4: The coordinates (x, y) of each key point are calculated as x = ∑ i p i *c ix , y = ∑ i p i *ciy .

[0171] Rule #5: During inference, the keypoint coordinates (x, y) can also be calculated based on the detected bounding box size parameters (W B , H B ) to the size parameter (112, 112) of the classification graph (e.g., (W B / 112,H B / 112)) for scaling (e.g., multiplication).

[0172] Rule #6: During training, the classification logistic map can be supervised by the softmax cross entropy loss function, and the keypoint coordinates (x, y) can be supervised by the Smooth L1 loss function.

[0173] The present embodiment provides a hand posture estimation method. The specific implementation of the aforementioned embodiment is elaborated in detail through the above embodiment. It can be seen that through the technical scheme of the aforementioned embodiment, for each key point, it is possible to first classify it using a classification logic diagram, and then use a regression model to determine the coordinate information of the key point. After determining the coordinate information of each of these multiple key points, the posture estimation result of the target hand can be obtained. In this way, the two methods of classification and regression are comprehensively considered, so that the accuracy of hand posture estimation can be improved, and a high-precision hand posture estimation result can be obtained.

[0174] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Fig.15 , which shows a schematic diagram of the composition structure of a hand posture estimation device 150 provided in an embodiment of the present application. Fig.15 As shown, the hand posture estimation device 150 may include: a determination unit 1501, a calculation unit 1502 and an estimation unit 1503; wherein,

[0175] The determining unit 1501 is configured to determine a classification logic diagram corresponding to each of the plurality of key points; wherein the plurality of key points represent key nodes of the skeleton of the target hand, and the first key point is any key point among the plurality of key points;

[0176] A calculation unit 1502 is configured to determine the coordinate information of the first key point based on a preset classification diagram and a classification logic diagram corresponding to the first key point;

[0177] The estimation unit 1503 is configured to obtain a posture estimation result of the target hand after determining the coordinate information of each of the multiple key points.

[0178] In some embodiments, see Fig.15, the hand posture estimation device 150 may further include an acquisition unit 1504 configured to acquire a feature map corresponding to the target hand;

[0179] The calculation unit 1502 is further configured to perform classification processing on the multiple key points in the feature map to obtain classification logic maps corresponding to each of the multiple key points.

[0180] In some embodiments, the acquisition unit 1504 is further configured to acquire a depth image including the target hand;

[0181] The computing unit 1502 is further configured to perform hand detection processing on the depth image using a preset feature extractor to obtain an initial feature map including the target hand; and perform convolution processing on the initial feature map using a segmentation model head to obtain a feature map corresponding to the target hand.

[0182] In some embodiments, the determination unit 1501 is specifically configured to determine the triplet information of each grid in the preset classification map based on the classification logic map corresponding to the first key point; and determine the coordinate information of the first key point based on the triplet information of each grid in the preset classification map.

[0183] In some embodiments, the triplet information includes a grid probability value, a horizontal coordinate value and a vertical coordinate value of a grid center pixel.

[0184] In some embodiments, the calculation unit 1502 is further configured to obtain a grid probability value corresponding to each grid using a first calculation model based on the classification logic diagram corresponding to the first key point;

[0185] The acquisition unit 1504 is further configured to acquire the horizontal coordinate value and the vertical coordinate value of the center pixel of each grid corresponding to the grid;

[0186] The determination unit 1501 is further configured to compose triplet information of each grid in the preset classification map according to the grid probability value corresponding to each grid and the horizontal coordinate value and the vertical coordinate value of the center pixel of the grid.

[0187] In some embodiments, the calculation unit 1502 is further configured to perform regression calculation based on the grid probability value corresponding to each grid in the preset classification map, the horizontal coordinate value and the vertical coordinate value of the grid center pixel to obtain the coordinate information of the first key point.

[0188] Further, the calculation unit 1502 is specifically configured to obtain the horizontal coordinate information of the first key point through a second calculation model based on the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the center pixel of the grid; and obtain the vertical coordinate information of the first key point through a third calculation model based on the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the center pixel of the grid;

[0189] The determining unit 1501 is further configured to obtain the coordinate information of the first key point according to the horizontal coordinate information of the first key point and the vertical coordinate information of the first key point.

[0190] In some embodiments, the computing unit 1502 is specifically configured to use the second computing model to perform weighted sum calculations on the grid probability values ​​corresponding to each grid in the preset classification map and the horizontal coordinate values ​​of the grid center pixels, and determine the calculated weighted sum value as the horizontal coordinate information of the first key point.

[0191] In some embodiments, the computing unit 1502 is specifically configured to use the third computing model to perform weighted sum calculations on the grid probability values ​​corresponding to each grid in the preset classification map and the vertical coordinate values ​​of the grid center pixels, and determine the calculated weighted sum value as the vertical coordinate information of the first key point.

[0192] In some embodiments, see Fig.15 The hand posture estimation device 150 may further include a scaling unit 1505 configured to scale the coordinate information of the first key point and determine the scaled coordinate information as the coordinate information of the first key point.

[0193] In some embodiments, the coordinate information of the first key point includes horizontal coordinate information of the first key point and vertical coordinate information of the first key point;

[0194] The scaling unit 1505 is specifically configured to scale the horizontal coordinate information of the first key point using a first preset value to obtain the scaled horizontal coordinate information of the first key point; and scale the vertical coordinate information of the first key point using a second preset value to obtain the scaled vertical coordinate information of the first key point;

[0195] The determining unit 1501 is further configured to obtain coordinate information of the first key point according to the scaled horizontal coordinate information of the first key point and the scaled vertical coordinate information of the first key point.

[0196] In some embodiments, the acquisition unit 1504 is further configured to acquire size parameters of a bounding box including the target hand and size parameters of the preset classification map;

[0197] The calculation unit 1502 is further configured to calculate a first ratio of the bounding box size parameters and the size parameters of the preset classification map in the horizontal direction, and determine the first ratio as the first preset value; and calculate a second ratio of the bounding box size parameters and the size parameters of the preset classification map in the vertical direction, and determine the second ratio as the second preset value.

[0198] It can be understood that in this embodiment, a "unit" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0199] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0200] Therefore, this embodiment provides a computer storage medium, which stores a hand posture estimation program. When the hand posture estimation program is executed by at least one processor, the steps of the method described in any one of the above embodiments are implemented.

[0201] Based on the above-mentioned composition of the hand posture estimation device 150 and the computer storage medium, see Fig.16 , which shows a specific hardware structure diagram of the electronic device 160 provided in the embodiment of the present application. Fig.16As shown, it may include: a communication interface 1601, a memory 1602 and a processor 1603; each component is coupled together through a bus system 1604. It is understood that the bus system 1604 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1604 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.16 In FIG. 1 , various buses are labeled as bus system 1604. Among them,

[0202] Communication interface 1601, used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0203] Memory 1602, used to store executable instructions that can be run on processor 1603;

[0204] The processor 1603 is configured to execute, when running the executable instruction:

[0205] Determine a classification logic diagram corresponding to each of the multiple key points; wherein the multiple key points represent the skeleton key nodes of the target hand, and the first key point is any key point among the multiple key points;

[0206] Determine the coordinate information of the first key point based on a preset classification map and a classification logic map corresponding to the first key point;

[0207] After determining the coordinate information of each of the plurality of key points, the posture estimation result of the target hand is obtained.

[0208] It can be understood that the memory 1602 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 1602 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0209] The processor 1603 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1603. The above-mentioned processor 1603 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1602, and the processor 1603 reads the information in the memory 1602 and completes the steps of the above method in combination with its hardware.

[0210] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0211] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0212] Optionally, as another embodiment, the processor 1603 is further configured to execute the steps of the method described in any one of the aforementioned embodiments when running the computer program.

[0213] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0214] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0215] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0216] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0217] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0218] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0219] Industrial Applicability

[0220] In the embodiment of the present application, firstly, the classification logic diagram corresponding to each of the multiple key points is determined, and the first key point is any key point among the multiple key points; then, based on the preset classification diagram and the classification logic diagram corresponding to the first key point, the coordinate information of the first key point is determined; after the coordinate information of each of the multiple key points is determined, the posture estimation result of the target hand is obtained. In this way, for each key point, the classification logic diagram can be used for classification first, and then the coordinate information of the key point can be determined by the regression model. After the coordinate information of each of the multiple key points is determined, the posture estimation result of the target hand can be obtained. In this way, the classification and regression methods are comprehensively considered, so that the accuracy of the hand posture estimation can be improved, and a high-precision hand posture estimation result can be obtained.

Claims

1. A hand posture estimation method, the method comprising: Determine a classification logic diagram corresponding to each of the multiple key points; wherein the multiple key points represent the skeleton key nodes of the target hand, and the first key point is any key point among the multiple key points; Determine the coordinate information of the first key point based on a preset classification map and a classification logic map corresponding to the first key point; After determining the coordinate information of each of the plurality of key points, the posture estimation result of the target hand is obtained. The determining of the coordinate information of the first key point based on a preset classification diagram and a classification logic diagram corresponding to the first key point includes: Based on the classification logic diagram corresponding to the first key point, determine the triple information of each grid in the preset classification diagram; Determine the coordinate information of the first key point according to the triple information of each grid in the preset classification map, The triplet information includes a grid probability value, a horizontal coordinate value and a vertical coordinate value of a grid center pixel. The step of determining the coordinate information of the first key point according to the triple information of each grid in the preset classification map includes: Regression calculation is performed based on the grid probability value corresponding to each grid in the preset classification map, the horizontal coordinate value and the vertical coordinate value of the grid center pixel to obtain the coordinate information of the first key point.

2. The method according to claim 1, wherein: The step of determining the classification logic diagrams corresponding to the multiple key points includes: Get the feature map corresponding to the target hand; The multiple key points in the feature map are classified to obtain classification logic maps corresponding to each of the multiple key points.

3. The method according to claim 2, wherein: The step of obtaining a feature map corresponding to the target hand includes: Acquire a depth image including the target hand; Performing hand detection processing on the depth image using a preset feature extractor to obtain an initial feature map including the target hand; The initial feature map is convolved using the segmentation model head to obtain a feature map corresponding to the target hand.

4. The method according to claim 1, wherein: The step of determining the triplet information of each grid in the preset classification diagram based on the classification logic diagram corresponding to the first key point includes: Based on the classification logic diagram corresponding to the first key point, using the first calculation model to obtain a grid probability value corresponding to each grid; Get the horizontal coordinate value and vertical coordinate value of the center pixel of each grid; According to the grid probability value corresponding to each grid and the horizontal coordinate value and the vertical coordinate value of the center pixel of the grid, the triplet information of each grid in the preset classification map is composed.

5. The method according to claim 1, wherein: The performing regression calculation based on the grid probability value corresponding to each grid in the preset classification map, the horizontal coordinate value and the vertical coordinate value of the center pixel of the grid to obtain the coordinate information of the first key point includes: Based on the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the center pixel of the grid, the horizontal coordinate information of the first key point is obtained by a second calculation model; Based on the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the center pixel of the grid, the vertical coordinate information of the first key point is obtained by a third calculation model; The coordinate information of the first key point is obtained according to the horizontal coordinate information of the first key point and the vertical coordinate information of the first key point.

6. The method according to claim 5, wherein: The obtaining the horizontal coordinate information of the first key point by a second calculation model based on the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the center pixel of the grid includes: The second calculation model is used to perform weighted sum calculation on the grid probability value corresponding to each grid in the preset classification map and the horizontal coordinate value of the grid center pixel, and the calculated weighted sum value is determined as the horizontal coordinate information of the first key point.

7. The method according to claim 5, wherein: The obtaining of the vertical coordinate information of the first key point by a third calculation model based on the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the center pixel of the grid includes: The third calculation model is used to perform weighted sum calculation on the grid probability value corresponding to each grid in the preset classification map and the vertical coordinate value of the grid center pixel, and the calculated weighted sum value is determined as the vertical coordinate information of the first key point.

8. The method according to claim 1, wherein: After determining the coordinate information of the first key point, the method further includes: The coordinate information of the first key point is scaled, and the scaled coordinate information is determined as the coordinate information of the first key point.

9. The method according to claim 8, wherein: The coordinate information of the first key point includes horizontal coordinate information of the first key point and vertical coordinate information of the first key point; Scaling the coordinate information of the first key point and determining the scaled coordinate information as the coordinate information of the first key point includes: Scaling the horizontal coordinate information of the first key point using a first preset value to obtain scaled horizontal coordinate information of the first key point; Scaling the vertical coordinate information of the first key point using a second preset value to obtain scaled vertical coordinate information of the first key point; The coordinate information of the first key point is obtained according to the scaled horizontal coordinate information of the first key point and the scaled vertical coordinate information of the first key point.

10. The method according to claim 9, wherein: The method further comprises: Acquire size parameters of a bounding box including the target hand and size parameters of the preset classification map; Calculating a first ratio of the size parameter of the bounding box to the size parameter of the preset classification graph in the horizontal direction, and determining the first ratio as the first preset value; A second ratio of the size parameter of the bounding box to the size parameter of the preset classification map in the vertical direction is calculated, and the second ratio is determined as the second preset value.

11. A hand posture estimation device, comprising a determination unit, a calculation unit and an estimation unit; wherein: The determining unit is configured to determine a classification logic diagram corresponding to each of the plurality of key points; wherein the plurality of key points represent key nodes of the skeleton of the target hand, and the first key point is any key point among the plurality of key points; The calculation unit is configured to determine the coordinate information of the first key point based on a preset classification diagram and a classification logic diagram corresponding to the first key point; The estimation unit is configured to obtain a posture estimation result of the target hand after determining the coordinate information of each of the plurality of key points, The determining of the coordinate information of the first key point based on a preset classification diagram and a classification logic diagram corresponding to the first key point includes: Based on the classification logic diagram corresponding to the first key point, determine the triple information of each grid in the preset classification diagram; Determine the coordinate information of the first key point according to the triple information of each grid in the preset classification map, The triplet information includes a grid probability value, a horizontal coordinate value and a vertical coordinate value of a grid center pixel. The step of determining the coordinate information of the first key point according to the triple information of each grid in the preset classification map includes: Regression calculation is performed based on the grid probability value corresponding to each grid in the preset classification map, the horizontal coordinate value and the vertical coordinate value of the grid center pixel to obtain the coordinate information of the first key point.

12. An electronic device, comprising a memory and a processor; wherein: The memory is used to store executable instructions that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 10 when running the executable instructions.

13. A computer storage medium, wherein: The computer storage medium stores a hand posture estimation program, and when the hand posture estimation program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images