Monocular dense keypoint estimation method based on convolutional network, medium and equipment

Abstract

The application provides a monocular dense key point estimation method based on a convolution network, a medium and equipment; wherein the method is: inputting an image of a gesture to be estimated into a dense gesture estimation network; the dense gesture estimation network comprises a two-dimensional coding module, a feature dimension increasing module and a three-dimensional decoding module; in the two-dimensional coding module, a backbone network extracts features from the image to obtain intermediate features; a two-dimensional decoder obtains sparse key points from the intermediate features through a spiral convolution module, and gradually restores the two-dimensional dense key points by using a coarse-grained to fine-grained method; the feature dimension increasing module obtains features representing three dimensions; in the three-dimensional decoding module, a separable spiral convolution module performs depth separable operation and point operation on the features representing three dimensions to obtain three-dimensional dense key points. The method can obtain dense key points of a gesture in an image, the dense key points contain more rich hand detail information, and are helpful to obtain more fine hand motion details and hand contour information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, more particularly, to a monocular dense key point estimation method based on a convolutional network, a medium and an apparatus. BACKGROUND

[0002] Gesture is an important medium for human to express their thoughts and emotions. It has a very wide application in the field of human-computer interaction, augmented reality, virtual reality and gesture authentication. Efficient and accurate hand pose estimation is a key step to achieve the above technologies, which will directly determine the continuity and convenience of user use and affect the user experience.

[0003] The current mainstream monocular gesture estimation methods are:

[0004] I. Gesture estimation method based on monocular depth image: this method sends the monocular depth image into the neural network model to estimate the position of each key point end-to-end. Since the depth map itself contains depth information, the three-dimensional coordinates estimated by this method are relatively accurate. For example, in the Chinese invention patent application "Three-dimensional gesture pose estimation method based on topological perception" (publication number: CN115690908A), a single depth image is sent into a feature extraction module to extract local image features and convert them into a feature sequence. Then, an encoder module is used to capture the global context information based on the feature sequence. Subsequently, a decoder module is used to determine the relationship between joint nodes. Finally, a topological perception regression module is used to extract semantic information of each layer to obtain three-dimensional joint coordinate information of the hand pose. However, the price of the depth camera is relatively high and the effective collection distance interval is small. The image outside the effective collection distance interval has many noise points, which will seriously affect the effect of three-dimensional gesture estimation. In addition, ultraviolet light will interfere with the image acquisition of the depth camera, which seriously limits the use scenario of this method.

[0005] II. Gesture estimation method based on monocular RGB image: This method inputs monocular RGB image into neural network model to estimate the position of each key point end-to-end. Since the cost of RGB camera is low and the use scene is basically unrestricted, the application scene of this method is relatively wide. For example, in the Chinese invention patent application "Monocular three-dimensional gesture estimation method based on full convolutional neural network" (publication number: CN114882524A), three-dimensional hand key point information is obtained by decoupling the hand scale information and the depth prediction process of the hand key point of the neural network. However, this method obtains sparse key points, which cannot be used for fine measurement of hand movement. In the Chinese invention patent application "Gesture posture recognition method based on hierarchical convolutional neural network" (publication number: CN113673313A), a convolutional neural network is used to convert an RGB image into a key point heat map and then estimate the position of each key point. However, using a key point heat map to obtain position information may cause key point jitter during the inference process of the model, which may cause devastating damage to downstream tasks such as augmented reality, virtual reality, and gesture authentication.

[0006] At present, both in academia and in industry, only sparse key points are estimated, and the sparse key points contain less hand detail information, which is not helpful for obtaining more detailed hand movement details and hand contour information. Therefore, how to accurately and efficiently estimate the two-dimensional and three-dimensional positions of hand dense key points from a single image has become a problem to be solved in the field of gesture estimation and downstream. SUMMARY

[0007] In order to overcome the defects and deficiencies in the prior art, the purpose of the present application is to provide a monocular dense key point estimation method based on a convolutional network, a medium and an equipment; the method can obtain dense key points of gestures in an image, and the dense key points contain more detailed hand information, which is helpful for obtaining more detailed hand movement details and hand contour information, and provides more development space for downstream fields such as human-computer interaction, augmented reality, virtual reality and gesture authentication.

[0008] In order to achieve the above purpose, the present application is implemented by the following technical scheme: a monocular dense key point estimation method based on a convolutional network, obtaining an image of a gesture to be estimated; then inputting the image into a dense gesture estimation network;

[0009] The dense gesture estimation network comprises a two-dimensional encoding module, a feature dimension increasing module and a three-dimensional decoding module;

[0010] The two-dimensional encoding module comprises a backbone network and a two-dimensional decoder; the backbone network extracts features from the image to obtain intermediate features F b ; the two-dimensional decoder first obtains intermediate features F bThe sparse key points are obtained, and a coarse-to-fine method is used to gradually restore the sparse key points to two-dimensional dense key points;

[0011] The feature dimension increasing module takes the intermediate feature F b and the result after splicing the two-dimensional dense key points as input, performs feature dimension increasing calculation, and obtains a feature representing three dimensions;

[0012] The three-dimensional decoding module is composed of a separable spiral convolution module; the separable spiral convolution module performs depth separable operation and point operation on the feature representing three dimensions to obtain three-dimensional dense key points.

[0013] Preferably, in the two-dimensional encoding module, the two-dimensional decoder includes N decoding layers; the two-dimensional decoder first restores the intermediate feature F b to obtain M sparse key points, and the M sparse key points form a preliminary outline of the hand; then the N decoding layers gradually restore the number of the M sparse key points to 778 two-dimensional dense key points through upsampling in sequence.

[0014] Preferably, the separable spiral convolution module of the three-dimensional decoding module first constructs a spiral neighborhood k_disk(v):

[0015] 0_ring(v)={v}

[0016]

[0017] k_disk(v)=∪ i＝0，...，k i_ring(v)

[0018] Wherein, v represents a vertex of the hand in a fully specified state; represents extraction of a neighborhood of the vertex v; ring() represents a spiral line; k represents an order of adjacent points; 0_ring(v) represents a spiral line of second-order adjacent points with v as a vertex; k_disk(v) represents a set of spiral lines of the first k-order adjacent points with v as a vertex;

[0019] For a vertex v, k_disk(v) S , k_disk(v) S represents the first S elements in the spiral neighborhood k_disk(v);

[0020] Then, depth separable operation is performed to obtain a feature

[0021]

[0022] Wherein, i represents a current depth value; W i dlearnable parameters representing the i-th layer depthwise separable operation; f v，，i representing the input i-th layer features about vertex v'; D represents the total depth value; f v′ representing the input features about vertex v' representing the representation of three-dimensional features;

[0023] performing point operation to obtain features

[0024]

[0025] wherein, W p representing the weight of point operation;

[0026] Then, the features and the features are passed through a fully connected layer to regress the coordinates of three-dimensional dense key points.

[0027] Preferably, the dense gesture estimation network refers to a trained dense gesture estimation network; in the training process, the loss function includes the loss of measuring three-dimensional dense key points the loss of measuring two-dimensional dense key points normal loss edge length loss three-dimensional consistency loss and two-dimensional consistency loss

[0028] the loss of measuring three-dimensional dense key points the loss of measuring two-dimensional dense key points normal loss edge length loss respectively:

[0029]

[0030]

[0031]

[0032] wherein, C, V represent the faces and vertices of the dense gesture estimation network respectively; L p representing the two-dimensional coordinates of the key point p; representing the unit normal vector of the face; * represents the true value;

[0033] According to the two-dimensional affine transformation and color jitter, two views view1 and view2 are derived, and the affine transformation between the two views view1 and view2 is marked as T 1→2 , and the relative rotation is marked as R 1→2 three-dimensional consistency loss and two-dimensional consistency loss respectively:

[0034]

[0035]

[0036] total loss function is:

[0037]

[0038] Preferably, before training the dense gesture estimation network, a dataset and dense annotations are constructed; the method of constructing the dataset and dense annotations comprises the following steps:

[0039] S1, fitting out the pose parameters P and contour parameters θ of the non-rigid deformable hand model; using the non-rigid deformable hand model to calculate 21 sparse key point coordinates K s of the samples in the dataset respectively; wherein J represents a sparse key point regressor, and MO represents a non-rigid deformable hand model;

[0040] Subsequently, the 21 sparse key point coordinates K s calculated are used to calculate the loss with the 21 sparse key point true values Y in the dataset, and the loss function Cost is: Then, gradient descent is performed on the loss function Cost to obtain the optimal pose parameters P and contour parameters θ.

[0041] S2, using the non-rigid deformable hand model to obtain the three-dimensional annotations of 778 two-dimensional dense key points according to the pose P and contour parameters θ obtained in S1.

[0042] S3, calculating the two-dimensional annotations of the 778 key points according to the three-dimensional annotations calculated in S2 and the re-projection formula ; wherein represents the coordinates in the pixel coordinate system, Z c represents the Z coordinates in the camera coordinate system, M represents the intrinsic parameters, A represents the extrinsic parameters, and represents the coordinates in the world coordinate system.

[0043] The dense gesture estimation network is trained using the samples in the dataset with three-dimensional annotations and two-dimensional annotations.

[0044] Preferably, before training the dense gesture estimation network, the samples in the dataset are also subjected to random image enhancement processing; the random image enhancement processing comprises one or more of the following operations:

[0045] In the HSV color domain, the saturation of the samples in the dataset is scaled by a random value in the range of 50% to 150%.

[0046] In the HSV color domain, the hue of the dataset sample is scaled by a random value in the range of 82% to 118%;

[0047] In the HSV color domain, the contrast of the dataset sample is scaled by a random value in the range of 50% to 150%;

[0048] In the HSV color domain, the brightness of the dataset sample is scaled by a random value in the range of 67% to 132%;

[0049] The maximum circumscribed rectangle is calculated based on the mask to obtain the ROI;

[0050] The dataset sample is randomly rotated by -45° to 45°.

[0051] A readable storage medium, wherein the storage medium stores a computer program, the computer program causes the processor to execute the above-mentioned monocular dense key point estimation method based on a convolutional network when the processor executes the computer program.

[0052] A computer device, comprising a processor and a memory for storing a program executable by the processor, and the processor implements the above-mentioned monocular dense key point estimation method based on a convolutional network when executing the program stored in the memory.

[0053] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0054] The present application can obtain dense key points of gestures in images, and the dense key points contain more detailed information of hands, which is helpful to obtain more detailed motion details and hand contour information of hands, and provides more development space for downstream fields such as human-computer interaction, augmented reality, virtual reality and gesture authentication.

[0055] The two-dimensional coding module of the present application adopts a two-dimensional dense key point estimation method from coarse granularity to fine granularity; the layer-by-layer regression not only accelerates the network convergence and achieves better estimation effect, but also attaches higher-level semantic information to each layer of key points;

[0056] The traditional spiral convolution module uses LSTM for feature extraction, which will slow down due to sequential processing; the present application samples the spiral neighborhood with a fixed size, and fuses the features by using a fully connected layer; the present application also separates the operation of each channel, further improving the calculation efficiency of the spiral convolution;

[0057] All existing public gesture estimation datasets only have two-dimensional / three-dimensional sparse key point annotations, and cannot provide supervision information for dense key point estimation networks; in order to solve the problem, the sparse key point coordinates are used as an optimization target to optimize the posture and contour parameters in a non-rigid deformable hand model from zero by using gradient descent; after the posture and contour parameters are obtained, the three-dimensional coordinates of the dense key points can be obtained, and the two-dimensional coordinates of the dense key points can be calculated by using camera parameters; the annotation lays a foundation for the generation of a dense key point estimation model. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a processing flow schematic diagram of the dense gesture estimation network of the present application;

[0059] Figure 2 is a processing flow schematic diagram of the two-dimensional encoding module of the present application;

[0060] Figure 3 is a processing flow schematic diagram of the feature dimension lifting module of the present application;

[0061] Figure 4 is a processing flow schematic diagram of the three-dimensional decoding module of the present application;

[0062] Figure 5 is a processing flow schematic diagram of the dense gesture estimation network training process of the present application;

[0063] Figure 6 is a three-dimensional annotation effect diagram of the two-dimensional dense key points of the present application;

[0064] Figure 7 is a two-dimensional key point annotation effect diagram of the present application. DETAILED DESCRIPTION

[0065] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0066] Embodiment one

[0067] The present embodiment is a monocular dense key point estimation method based on a convolutional network, and the monocular dense gesture estimation refers to obtaining two-dimensional and three-dimensional position information of dense key points from a single hand image. The method is: obtaining an image of a gesture to be estimated; then inputting the image into a dense gesture estimation network.

[0068] As shown in Figure 1 , the dense gesture estimation network comprises a two-dimensional encoding module, a feature dimension lifting module and a three-dimensional decoding module.

[0069] The two-dimensional encoding module comprises a backbone network and a two-dimensional decoder, as shown in Figure 2 ; the backbone network extracts features from the image to obtain intermediate features F bThe two-dimensional decoder first uses a spiral convolution module to extract intermediate features F. b After obtaining sparse keypoints, a coarse-to-fine approach is used to gradually restore the sparse keypoints to two-dimensional dense keypoints.

[0070] In the two-dimensional encoding module, the two-dimensional decoder includes N decoding layers; the two-dimensional decoder first uses a spiral convolution module to extract the intermediate feature F. b M sparse keypoints are obtained, and the M sparse keypoints form the initial outline of the hand; then N decoding layers successively upsample to gradually restore the number of M sparse keypoints to 778 two-dimensional dense keypoints.

[0071] For example, the intermediate features F extracted from the backbone network b First, a spiral convolution module is used to obtain 49 keypoints, which form the general outline of the hand. Then, upsampling is used to gradually reduce the number of keypoints from 49 to 98, then to 196, then to 392, and finally to 778 dense keypoints. This hierarchical regression attaches higher-level semantic information to the keypoints at each level. For example, the 49 keypoints in the first level form the overall outline of the hand, carrying structural semantic information; the 49 additional keypoints estimated in the second level make the bending of the hand more dynamic, as if grasping the pattern of joint bending, enriching the details of joint bending, and carrying joint-related semantic information; the 682 additional keypoints estimated in the third and fourth levels enrich the details of the entire hand, making the fingers look more rounded and the curves closer to the shape of a real hand, carrying detailed semantic information. The addition of high-level semantic information at each level accelerates network convergence and improves the accuracy of 2D gesture estimation.

[0072] The feature dimensionality enhancement module performs a dimensionality increase on intermediate features F b The result of concatenating two-dimensional dense keypoints is used as input to perform feature upscaling calculations, obtaining features representing three dimensions, such as... Figure 3 As shown.

[0073] The 3D decoding module consists of separable spiral convolution modules (DSConv), such as... Figure 4 As shown, the separable spiral convolution module performs depth-separable and point operations on the features representing 3D to obtain dense 3D keypoints.

[0074] The separable spiral convolution module of the 3D decoding module first constructs the spiral neighborhood k_disk(v):

[0075] 0_ring(v) = {v}

[0076]

[0077] k_disk(v)=∪ i＝0，...，ki_ring(v)

[0078] where v represents a vertex set by the hand in a fully specified state; represents the neighborhood extraction of vertex v; ring() represents a spiral line; k represents the order of adjacent points; 0_ring(v) represents the spiral line of the second-order adjacent points with v as the vertex; k_disk(v) represents the set of spiral lines of the first k-order adjacent points with v as the vertex;

[0079] a fixed-size spiral neighborhood is used, and features are fused by using a fully connected layer (FC)

[0080]

[0081] where [·] represents a connection function; k_disk(v) S represents the first S elements in the spiral neighborhood k_disk(v); W and b are learnable parameters; f v′ represents the input feature representing a three-dimensional space with respect to vertex v';

[0082] However, since the dimension of the fully connected layer is large, the present application separately operates each channel to reduce the parameter quantity and the calculation quantity; for a vertex v, k_disk(v) S is sampled;

[0083] Then, a depth separable operation is performed to obtain feature

[0084]

[0085] where i represents a current depth value; W i d represents a learnable parameter of the i-th layer of depth separable operation; f v′，i represents the feature input to the i-th layer with respect to vertex v'; D represents a total depth value; f v′ represents the input feature representing a three-dimensional space with respect to vertex v';

[0086] A point operation is performed to obtain feature

[0087]

[0088] where Wp represents the weight of the point operation;

[0089] Then, the feature and the feature are input to a fully connected layer to regress the coordinates of the three-dimensional dense key points.

[0090] The dense gesture estimation network refers to a trained dense gesture estimation network, as shown in Figure 5

[0091] Before training the dense gesture estimation network, a data set is constructed, and random image enhancement processing is performed on samples of the data set; due to the difficulty in obtaining three-dimensional gesture data GT, existing public data sets and self-built data sets cannot cover all scenes and light environments. Therefore, a data enhancement module is designed to increase the richness of the data and improve the generalization ability of the model.

[0092] The random image enhancement processing includes one or more of the following operations:

[0093] In the HSV color domain, the saturation of the data set sample is scaled by a random value in the range of 50% to 150%;

[0094] In the HSV color domain, the hue of the data set sample is scaled by a random value in the range of 82% to 118%;

[0095] In the HSV color domain, the contrast of the data set sample is scaled by a random value in the range of 50% to 150%;

[0096] In the HSV color domain, the brightness of the data set sample is scaled by a random value in the range of 67% to 132%;

[0097] Based on the mask, the maximum circumscribed rectangle is calculated to obtain the ROI, and compared with the previous target detection or key point based method, the ROI obtained by this method is closer to the hand;

[0098] The data set sample is randomly rotated by -45° to 45°.

[0099] The data set and the dense annotation are constructed, including the following steps:

[0100] S1, the pose parameters P and the contour parameters theta of the non-rigid deformable hand model (MANO) are fitted; the non-rigid deformable hand model is used to calculate 21 sparse key point coordinates K s = J (MO (P, theta)) of the samples in the data set; wherein J represents a sparse key point regressor, and MO represents a non-rigid deformable hand model;

[0101] Then, the 21 sparse key point coordinates K s calculated are used to calculate the loss with the 21 sparse key point true values Y in the data set, and the loss function Cost is: Then, the loss function Cost is subjected to gradient descent to obtain the optimal pose parameters P and contour parameters theta.

[0102] ​S2, using the non-rigid deformable hand model, the three-dimensional labels of 778 two-dimensional dense key points are obtained according to the pose P and the contour parameter theta obtained in S1, as shown in the following formula. Figure 6

[0103] S3, the two-dimensional labels of 778 key points are calculated according to the three-dimensional labels calculated in S2 and the re-projection formula , as shown in the following formula. Figure 7 , wherein represents the coordinates in the pixel coordinate system, Z c represents the Z coordinate in the camera coordinate system, M represents the intrinsic parameter, and A represents the extrinsic parameter,

[0104] The dense gesture estimation network is trained using samples with three-dimensional labels and two-dimensional labels in the data set.

[0105] The dense gesture estimation network training method is:

[0106] I. Network forward inference: the original image and the image after data enhancement are input into the dense three-dimensional gesture estimation network as a group, and the predicted dense key point two-dimensional coordinates and three-dimensional coordinates are obtained;

[0107] II. Loss calculation: the output of the network forward inference is input into the loss function to calculate the loss function value;

[0108] III. Gradient back propagation: calculate the gradient value of the loss function value with respect to the network parameters of the dense gesture estimation network, and update the network parameters using the back propagation algorithm;

[0109] IV. Repeat the above steps in batches for the input image until the loss function value no longer decreases, then the training of the dense gesture estimation network is completed.

[0110] In the training process, the loss function includes the loss of three-dimensional dense key points the loss of two-dimensional dense key points normal loss edge length loss three-dimensional consistency loss and two-dimensional consistency loss

[0111] The loss of three-dimensional dense key points the loss of two-dimensional dense key points normal loss edge length loss are respectively:

[0112] ​​

[0113]

[0114]

[0115] where C, V represent the face and vertex of the dense gesture estimation network respectively; L p represents the two-dimensional coordinates of the key point p; represents the unit normal vector of the face; * represents the true value;

[0116] Two views view1 and view2 are derived according to the two-dimensional affine transformation and color jittering, and the affine transformation between the two views view1 and view2 is marked as T 1→2 , and the relative rotation is marked as R 1→2 ; the three-dimensional consistency loss and the two-dimensional consistency loss are respectively:

[0117]

[0118]

[0119] The present application adopts L1 norm to measure the loss L 3D of three-dimensional dense key points and the loss L 2D of two-dimensional dense key points; normal loss L norm and edge length loss L edge are also used to measure the smoothness: inspired by self-supervised tasks, the present application designs three-dimensional consistency loss and two-dimensional consistency loss on the basis of data enhancement, which can supervise the gesture estimation task in two-dimensional and three-dimensional space. It should be noted that although T contains rotation, translation and scale in two-dimensional space, only R will affect the three-dimensional estimation result.

[0120] The total loss function is:

[0121]

[0122] Embodiment two

[0123] The present embodiment is a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program makes the processor execute the method of embodiment one when the processor executes the computer program.

[0124] Embodiment three

[0125] The computer device of the embodiment comprises a processor and a memory for storing a program executable by the processor, and the processor implements the method of the embodiment when executing the program stored in the memory.

[0126] The above embodiments are preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principle of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A monocular dense keypoint estimation method based on a convolutional network, characterized in that: An image of a gesture to be estimated is acquired; the image is then input to a dense gesture estimation network; The dense gesture estimation network comprises a two-dimensional encoding module, a feature dimension lifting module, and a three-dimensional decoding module; The two-dimensional coding module comprises a backbone network and a two-dimensional decoder; the backbone network extracts features of an image to obtain intermediate features F b ; The two-dimensional decoder first restores the intermediate feature F b from the sparse key points, and then gradually restores the two-dimensional dense key points from the sparse key points by using a coarse-to-fine method. The feature dimension increasing module takes the intermediate feature F b and the result after splicing the two-dimensional dense key points as input, performs feature dimension increasing calculation, and obtains a feature representing three dimensions. The three-dimensional decoding module is composed of a separable spiral convolution module; the separable spiral convolution module performs depth separable operation and point operation on features representing three dimensions to obtain three-dimensional dense key points; The separable spiral convolution module of the three-dimensional decoding module first constructs a spiral neighborhood k_disk(v): 0_ring(v)={v} k_disk(v) = U m＝0,…,k m_ring(v) where v represents a vertex set by the hand in a fully specified state; represents neighborhood extraction for vertex v; ring() represents a ring; k represents the order of the neighboring points; 0_ring(v) represents a ring of second-order neighboring points with v as the vertex; and k_disk(v) represents a set of rings of the first k-order neighboring points with v as the vertex. For a vertex v, sampling yields k_disk(v) S , k_disk(v) S represent the first S elements in the spiral neighborhood k_disk(v). Then, a depthwise separable operation is performed to obtain features where i represents the current depth value; W i d represents the learnable parameters of the depth separable operation of the i-th layer; f v′,i represents the input feature of the i-th layer with respect to the vertex v'; D represents the total depth value; f v′ represents the input feature of the i-th layer with respect to the vertex v'; D represents the total depth value; f Performing point operations to obtain features wherein W p represent the weights of the point operations; After that, the features and the features After a fully connected layer, the coordinates of the three-dimensional dense key points are regressed.

2. The convolutional network-based monocular dense keypoint estimation method according to claim 1, characterized in that: In the two-dimensional encoding module, the two-dimensional decoder comprises N decoding layers; the two-dimensional decoder firstly restores the intermediate feature F b M sparse key points are obtained, and the M sparse key points form a preliminary outline of the hand; then the N decoding layers gradually restore the number of the M sparse key points to 778 two-dimensional dense key points through upsampling.

3. The convolutional network-based monocular dense keypoint estimation method according to claim 1, characterized in that: The dense gesture estimation network refers to a trained dense gesture estimation network; in the training process, the loss function includes a loss of measuring three-dimensional dense key points a loss of measuring two-dimensional dense key points a normal loss an edge length loss a three-dimensional consistency loss and a two-dimensional consistency loss The loss of measuring three-dimensional dense key points The loss of measuring two-dimensional dense key points Normal loss Edge length loss Respectively: where C, V represent the faces and vertices of the dense gesture estimation network, respectively; L p represents the two-dimensional coordinates of the key point p; represents the unit normal vector of the face; * represents the true value; Two views, viewl and view2, are derived from the two-dimensional affine transformation and color dithering, and the affine transformation between the two views, viewl and view2, is marked as T 1→2 , and the relative rotation is marked as R 1→2 ; three-dimensional consistency loss and two-dimensional consistency loss are respectively: Total loss function is:

4. The convolutional network-based monocular dense keypoint estimation method according to claim 1, characterized in that: Before training the dense gesture estimation network, a data set and dense annotations are first constructed; the method of constructing the data set and the dense annotations comprises the following steps: S1, fitting out the posture parameter P and the contour parameter θ of the non-rigid deformable hand model; using the non-rigid deformable hand model to calculate 21 sparse key point coordinates K of the sample in the data set respectively s = J(MO(P, θ)); wherein J represents the sparse key point regressor, and MO represents the non-rigid deformable hand model; The calculated 21 sparse key point coordinates K s The loss is calculated with the 21 sparse key point true values Y in the dataset, and the loss function Cost is: Then, the loss function Cost is subjected to gradient descent, and the optimal pose parameter P and contour parameter θ are solved. S2, according to the pose P and the contour parameter θ obtained in S1, the three-dimensional annotations of 778 two-dimensional dense key points are obtained by using a non-rigid deformable hand model; S3, three-dimensional annotation and re-projection formula calculated according to S2 Calculate the two-dimensional annotation of 778 key points; wherein represent the coordinates in the pixel coordinate system, Z c represent the Z coordinates in the camera coordinate system, M represents the intrinsic parameters, and A represents the extrinsic parameters, represent the coordinates in the world coordinate system; The dense gesture estimation network is trained by using samples with three-dimensional annotations and two-dimensional annotations in the data set.

5. The convolutional network-based monocular dense keypoint estimation method according to claim 4, characterized in that: Before training the dense gesture estimation network, the data set samples are also subjected to random image enhancement processing; the random image enhancement processing comprises one or more of the following operations: In the HSV color domain, the saturation of the data set sample is scaled by a random value in the range of 50% to 150%; In the HSV color domain, the hue of the data set sample is scaled by a random value in the range of 82% to 118%; In the HSV color domain, the contrast of the data set sample is scaled by a random value in the range of 50% to 150%; In the HSV color domain, the brightness of the data set sample is scaled by a random value in the range of 67% to 132%; The maximum circumscribed rectangle is calculated based on the mask to obtain the ROI; The data set sample is randomly rotated by -45° to 45°.

6. A readable storage medium characterized by, The storage medium stores a computer program, and the computer program causes the processor to execute the monocular dense key point estimation method based on the convolution network in any one of claims 1-5 when executed by the processor.

7. A computer device comprising a processor and a memory for storing a processor executable program, characterized in that, The processor executes the program stored in the memory to implement the monocular dense key point estimation method based on the convolution network in any one of claims 1-5.

Images