Hand three-dimensional reconstruction method and device and electronic equipment

By acquiring multi-view hand images in virtual reality and using preset three-dimensional hand reconstruction network and interactive feature fusion method, the problem of insufficient accuracy in three-dimensional reconstruction of virtual hand is solved, and more accurate virtual hand movement is achieved control.

CN118918239APending Publication Date: 2024-11-08BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310505262.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the three-dimensional reconstruction of hands in virtual reality, the accuracy of hand image fusion at multiple perspectives is insufficient, resulting in insufficient motion control of virtual hands in virtual scenes.

Method used

By acquiring hand images acquired at least two perspectives, the initial hand 3D reconstruction results are determined based on the preset hand 3D reconstruction network, and a multi-view consistency fusion method is adopted, combining interactive features and cross-view attention algorithms, Update and fuse hand features to improve the accuracy of 3D reconstruction results.

Benefits of technology

The accuracy and consistency of the three-dimensional reconstruction results of the hand are improved, making the motion control of the virtual hand in the virtual scene more accurate and stable.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918239A_ABST
    Figure CN118918239A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a hand three-dimensional reconstruction method and device and electronic equipment. A specific embodiment of the method comprises the following steps: acquiring hand images collected under at least two visual angles; based on a preset hand three-dimensional reconstruction network, determining an initial hand three-dimensional reconstruction result corresponding to each hand image, the hand three-dimensional reconstruction result comprising a hand three-dimensional model and a hand key point; and fusing the initial hand three-dimensional reconstruction results corresponding to the hand images collected under the at least two visual angles to obtain a fused hand three-dimensional reconstruction result. According to the embodiment, multi-view consistency is used, so that the fused hand three-dimensional reconstruction result is more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device and electronic device for three-dimensional reconstruction of a hand. Background Art

[0002] With the continuous development of science and technology, virtual reality technology has also developed rapidly. For example, a virtual hand corresponding to the user can be presented in a virtual scene, and the user can also control the generated virtual hand to move in the virtual scene. In this way, a better virtual browsing experience can be brought to the user. Summary of the invention

[0003] This disclosure section is provided to introduce concepts in a brief form, which will be described in detail in the detailed description section below. This disclosure section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for 3D reconstruction of a hand, comprising: acquiring hand images captured from at least two perspectives; determining an initial 3D reconstruction result of the hand corresponding to each hand image based on a preset 3D reconstruction network of the hand, wherein the 3D reconstruction result of the hand includes a 3D model of the hand and hand key points; fusing the initial 3D reconstruction results of the hand corresponding to the hand images captured from at least two perspectives to obtain a fused 3D reconstruction result of the hand.

[0005] In a second aspect, an embodiment of the present disclosure provides a three-dimensional hand reconstruction device, comprising: an acquisition unit, used to acquire hand images captured from at least two perspectives; a determination unit, used to determine the initial three-dimensional hand reconstruction results corresponding to each hand image based on a preset three-dimensional hand reconstruction network, wherein the three-dimensional hand reconstruction results include a three-dimensional hand model and hand key points; a fusion unit, used to fuse the initial three-dimensional hand reconstruction results corresponding to the hand images captured from at least two perspectives to obtain a fused three-dimensional hand reconstruction result.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional hand reconstruction method as described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the three-dimensional hand reconstruction method as described in the first aspect.

[0008] The 3D hand reconstruction method, device and electronic device provided by the embodiments of the present disclosure obtain hand images collected from at least two viewing angles; then, based on a preset 3D hand reconstruction network, determine the initial 3D hand reconstruction results corresponding to each hand image; finally, fuse the initial 3D hand reconstruction results corresponding to the hand images collected from the at least two viewing angles to obtain the fused 3D hand reconstruction results. In this way, the initial 3D hand reconstruction results corresponding to the hand images collected from each viewing angle are fused, and the consistency of multiple viewing angles is used to make the fused 3D hand reconstruction results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 is a flow chart of an embodiment of a method for three-dimensional reconstruction of a hand according to the present disclosure;

[0011] Figure 2 is a flow chart of another embodiment of the hand 3D reconstruction method according to the present disclosure;

[0012] Figure 3 is a flow chart of an embodiment of determining hand features corresponding to a hand image in a three-dimensional hand reconstruction method according to the present disclosure;

[0013] Figure 4 is a flow chart of another embodiment of a three-dimensional hand reconstruction method according to the present disclosure;

[0014] Figure 5 is a schematic diagram of an application scenario of the hand 3D reconstruction method according to the present disclosure;

[0015] Figure 6 is a schematic structural diagram of an embodiment of a hand 3D reconstruction device according to the present disclosure;

[0016] Figure 7 is an exemplary system architecture diagram in which various embodiments of the present disclosure may be applied;

[0017] Figure 8 It is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0020] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] Please refer to Figure 1 , shows a process 100 of an embodiment of a method for 3D reconstruction of a hand according to the present disclosure. The method for 3D reconstruction of a hand comprises the following steps:

[0025] Step 101: Acquire hand images collected from at least two viewing angles.

[0026] In this embodiment, the execution subject of the hand 3D reconstruction method can obtain hand images collected under at least two viewing angles. Here, a multi-camera can be used to shoot the same hand to obtain hand images under at least two viewing angles.

[0027] Step 102: determining an initial 3D hand reconstruction result corresponding to each hand image based on a preset 3D hand reconstruction network.

[0028] In this embodiment, the execution subject can determine the initial hand 3D reconstruction result corresponding to each hand image based on a preset hand 3D reconstruction network. The hand 3D reconstruction result can include a hand 3D model and hand key points. The hand 3D reconstruction network can be used to characterize the correspondence between an image and the hand 3D reconstruction result corresponding to the image.

[0029] Here, for each hand image captured at the above-mentioned at least two perspectives, the above-mentioned execution entity can input the hand image into the above-mentioned hand 3D reconstruction network to obtain the hand 3D reconstruction result corresponding to the image, and the hand 3D reconstruction result output by the above-mentioned hand 3D reconstruction network can be used as the initial hand 3D reconstruction result.

[0030] Here, the above-mentioned hand 3D reconstruction network can be a network structure shared by each perspective, that is, the hand images collected from at least two input perspectives will pass through the hand 3D reconstruction network with the same weights and structure. This part of the network estimates the parameters of the hand from a single perspective, thereby realizing 3D reconstruction of the hand.

[0031] The above-mentioned hand 3D reconstruction network belongs to the category of parameter algorithms, which usually includes an encoder part, a decoder part and a MANO layer (orchestration network model). Specifically:

[0032] The encoder part can be a residual network (for example, ResNet50), which can contain 50 layers of convolution and can be normalized using an activation function (for example, a rectified linear unit function (ReLU activation function)) and BatchNorm (also known as Batch Normalization, which is a method of normalizing the input of the layer by re-centering and rescaling to make the training of the artificial neural network faster and more stable), and finally use global average normalization to output a global feature to the decoder.

[0033] The decoder part can be a fully connected network, which can include 2 layers of 2048 hidden units and can use the ReLU activation function again. The decoder can predict an N-dimensional (e.g., 10-dimensional) shape parameter vector and an M-dimensional (e.g., 48-dimensional) hand posture vector. These two types of parameter vectors can be reconstructed into a three-dimensional hand model through the MANO layer.

[0034] The MANO layer is used as the 3D representation output, and 3D reconstruction is achieved by predicting the MANO parameters.

[0035] Furthermore, the relationship between the reconstructed three-dimensional hand model (M) and the hand key points (P) can be calculated through the regression matrix (J) predefined in the MANO layer, for example, P=J*M, where * is matrix multiplication.

[0036] Step 103 , fusing the initial 3D hand reconstruction results corresponding to the hand images collected from at least two viewing angles to obtain a fused 3D hand reconstruction result.

[0037] In this embodiment, the execution entity may fuse the initial three-dimensional hand reconstruction results corresponding to the hand images collected from the at least two viewing angles to obtain a fused three-dimensional hand reconstruction result.

[0038] Specifically, the execution subject may fuse the three-dimensional hand models of various perspectives by using the averaging method, or may fuse the key points of the hand of various perspectives by using the averaging method.

[0039] The method provided by the above-mentioned embodiment of the present disclosure obtains hand images collected from at least two viewing angles; then, based on a preset hand 3D reconstruction network, determines the initial hand 3D reconstruction results corresponding to each hand image; finally, fuses the initial hand 3D reconstruction results corresponding to the hand images collected from the above-mentioned at least two viewing angles to obtain a fused hand 3D reconstruction result. In this way, the initial hand 3D reconstruction results corresponding to the hand images collected from various viewing angles are fused, and the consistency of multiple viewing angles is used to make the fused hand 3D reconstruction result more accurate.

[0040] Continue to refer Figure 2 , which shows a process 200 of another embodiment of a method for 3D reconstruction of a hand. The process 200 of the method for 3D reconstruction of a hand comprises the following steps:

[0041] Step 201: Acquire hand images collected from at least two viewing angles.

[0042] In this embodiment, step 201 may be performed in a manner similar to step 101 , and will not be described in detail herein.

[0043] Step 202: for each hand image in each hand image, determine the hand feature corresponding to the hand image based on a preset hand three-dimensional reconstruction network.

[0044] In this embodiment, for each hand image in each hand image collected at at least two viewing angles, the execution subject of the hand 3D reconstruction method can determine the hand features corresponding to the hand image based on a preset hand 3D reconstruction network. The above hand 3D reconstruction network can be used to characterize the correspondence between the image and the hand 3D reconstruction result corresponding to the image.

[0045] Specifically, the above-mentioned three-dimensional hand reconstruction network usually includes an encoder. After the above-mentioned execution entity can input the hand image into the above-mentioned three-dimensional hand reconstruction network, the encoder of the above-mentioned three-dimensional hand reconstruction network will output the hand features corresponding to the hand image.

[0046] Step 203: Using the interactive features, the hand features corresponding to the hand image are updated to obtain updated hand features corresponding to the hand image.

[0047] In this embodiment, the execution subject can use the interactive feature to update the hand feature corresponding to the hand image to obtain the updated hand feature corresponding to the hand image. The interactive feature is usually obtained by interacting with the hand features corresponding to each hand image.

[0048] Specifically, the execution entity may perform matrix addition on the hand feature corresponding to the hand image and the interaction feature to obtain an updated hand feature corresponding to the hand image.

[0049] Step 204 , based on the hand 3D reconstruction network and the updated hand features corresponding to the hand image, determine an initial hand 3D reconstruction result corresponding to the hand image.

[0050] In this embodiment, the execution subject may determine an initial three-dimensional hand reconstruction result corresponding to the hand image based on the three-dimensional hand reconstruction network and the updated hand features corresponding to the hand image.

[0051] Specifically, the above-mentioned execution entity can use the updated hand features corresponding to the hand image as the input of the above-mentioned hand 3D reconstruction network, regress the new hand posture vector through the fully connected layer of the above-mentioned hand 3D reconstruction network at each perspective, and obtain the updated hand 3D reconstruction result through the MANO layer as the initial hand 3D reconstruction result corresponding to the hand image.

[0052] Step 205 , fusing the initial 3D hand reconstruction results corresponding to the hand images collected from at least two viewing angles to obtain a fused 3D hand reconstruction result.

[0053] In this embodiment, step 205 may be performed in a manner similar to step 103 , and will not be described in detail herein.

[0054] from Figure 2 It can be seen that Figure 1Compared with the corresponding embodiment, the process 200 of the hand 3D reconstruction method in this embodiment embodies the steps of updating the hand features corresponding to the hand image by using the interactive features, and determining the initial hand 3D reconstruction result corresponding to the hand image based on the hand 3D reconstruction network and the updated hand features corresponding to the hand image. Therefore, the scheme described in this embodiment can interact the hand features corresponding to the hand image of each perspective to achieve the update of the hand features, thereby using the consistency of multiple perspectives to correct the hand features corresponding to the perspective, and further improve the accuracy of the initial hand 3D reconstruction result corresponding to the perspective.

[0055] In some optional implementations, the execution subject may determine the hand features corresponding to the hand image based on a preset hand 3D reconstruction network in the following manner:

[0056] The execution subject may input the hand image into a preset 3D hand reconstruction network to determine target information, and the target information may include at least one level of feature maps and hand key points. Here, the execution subject may input the hand image into a preset 3D hand reconstruction network to obtain the hand key points corresponding to the hand image. In addition, after the hand image is input into the 3D hand reconstruction network, the encoder of the 3D hand reconstruction network may output at least one level of feature maps.

[0057] As an example, four levels of feature maps can be output, and the sizes of the feature maps can be: (256*64*64), (512*32*32), (1024*16*16), (2048*8*8), respectively, denoted as H1, H2, H3, H4. Of course, in a specific implementation, the number of at least one level can be limited according to actual conditions, and the number of at least one level is not limited here, nor is the size of the feature map of each level limited.

[0058] Afterwards, the hand features corresponding to the hand image can be determined based on the above target information. As an example, the above hand key points can be input into a pre-trained hand feature extraction model to obtain the hand features corresponding to the hand image. The above hand feature extraction model can be used to characterize the correspondence between the hand key points corresponding to the image and the hand features corresponding to the image. By using the hand key points and at least one level of feature maps to determine the hand features corresponding to the hand image in this way, the hand features can be better aligned with the hand image. It can be understood that the obtained hand features can better express the hand image, which can also make it easier to perform feature interaction, so that after the feature interaction, the hand features corresponding to each hand image can be more accurate.

[0059] In some optional implementations, the target information includes a hand posture vector. The execution subject may determine the hand features corresponding to the hand image based on the target information in the following manner: the execution subject may determine the hand features corresponding to the hand image based on at least one of the first hand feature, the second hand feature, and the third hand feature. In other words, any one of the first hand feature, the second hand feature, and the third hand feature may be determined as the hand feature corresponding to the hand image; or any two of the first hand feature, the second hand feature, and the third hand feature may be used to determine the hand features corresponding to the hand image.

[0060] Here, the first hand feature can be obtained by encoding the hand posture vector and the coordinates of the hand key points. In the encoding process, x layers of fully connected layers can be used to convert into a high-dimensional graph feature (G1) of a certain dimension, and the high-dimensional graph feature (G1) can be understood as the first hand feature.

[0061] The second hand feature may be determined by processing the feature map of the target level using a predefined graph algorithm, where the feature map of the target level may be the feature map with the smallest size among the feature maps of at least one level. The execution subject may use a spatially aware initial graph construction (SAIGB) algorithm to convert the feature map of the target level into a high-dimensional graph feature (G2) of another dimension, and the high-dimensional graph feature (G2) may be understood as the second hand feature.

[0062] The third hand feature can be obtained by projecting the hand key points onto feature maps of other levels except the target level, and determining the features of the projection position points on each feature map. Here, the features of the projection position points on each feature map can be organized into a high-dimensional map feature (G3) of another dimension, and the high-dimensional map feature (G3) can be understood as the third hand feature.

[0063] In this way, at least one of the first hand feature, the second hand feature and the third hand feature can be used to determine the hand feature corresponding to the hand image, so that the hand feature can be determined more flexibly.

[0064] In some optional implementations, the above-mentioned interactive features can be determined in the following manner: the above-mentioned execution subject can interact with the hand features corresponding to each hand image based on a preset cross-view attention algorithm and / or a preset view sharing algorithm to obtain interactive features. The principle can be understood as fusing and comparing the features corresponding to multiple single-view images to achieve interactive updates of features. The above-mentioned cross-view attention algorithm captures global feature interactions by calculating the attention scores between hand nodes. The above-mentioned view sharing algorithm obtains features shared by all views by sharing features under multiple views.

[0065] In some optional implementations, the cross-view attention algorithm can determine the interaction features in the following manner: the execution subject can determine the attention scores between the hand key points corresponding to each hand image, and use the attention scores to interact the hand features corresponding to each hand image to obtain the interaction features. Specifically, the cross-view attention algorithm can be composed of x layers of transformer encoder (feature extractor encoder) layers, and the attention scores are calculated between all nodes of the hands of multiple views to capture the global feature interaction.

[0066] In some optional implementations, the perspective sharing algorithm can determine the interactive features in the following manner: the execution subject can use the maximum value pooling method to determine the features with the highest response of each hand key point at different perspectives; then, the features with the highest response of each hand key point at different perspectives can be used to interact with the hand features corresponding to each hand image to obtain the interactive features. Specifically, the perspective sharing algorithm can first extract the normalized features through an adaptive graph convolutional network (Adaptive-GCN), and then use the maximum value pooling to obtain the features with the highest response of each hand key point at different perspectives, and finally reshape the data to obtain the features shared by all perspectives.

[0067] Using the above two algorithms to determine the interaction features can make the interaction features integrate the features of multiple perspectives, making the determined interaction features more reasonable.

[0068] Further references Figure 3 , which shows a process 300 of an embodiment of determining hand features corresponding to a hand image in a hand 3D reconstruction method. The process 300 of determining hand features corresponding to a hand image includes the following steps:

[0069] Step 301: input the hand image into a preset hand 3D reconstruction network to determine target information.

[0070] In this embodiment, the executor of the hand three-dimensional reconstruction method can input the hand image into a preset hand three-dimensional reconstruction network to determine the target information, and the above target information may include at least one level of feature maps, hand key points and hand posture vectors.

[0071] Here, the above-mentioned execution entity can input the hand image into a preset hand three-dimensional reconstruction network to obtain the hand key points corresponding to the hand image. In addition, after inputting the hand image into the above-mentioned hand three-dimensional reconstruction network, the encoder of the above-mentioned hand three-dimensional reconstruction network can output at least one level of feature map.

[0072] As an example, four levels of feature maps can be output, and the sizes of the feature maps can be: (256*64*64), (512*32*32), (1024*16*16), (2048*8*8), respectively, denoted as H1, H2, H3, H4. Of course, in a specific implementation, the number of at least one level can be limited according to actual conditions. The number of at least one level is not limited here, nor is the size of the feature map of each level, nor is the specific level selected. Just make a reasonable selection based on the actual situation.

[0073] Step 302: Encode the hand posture vector and the coordinates of the hand key points to obtain a first hand feature.

[0074] In this embodiment, the above-mentioned execution entity can encode the hand posture vector and the coordinates of the hand key points determined in step 301. During the encoding process, x layers of fully connected layers can be used to convert them into high-dimensional graph features (G1) of a certain dimension. The high-dimensional graph features (G1) can be understood as the first hand feature.

[0075] Step 303: Process the feature graph of the target level using a predefined graph algorithm to determine a second hand feature.

[0076] In this embodiment, the execution subject may process the feature graph of the target level using a predefined graph algorithm to determine the second hand feature. Here, the feature graph of the target level may be the feature graph with the smallest size among the feature graphs of the at least one level.

[0077] As an example, if the sizes of the feature maps are (256*64*64), (512*32*32), (1024*16*16), (2048*8*8), respectively, they are denoted as H1, H2, H3, and H4. The first item in the above sizes is the resolution, and the last two items are the sizes. Since the size of feature map H4 (2048*8*8) is the smallest, feature map H4 can be determined as the feature map of the target level.

[0078] Specifically, the above-mentioned execution entity can use the spatially aware initial graph construction algorithm to convert the feature graph of the above-mentioned target level (for example, H4) into a high-dimensional graph feature (G2) of another dimension, and the high-dimensional graph feature (G2) can be understood as a second-hand feature.

[0079] Step 304, projecting the hand key points onto feature maps of other levels except the feature map of the target level, and determining the features of the projected position points on each feature map as the third hand feature.

[0080] In this embodiment, the above-mentioned execution entity can project the hand key points determined in step 301 onto feature maps of other levels (for example, H1, H2, H3) except the feature map of the above-mentioned target level, and organize the features of the projection position points on each feature map into a high-dimensional graph feature (G3) of another dimension. The high-dimensional graph feature (G3) can be understood as the third hand feature.

[0081] Step 305: concatenate the first hand feature, the second hand feature and the third hand feature to obtain the hand feature corresponding to the hand image.

[0082] In this embodiment, the above-mentioned execution entity can splice the first hand feature obtained in step 302, the second hand feature obtained in step 303, and the third hand feature obtained in step 304 along the channel dimension to organize them into a graph feature, which can be understood as the hand feature corresponding to the hand image.

[0083] The method provided by the above-mentioned embodiment of the present disclosure inputs the hand image into a preset hand three-dimensional reconstruction network to determine target information such as hand key points, feature maps of at least one level, and hand posture vectors, and uses the target information to determine three types of hand features in three ways. Finally, the three types of hand features are used to determine the hand features corresponding to the hand image. Since the first hand feature can reflect the prediction result of the single-view image with a pseudo-label added for the hand model, and the second hand feature and the third hand feature can reflect features of different scales (fine-grained and coarse-grained), therefore, the hand features are determined by using the first hand feature, the second hand feature, and the third hand feature, so that the determined hand features include more feature categories and numbers, thereby making it possible to better and more accurately implement feature interactive updates.

[0084] Continue to refer Figure 4 , which shows a process 400 of another embodiment of a method for 3D reconstruction of a hand. The process 400 of the method for 3D reconstruction of a hand comprises the following steps:

[0085] Step 401: Acquire hand images collected from at least two viewing angles.

[0086] Step 402: Determine the initial 3D hand reconstruction results corresponding to each hand image based on a preset 3D hand reconstruction network.

[0087] Step 403: fuse the initial 3D hand reconstruction results corresponding to the hand images collected from at least two viewing angles to obtain a fused 3D hand reconstruction result.

[0088] In this embodiment, steps 401-403 may be performed in a manner similar to steps 101-103, and will not be described in detail herein.

[0089] Step 404 , based on the fused 3D reconstruction result of the hand and the initial 3D reconstruction result of the hand, a loss value is determined using a preset loss function.

[0090] In this embodiment, the execution body of the hand 3D reconstruction method can determine the loss value based on the above-mentioned fused hand 3D reconstruction result and the above-mentioned initial hand 3D reconstruction result by using a preset loss function (for example, L1 norm).

[0091] Step 405: Use the loss value to adjust the network parameters of the hand 3D reconstruction network to obtain an adjusted hand 3D reconstruction network.

[0092] In this embodiment, the execution subject can use the loss value to adjust the network parameters of the hand 3D reconstruction network to obtain the adjusted hand 3D reconstruction network. Here, the training can be terminated when the preset training termination condition is met. For example, the preset training termination condition here may include but is not limited to at least one of the following: the training time exceeds the preset duration; the number of training times exceeds the preset number of times; the calculated difference is less than the preset difference threshold.

[0093] Here, various implementation methods can be used to adjust the network parameters of the above-mentioned hand 3D reconstruction network based on the above-mentioned loss value. For example, the BP (Back Propagation) algorithm or the SGD (Stochastic Gradient Descent) algorithm can be used to adjust the network parameters of the above-mentioned hand 3D reconstruction network. It should be noted that any algorithm for adjusting network parameters can be used to adjust the network parameters, and there is no limitation here.

[0094] from Figure 4 It can be seen that Figure 1Compared with the corresponding embodiment, the process 400 of the hand 3D reconstruction method in this embodiment embodies the steps of determining the loss value based on the fused hand 3D reconstruction result and the initial hand 3D reconstruction result using a preset loss function, and adjusting the network parameters of the hand 3D reconstruction network using the loss value to obtain the adjusted hand 3D reconstruction network. Therefore, the scheme described in this embodiment can adjust the network parameters of the hand 3D reconstruction network so that the hand 3D reconstruction network can more accurately perform 3D reconstruction of the hand.

[0095] In some optional implementations, the above-mentioned execution entity can determine the loss value based on the above-mentioned fused three-dimensional hand reconstruction result and the above-mentioned initial three-dimensional hand reconstruction result by using a preset loss function in the following manner: the above-mentioned execution entity can determine the difference between the above-mentioned fused three-dimensional hand reconstruction result and the above-mentioned initial three-dimensional hand reconstruction result as the loss value, thereby supervising the output result of the above-mentioned three-dimensional hand reconstruction network.

[0096] Here, if the above-mentioned initial 3D hand reconstruction result is the 3D hand reconstruction result output by the above-mentioned 3D hand reconstruction network, then this supervision method can be called multi-view consistency self-distillation supervision, and the corresponding loss function can be called multi-view consistency self-distillation loss function. Through this supervision method, the output result of the single-view network module can be made as close as possible to the fused result, thereby improving the accuracy of the output result of the single-view network module. If the above-mentioned initial 3D hand reconstruction result is the updated 3D hand reconstruction result, then this supervision method can be called multi-view fusion supervision, and the corresponding loss function can be called multi-view fusion supervision loss function. Through this supervision method, the output result of the cross-view interaction module can be made as close as possible to the fused result, thereby improving the accuracy of the output result of the cross-view interaction module.

[0097] In some optional implementations, the execution subject may determine the loss value based on the above-mentioned fused three-dimensional hand reconstruction result and the above-mentioned initial three-dimensional hand reconstruction result by using a preset loss function in the following manner: for each of the above-mentioned at least two perspectives, the execution subject may project the initial three-dimensional hand model corresponding to the perspective (usually the updated three-dimensional hand model) onto the hand image under the perspective to obtain a first set of projection points. Afterwards, the initial three-dimensional hand model (usually the updated three-dimensional hand model) corresponding to other perspectives other than the perspective may be rotated to the perspective according to the camera position relationship, and the rotated three-dimensional hand model may be projected onto the hand image under the perspective to obtain a second set of projection points; then, the difference between the above-mentioned first set of projection points and the above-mentioned second set of projection points may be determined as the loss value, thereby supervising the output result of the above-mentioned three-dimensional hand reconstruction network. This supervision method may be referred to as multi-perspective two-dimensional consistency supervision, and the corresponding loss function may be referred to as a multi-perspective two-dimensional consistency loss function.

[0098] As an example, if images are collected from three perspectives, for the first perspective of the three perspectives, the updated three-dimensional hand model corresponding to the first perspective is projected onto the hand image collected at the first perspective to obtain a first set of projection points; then, the updated three-dimensional hand models corresponding to other perspectives (i.e., the second perspective or the third perspective) except the first perspective are rotated to the first perspective according to the camera position relationship, and the rotated three-dimensional hand model is projected onto the hand image collected at the first perspective to obtain a second set of projection points; then, the difference between the first projection point set and the second projection point set is compared, and the difference is determined as the loss value.

[0099] Through this supervision method, the 3D hand models corresponding to all viewpoints are converted to the same viewpoint, and the differences between the projection points corresponding to each 3D hand model under the same viewpoint are compared. The output results of the 3D hand reconstruction network are supervised through the consistency between the viewpoints, making the output results of the 3D hand reconstruction network more accurate.

[0100] In some optional implementations, the execution subject may determine the loss value based on the fused 3D hand reconstruction result and the initial 3D hand reconstruction result by using a preset loss function in the following manner: the execution subject may use a predefined hand key point positioning algorithm (e.g., OpenPose algorithm) to identify the hand key points in the hand images collected from at least two perspectives as pseudo labels. Afterwards, the difference between the pseudo labels and the initial hand key points (the hand key points output by the 3D hand reconstruction network) may be used as the loss value to supervise the output result of the 3D hand reconstruction network. This supervision method may be referred to as pseudo label supervision, and the corresponding loss function may be referred to as a pseudo label loss function.

[0101] This supervision method can make the hand key points output by the network as close as possible to the pseudo-label, thereby constraining the output results of the single-view network module.

[0102] In some optional implementations, the execution subject may determine the loss value based on the fused 3D hand reconstruction result and the initial 3D hand reconstruction result by using a preset loss function in the following manner: the execution subject may obtain a preset 3D hand model reference template, and the 3D hand model reference template is a preset standard 3D hand model. Afterwards, the difference between the 3D hand model reference template and the initial 3D hand model (the 3D hand model output by the 3D hand reconstruction network) may be used as the loss value to supervise the output result of the 3D hand reconstruction network. This supervision method may be referred to as parameter prior supervision, and the corresponding loss function may be referred to as parameter prior loss function.

[0103] This supervision method can make the 3D hand model output by the network as close as possible to the average gesture, thereby further constraining the output results of the single-view network module.

[0104] Continue to see Figure 5 , Figure 5 FIG. 1 is a schematic diagram of an application scenario of the hand 3D reconstruction method according to this embodiment. Figure 5 In the application scenario, multiple single-view images input will pass through the single-view network module, the cross-view interaction module and the multi-view consistency supervision module.

[0105] In the single-view network module, the single-view images collected from multiple perspectives are input into the single-view network (i.e., the hand 3D reconstruction network), and the hand 3D model and key points (i.e., the hand 3D reconstruction result) can be output. Here, the single-view network can include an encoder, a decoder, and a MANO layer.

[0106] In the cross-view interaction module, the hand features of each single-view image can be first extracted. Here, three types of hand features (i.e., first-class features, second-class features, and third-class features) can be extracted, and these three types of hand features can be spliced ​​to obtain the graph features of the single-view image. After that, the graph features of each single-view image can be fused to obtain multi-view graph features. Then, the cross-view attention branch and the view sharing feature branch can be used to interact with the multi-view graph features to obtain interactive features. For each view, the interactive features are used to update the graph features corresponding to the view to obtain updated features. Then, the updated features can be input into the encoder and MANO layer of the hand 3D reconstruction network for parameter regression to obtain the interactive 3D hand model and key points.

[0107] In the multi-view consistency supervision module, the interactive 3D reconstruction results of the hand corresponding to the images collected from multiple views can be fused to obtain the fused 3D hand model and key points. Here, the output results of the hand 3D reconstruction network can be supervised using the multi-view consistency self-distillation supervision method, the multi-view 2D consistency supervision method, and the multi-view fusion supervision method.

[0108] Specifically, a multi-view consistency self-distillation supervision method is used to determine the difference between the fused 3D reconstruction result of the hand and the 3D reconstruction result of the hand output by the single-view network, thereby supervising the output result of the single-view network. A multi-view fusion supervision method is used to determine the difference between the fused 3D reconstruction result of the hand and the 3D reconstruction result of the hand after interaction, thereby supervising the output result of the single-view network. A multi-view two-dimensional consistency supervision method is used to determine the difference between the projection points of the 3D hand model corresponding to a certain perspective and other perspectives, thereby supervising the output result of the single-view network.

[0109] In addition, pseudo-label supervision and parameter prior supervision can be used to supervise the output results of the single-view network module.

[0110] It should be noted that, during the data input stage, hand images taken from multiple perspectives can be collected, the hand area in each hand image can be detected and extracted, the extracted hand image can be scaled to the target pixel size (for example, 256 pixels) as the input image, and the input image can be input into an existing hand key point positioning algorithm (for example, OpenPose) to obtain a two-dimensional pseudo-label.

[0111] The network training phase can be divided into two sub-phases, namely the first sub-phase and the second sub-phase. In the first sub-phase, the single-view network module and the cross-view interaction module are trained independently for each view. During the initial training process, the projection result of the three-dimensional hand result output by the network onto the two-dimensional image is close to the two-dimensional pseudo-label. At this time, the constraints of multi-view consistency self-distillation supervision, multi-view two-dimensional consistency supervision, and multi-view fusion supervision are not used. In the second sub-phase, the constraints of multi-view consistency self-distillation supervision, multi-view two-dimensional consistency supervision, and multi-view fusion supervision are added. Finally, the results of multiple views are fused to output the fused three-dimensional hand model and hand key points.

[0112] In the network testing phase, the trained network supports two testing modes, including single image testing and multi-image testing. For single image testing, a single hand image is input, and only the single-view network module is required for hand estimation, and the 3D hand model and key points are output. For multi-image testing, multiple hand images are input, and each view passes through the single-view network module, and then passes through the cross-view interaction module and the multi-view consistency supervision module in turn to obtain a multi-view consistent 3D hand model and key points.

[0113] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a hand three-dimensional reconstruction device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0114] like Figure 6 As shown, the hand 3D reconstruction device 600 of this embodiment includes: an acquisition unit 601, a determination unit 602 and a fusion unit 603. The acquisition unit 601 is used to acquire hand images collected from at least two viewing angles; the determination unit 602 is used to determine the initial hand 3D reconstruction results corresponding to each hand image based on a preset hand 3D reconstruction network, wherein the hand 3D reconstruction results include a hand 3D model and hand key points; the fusion unit 603 is used to fuse the initial hand 3D reconstruction results corresponding to the hand images collected from at least two viewing angles to obtain the fused hand 3D reconstruction results.

[0115] In this embodiment, the specific processing of the acquisition unit 601, the determination unit 602 and the fusion unit 603 of the hand 3D reconstruction device 600 can be referred to. Figure 1 This corresponds to step 101, step 102 and step 103 in the embodiment.

[0116] In some optional implementations, the determination unit 602 may be further used to determine an initial 3D hand reconstruction result corresponding to each hand image based on a preset 3D hand reconstruction network in the following manner: for each hand image in each hand image, determine the hand features corresponding to the hand image based on a preset 3D hand reconstruction network, update the hand features corresponding to the hand image using interactive features to obtain updated hand features corresponding to the hand image, and determine the initial 3D hand reconstruction result corresponding to the hand image based on the 3D hand reconstruction network and the updated hand features corresponding to the hand image, wherein the interactive features are obtained by interacting with the hand features corresponding to each hand image.

[0117] In some optional implementations, the determination unit 602 may be further used to determine the hand features corresponding to the hand image based on a preset three-dimensional hand reconstruction network in the following manner: inputting the hand image into a preset three-dimensional hand reconstruction network to determine target information, wherein the target information includes at least one level of feature maps and hand key points; and determining the hand features corresponding to the hand image based on the target information.

[0118] In some optional implementations, the target information includes a hand posture vector; and the determination unit 602 may be further used to determine the hand features corresponding to the hand image based on the target information in the following manner: based on at least one of the first hand feature, the second hand feature and the third hand feature, the hand features corresponding to the hand image are determined; and the first hand feature is obtained by encoding the hand posture vector and the coordinates of the hand key points; the second hand feature is determined by processing the feature map of the target level using a predefined graph algorithm; the third hand feature is obtained by projecting the hand key points onto feature maps of other levels except the feature map of the target level, and determining the features of the projection position points on each feature map. .

[0119] In some optional implementations, the above-mentioned determination unit 602 can be further used to determine the hand feature corresponding to the hand image based on at least one of the first hand feature, the second hand feature and the third hand feature in the following manner: splicing the first hand feature, the above-mentioned second hand feature and the above-mentioned third hand feature to obtain the hand feature corresponding to the hand image.

[0120] In some optional implementations, the above-mentioned interaction features are determined in the following manner: based on a preset cross-view attention algorithm and / or a preset view sharing algorithm, the hand features corresponding to each hand image are interacted to obtain the interaction features.

[0121] In some optional implementations, the cross-view attention algorithm determines the interaction features in the following manner: determining the attention scores between the hand key points corresponding to each hand image, and using the attention scores to interact the hand features corresponding to each hand image to obtain the interaction features.

[0122] In some optional implementations, the perspective sharing algorithm determines the interaction features in the following manner: using the maximum pooling method to determine the features with the highest response of each hand key point at different perspectives; using the features with the highest response of each hand key point at different perspectives, the hand features corresponding to each hand image are interacted to obtain the interaction features.

[0123] In some optional implementations, the hand 3D reconstruction device 600 may further include: a loss value determination unit (not shown in the figure) and an adjustment unit (not shown in the figure). The loss value determination unit is used to determine the loss value based on the fused hand 3D reconstruction result and the initial hand 3D reconstruction result using a preset loss function; the adjustment unit is used to adjust the network parameters of the hand 3D reconstruction network using the loss value to obtain an adjusted hand 3D reconstruction network.

[0124] In some optional implementations, the loss value determination unit may be further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result by using a preset loss function in the following manner: the difference between the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result is determined as the loss value.

[0125] In some optional implementations, the loss value determination unit can be further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result by using a preset loss function in the following manner: for each of the at least two perspectives, the initial three-dimensional hand model corresponding to the perspective is projected onto the hand image under the perspective to obtain a first set of projection points, and the initial three-dimensional hand models corresponding to other perspectives except the perspective are rotated to the perspective, and the rotated three-dimensional hand models are projected onto the hand image under the perspective to obtain a second set of projection points, and the difference between the first set of projection points and the second set of projection points is determined as the loss value.

[0126] In some optional implementations, the loss value determination unit can be further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result using a preset loss function in the following manner: using a predefined hand key point positioning algorithm to identify the hand key points in the hand images captured from the at least two perspectives as pseudo labels; and taking the difference between the pseudo labels and the initial hand key points as the loss value.

[0127] In some optional implementations, the loss value determination unit can be further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result by using a preset loss function in the following manner: obtaining a preset three-dimensional hand model reference template; and taking the difference between the three-dimensional hand model reference template and the initial three-dimensional hand model as the loss value.

[0128] Figure 7 An exemplary system architecture 700 is shown to which an embodiment of the disclosed three-dimensional hand reconstruction method can be applied.

[0129] like Figure 7 As shown, the system architecture 700 may include terminal devices 7011, 7012, 7013, a network 702, and a server 703. The network 702 is used to provide a medium for communication links between the terminal devices 7011, 7012, 7013 and the server 703. The network 702 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0130] The user can use the terminal devices 7011, 7012, 7013 to interact with the server 703 through the network 702 to send or receive messages, etc. For example, the server 703 can receive hand images sent by the terminal devices 7011, 7012, 7013. Various communication client applications can be installed on the terminal devices 7011, 7012, 7013, such as image processing applications, image shooting applications, instant messaging software, etc.

[0131] Terminal devices 7011, 7012, 7013 can be hardware or software. When terminal devices 7011, 7012, 7013 are hardware, they can be various electronic devices with display screens and supporting information interaction, including but not limited to smart cameras, smart phones, tablet computers, laptop portable computers, etc. When terminal devices 7011, 7012, 7013 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0132] The server 703 may be a server that provides various services. For example, it may be a background server that processes hand images at various viewing angles. The server 703 may first obtain hand images collected at at least two viewing angles from the terminal devices 7011, 7012, and 7013; then, based on a preset hand 3D reconstruction network, determine the initial hand 3D reconstruction results corresponding to each hand image; then, fuse the initial hand 3D reconstruction results corresponding to the hand images collected at the at least two viewing angles to obtain a fused hand 3D reconstruction result.

[0133] It should be noted that the server 703 can be hardware or software. When the server 703 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 703 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0134] It should also be noted that the three-dimensional hand reconstruction method provided in the embodiment of the present disclosure is usually executed by the server 703 , and the three-dimensional hand reconstruction device is usually set in the server 703 .

[0135] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0136] Reference below Figure 8 , which shows an electronic device (eg, Figure 7 A schematic diagram of the structure of the server in FIG. 800. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0137] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0138] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0139] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0140] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains hand images collected from at least two viewing angles; determines the initial hand 3D reconstruction results corresponding to each hand image based on a preset hand 3D reconstruction network, wherein the hand 3D reconstruction results include a hand 3D model and hand key points; fuses the initial hand 3D reconstruction results corresponding to the hand images collected from the at least two viewing angles to obtain a fused hand 3D reconstruction result.

[0141] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] According to one or more embodiments of the present disclosure, a method for 3D reconstruction of a hand is provided, the method comprising: acquiring hand images captured from at least two perspectives; determining initial 3D reconstruction results of the hand corresponding to each hand image based on a preset 3D reconstruction network of the hand, wherein the 3D reconstruction results of the hand include a 3D model of the hand and key points of the hand; and fusing the initial 3D reconstruction results of the hand corresponding to the hand images captured from at least two perspectives to obtain a fused 3D reconstruction result of the hand.

[0144] According to one or more embodiments of the present disclosure, an initial 3D hand reconstruction result corresponding to each hand image is determined based on a preset 3D hand reconstruction network, including: for each hand image in each hand image, based on the preset 3D hand reconstruction network, a hand feature corresponding to the hand image is determined, the hand feature corresponding to the hand image is updated using an interactive feature to obtain an updated hand feature corresponding to the hand image, and the initial 3D hand reconstruction result corresponding to the hand image is determined based on the 3D hand reconstruction network and the updated hand feature corresponding to the hand image, wherein the interactive feature is obtained by interacting with the hand features corresponding to each hand image.

[0145] According to one or more embodiments of the present disclosure, based on a preset three-dimensional hand reconstruction network, determining hand features corresponding to the hand image includes: inputting the hand image into a preset three-dimensional hand reconstruction network to determine target information, wherein the target information includes at least one level of feature maps and hand key points; based on the target information, determining hand features corresponding to the hand image.

[0146] According to one or more embodiments of the present disclosure, the target information includes a hand posture vector; and based on the target information, the hand features corresponding to the hand image are determined, including: based on at least one of the first hand feature, the second hand feature and the third hand feature, the hand features corresponding to the hand image are determined; and the first hand feature is obtained by encoding the hand posture vector and the coordinates of the hand key points; the second hand feature is determined by processing the feature map of the target level using a predefined graph algorithm; the third hand feature is obtained by projecting the hand key points onto feature maps of other levels except the feature map of the target level, and determining the features of the projection position points on each feature map.

[0147] According to one or more embodiments of the present disclosure, based on at least one of the first hand feature, the second hand feature and the third hand feature, determining the hand feature corresponding to the hand image includes: splicing the first hand feature, the second hand feature and the third hand feature to obtain the hand feature corresponding to the hand image.

[0148] According to one or more embodiments of the present disclosure, the interaction feature is determined in the following manner: based on a preset cross-view attention algorithm and / or a preset view sharing algorithm, the hand features corresponding to each hand image are interacted to obtain the interaction feature.

[0149] According to one or more embodiments of the present disclosure, the cross-view attention algorithm determines the interaction features in the following manner: determining the attention scores between the hand key points corresponding to each hand image, and using the attention scores to interact the hand features corresponding to each hand image to obtain the interaction features.

[0150] According to one or more embodiments of the present disclosure, the perspective sharing algorithm determines the interaction features in the following manner: using the maximum pooling method to determine the features with the highest response of each hand key point at different perspectives; using the features with the highest response of each hand key point at different perspectives, the hand features corresponding to each hand image are interacted to obtain the interaction features.

[0151] According to one or more embodiments of the present disclosure, the method further includes: determining a loss value using a preset loss function based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result; and adjusting the network parameters of the three-dimensional hand reconstruction network using the loss value to obtain an adjusted three-dimensional hand reconstruction network.

[0152] According to one or more embodiments of the present disclosure, based on the fused 3D reconstruction result of the hand and the initial 3D reconstruction result of the hand, a loss value is determined using a preset loss function, including: determining the difference between the fused 3D reconstruction result of the hand and the initial 3D reconstruction result of the hand as the loss value.

[0153] According to one or more embodiments of the present disclosure, based on the fused 3D hand reconstruction result and the initial 3D hand reconstruction result, a preset loss function is used to determine the loss value, including: for each of at least two perspectives, projecting the initial 3D hand model corresponding to the perspective onto the hand image under the perspective to obtain a first set of projection points, and rotating the initial 3D hand models corresponding to other perspectives except the perspective into the perspective, and projecting the rotated 3D hand models onto the hand image under the perspective to obtain a second set of projection points, and determining the difference between the first set of projection points and the second set of projection points as the loss value.

[0154] According to one or more embodiments of the present disclosure, based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result, a preset loss function is used to determine the loss value, including: using a predefined hand key point positioning algorithm to identify hand key points in hand images captured from at least two perspectives as pseudo labels; and taking the difference between the pseudo label and the initial hand key point as the loss value.

[0155] According to one or more embodiments of the present disclosure, based on the fused 3D hand reconstruction result and the initial 3D hand reconstruction result, a loss value is determined using a preset loss function, including: obtaining a preset 3D hand model reference template; and taking the difference between the 3D hand model reference template and the initial 3D hand model as the loss value.

[0156] According to one or more embodiments of the present disclosure, a three-dimensional hand reconstruction device is provided, and the device includes: an acquisition unit, used to acquire hand images collected from at least two perspectives; a determination unit, used to determine the initial three-dimensional hand reconstruction results corresponding to each hand image based on a preset three-dimensional hand reconstruction network, wherein the three-dimensional hand reconstruction results include a three-dimensional hand model and hand key points; and a fusion unit, used to fuse the initial three-dimensional hand reconstruction results corresponding to the hand images collected from at least two perspectives to obtain a fused three-dimensional hand reconstruction result.

[0157] According to one or more embodiments of the present disclosure, the determination unit is further used to determine the initial 3D hand reconstruction results corresponding to each hand image based on a preset 3D hand reconstruction network in the following manner: for each hand image in each hand image, based on the preset 3D hand reconstruction network, determine the hand features corresponding to the hand image, use the interactive features to update the hand features corresponding to the hand image to obtain the updated hand features corresponding to the hand image, and determine the initial 3D hand reconstruction results corresponding to the hand image based on the 3D hand reconstruction network and the updated hand features corresponding to the hand image, wherein the interactive features are obtained by interacting with the hand features corresponding to each hand image.

[0158] According to one or more embodiments of the present disclosure, the determination unit is further used to determine the hand features corresponding to the hand image based on a preset hand three-dimensional reconstruction network in the following manner: input the hand image into a preset hand three-dimensional reconstruction network to determine target information, wherein the target information includes at least one level of feature maps and hand key points; based on the target information, determine the hand features corresponding to the hand image.

[0159] According to one or more embodiments of the present disclosure, the target information includes a hand posture vector; and the determination unit is further used to determine the hand features corresponding to the hand image based on the target information in the following manner: based on at least one of the first hand feature, the second hand feature and the third hand feature, determine the hand features corresponding to the hand image; and the first hand feature is obtained by encoding the hand posture vector and the coordinates of the hand key points; the second hand feature is determined by processing the feature map of the target level using a predefined graph algorithm; the third hand feature is obtained by projecting the hand key points onto feature maps of other levels except the feature map of the target level, and determining the features of the projection position points on each feature map.

[0160] According to one or more embodiments of the present disclosure, the determination unit is further used to determine the hand feature corresponding to the hand image based on at least one of the first hand feature, the second hand feature and the third hand feature in the following manner: the first hand feature, the second hand feature and the third hand feature are spliced ​​to obtain the hand feature corresponding to the hand image.

[0161] According to one or more embodiments of the present disclosure, the interaction feature is determined in the following manner: based on a preset cross-view attention algorithm and / or a preset view sharing algorithm, the hand features corresponding to each hand image are interacted to obtain the interaction feature.

[0162] According to one or more embodiments of the present disclosure, the cross-view attention algorithm determines the interaction features in the following manner: determining the attention scores between the hand key points corresponding to each hand image, and using the attention scores to interact the hand features corresponding to each hand image to obtain the interaction features.

[0163] According to one or more embodiments of the present disclosure, the perspective sharing algorithm determines the interaction features in the following manner: using the maximum pooling method to determine the features with the highest response of each hand key point at different perspectives; using the features with the highest response of each hand key point at different perspectives, the hand features corresponding to each hand image are interacted to obtain the interaction features.

[0164] According to one or more embodiments of the present disclosure, the hand 3D reconstruction device includes a loss value determination unit and an adjustment unit. The loss value determination unit is used to determine the loss value based on the fused hand 3D reconstruction result and the initial hand 3D reconstruction result using a preset loss function; the adjustment unit is used to adjust the network parameters of the hand 3D reconstruction network using the loss value to obtain an adjusted hand 3D reconstruction network.

[0165] According to one or more embodiments of the present disclosure, the loss value determination unit is further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result by using a preset loss function in the following manner: the difference between the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result is determined as the loss value.

[0166] According to one or more embodiments of the present disclosure, the loss value determination unit is further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result by using a preset loss function in the following manner: for each of at least two perspectives, the initial three-dimensional hand model corresponding to the perspective is projected onto the hand image under the perspective to obtain a first set of projection points, and the initial three-dimensional hand models corresponding to other perspectives except the perspective are rotated into the perspective, and the rotated three-dimensional hand models are projected onto the hand image under the perspective to obtain a second set of projection points, and the difference between the first set of projection points and the second set of projection points is determined as the loss value.

[0167] According to one or more embodiments of the present disclosure, the loss value determination unit is further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result using a preset loss function in the following manner: using a predefined hand key point positioning algorithm to identify hand key points in hand images captured from at least two perspectives as pseudo labels; and taking the difference between the pseudo label and the initial hand key point as the loss value.

[0168] According to one or more embodiments of the present disclosure, the loss value determination unit is further used to determine the loss value based on the fused three-dimensional hand reconstruction result and the initial three-dimensional hand reconstruction result using a preset loss function in the following manner: obtaining a preset three-dimensional hand model reference template; and taking the difference between the three-dimensional hand model reference template and the initial three-dimensional hand model as the loss value.

[0169] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, a determination unit, and a fusion unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the acquisition unit may also be described as "a unit for acquiring hand images collected from at least two viewing angles".

[0170] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A method for 3D reconstruction of a hand, It is characterized in that include: Acquire images of a hand captured at at least two viewing angles; Based on a preset hand 3D reconstruction network, determine an initial hand 3D reconstruction result corresponding to each hand image, wherein the hand 3D reconstruction result includes a hand 3D model and hand key points; The initial three-dimensional hand reconstruction results corresponding to the hand images collected from the at least two viewing angles are fused to obtain a fused three-dimensional hand reconstruction result.

2. The method according to claim 1, It is characterized in that The determining of the initial hand 3D reconstruction results corresponding to each hand image based on the preset hand 3D reconstruction network includes: For each hand image in each hand image, based on a preset hand 3D reconstruction network, the hand features corresponding to the hand image are determined, and the hand features corresponding to the hand image are updated using the interactive features to obtain the updated hand features corresponding to the hand image, and based on the hand 3D reconstruction network and the updated hand features corresponding to the hand image, the initial hand 3D reconstruction result corresponding to the hand image is determined, wherein the interactive features are obtained through interaction between the hand features corresponding to each hand image.

3. The method according to claim 2, It is characterized in that The determining of the hand features corresponding to the hand image based on a preset hand 3D reconstruction network includes: Inputting the hand image into a preset hand 3D reconstruction network to determine target information, wherein the target information includes at least one level of feature maps and hand key points; Based on the target information, hand features corresponding to the hand image are determined.

4. The method according to claim 3, It is characterized in that The target information includes a hand posture vector; and The determining, based on the target information, a hand feature corresponding to the hand image includes: Determining a hand feature corresponding to the hand image based on at least one of the first hand feature, the second hand feature, and the third hand feature; and The first hand feature is obtained by encoding the hand posture vector and the coordinates of the hand key points; The second hand feature is determined by processing a feature map of the target level using a predefined map algorithm; The third hand feature is obtained by projecting the hand key point onto feature maps of other levels except the feature map of the target level, and by determining the features of the projection position points on each feature map.

5. The method according to claim 4, It is characterized in that The determining, based on at least one of the first hand feature, the second hand feature, and the third hand feature, a hand feature corresponding to the hand image includes: The first hand feature, the second hand feature and the third hand feature are concatenated to obtain a hand feature corresponding to the hand image.

6. The method according to claim 2, It is characterized in that The interaction characteristics are determined as follows: Based on a preset cross-view attention algorithm and / or a preset view sharing algorithm, hand features corresponding to each hand image are interacted to obtain interactive features.

7. The method according to claim 6, It is characterized in that The cross-view attention algorithm determines the interaction features in the following way: Attention scores between hand key points corresponding to each hand image are determined, and hand features corresponding to each hand image are interacted using the attention scores to obtain interaction features.

8. The method according to claim 6, It is characterized in that The perspective sharing algorithm determines the interaction characteristics in the following way: Use the maximum pooling method to determine the features with the highest response at each hand key point under different viewing angles; By utilizing the features with the highest response of each hand key point under different viewing angles, hand features corresponding to each hand image are interacted to obtain interactive features.

9. The method according to claim 1, It is characterized in that The method further comprises: Based on the fused three-dimensional reconstruction result of the hand and the initial three-dimensional reconstruction result of the hand, a loss value is determined using a preset loss function; The loss value is used to adjust the network parameters of the hand 3D reconstruction network to obtain an adjusted hand 3D reconstruction network.

10. The method according to claim 9, It is characterized in that The determining of the loss value by using a preset loss function based on the fused three-dimensional reconstruction result of the hand and the initial three-dimensional reconstruction result of the hand includes: The difference between the fused three-dimensional reconstruction result of the hand and the initial three-dimensional reconstruction result of the hand is determined as a loss value.

11. The method according to claim 9, It is characterized in that The determining of the loss value by using a preset loss function based on the fused three-dimensional reconstruction result of the hand and the initial three-dimensional reconstruction result of the hand includes: For each of the at least two perspectives, the initial three-dimensional hand model corresponding to the perspective is projected onto the hand image under the perspective to obtain a first set of projection points, and the initial three-dimensional hand models corresponding to other perspectives except the perspective are rotated into the perspective, and the rotated three-dimensional hand models are projected onto the hand image under the perspective to obtain a second set of projection points, and the difference between the first set of projection points and the second set of projection points is determined as a loss value.

12. The method according to claim 9, It is characterized in that The determining of the loss value by using a preset loss function based on the fused three-dimensional reconstruction result of the hand and the initial three-dimensional reconstruction result of the hand includes: Using a predefined hand key point positioning algorithm to identify hand key points in the hand images collected from the at least two viewing angles as pseudo labels; The difference between the pseudo-label and the initial hand keypoints is taken as the loss value.

13. The method according to claim 9, It is characterized in that The determining of the loss value by using a preset loss function based on the fused three-dimensional reconstruction result of the hand and the initial three-dimensional reconstruction result of the hand includes: Obtain a preset hand 3D model reference template; The difference between the hand 3D model reference template and the initial hand 3D model is used as the loss value.

14. A three-dimensional reconstruction device for a hand, It is characterized in that include: An acquisition unit, used to acquire hand images collected from at least two viewing angles; A determination unit, configured to determine an initial hand 3D reconstruction result corresponding to each hand image based on a preset hand 3D reconstruction network, wherein the hand 3D reconstruction result includes a hand 3D model and hand key points; The fusion unit is used to fuse the initial three-dimensional hand reconstruction results corresponding to the hand images collected from the at least two viewing angles to obtain a fused three-dimensional hand reconstruction result.

15. An electronic device, It is characterized in that include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 13.

16. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.