A method and apparatus for 3D hand reconstruction based on multi-view feature interaction

Abstract

This application proposes a 3D hand reconstruction method and apparatus based on multi-view feature interaction, comprising the following steps: acquiring hand images corresponding to the same hand from at least three viewpoints; inputting each preprocessed hand image into a pre-trained hand feature reconstruction network to obtain a vertex offset matrix; and reconstructing the 3D hand based on the vertex offset matrix and a preset standard hand mesh to obtain the 3D hand reconstruction result. In the feature fusion stage, this scheme simultaneously extracts geometric features and key information through a dynamic feature fusion module and a multi-view feature fusion module. In the decoding stage, it delves into the interaction relationships between joints and vertices, as well as the physical characteristics of hand movement, thereby achieving accurate 3D reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a method and apparatus for three-dimensional hand reconstruction based on multi-view feature interaction. Background Technology

[0002] 3D hand reconstruction has crucial application value in fields such as virtual reality (VR) and augmented reality (AR). Its core objective is to accurately recover the 3D structure and posture of the hand from 2D images, providing technical support for immersive human-computer interaction, virtual operation simulation, and other scenarios. However, current 3D hand reconstruction schemes based on monocular RGB images still face significant challenges: RGB images only contain 2D color and texture information, naturally lacking depth dimension data, making it difficult to directly reflect the spatial positional relationships of different parts of the hand; at the same time, in scenarios where the hand interacts with objects (such as grasping tools or touching objects), occlusion between fingers and between the hand and objects is very likely to occur, further losing effective structural information. These problems directly lead to defects in depth perception and scale judgment during the reconstruction process, significantly increasing the difficulty of 3D hand posture estimation and making it difficult to output a 3D model that conforms to physical laws.

[0003] To overcome the limitations of single-view methods, multi-view 3D hand reconstruction methods have emerged. These methods acquire hand images from different perspectives and utilize the complementary information between these images to capture hand structural features more comprehensively, achieving significant improvements in reconstruction accuracy and robustness. However, existing multi-view reconstruction schemes still face insurmountable technical bottlenecks: First, most methods rely on linear mapping relationships to establish the association between image features and 3D structures, failing to effectively capture the complex nonlinear dependencies between hand joints and mesh vertices. This easily leads to error accumulation during feature transfer and mapping, reducing reconstruction accuracy. Second, the model design generally lacks a dynamic interaction mechanism between the skeleton structure and mesh vertices, resulting in a disconnect between vertex positions and skeleton movement in the reconstruction results. For example, when finger joints bend, the corresponding mesh vertices do not exhibit reasonable following displacement, leading to distorted hand shapes and unreasonable structures. Third, in the multi-view feature fusion stage, differences in shooting angles and lighting conditions between different perspective images easily generate geometric inconsistencies and mathematical ill-posedness, directly affecting the stability of the reconstruction algorithm. Consequently, the final output 3D model fails to meet practical application requirements in terms of detail integrity and structural consistency.

[0004] In summary, how to overcome existing technological bottlenecks, establish and effectively utilize the complex relationships between key points and mesh vertices, realize dynamic interaction between the skeleton and mesh vertices to ensure the rationality of the reconstructed structure, and optimize multi-view feature fusion strategies to improve geometric consistency and algorithm stability have become key issues that urgently need to be addressed in the current field of multi-view 3D hand reconstruction. They are also the core obstacles to promoting the large-scale application of this technology in VR / AR and other scenarios. Summary of the Invention

[0005] This application provides a three-dimensional hand reconstruction method based on multi-view feature interaction. In the feature fusion stage, a dynamic feature fusion module and a multi-view feature fusion module are used to extract geometric features and key information simultaneously. In the decoding stage, the interaction relationship between joints and vertices, as well as the physical characteristics of hand movement, are explored in depth to complete accurate three-dimensional reconstruction.

[0006] In a first aspect, embodiments of this application provide a three-dimensional hand reconstruction method based on multi-view feature interaction, the method comprising:

[0007] Acquire hand images corresponding to the same hand from at least three perspectives, input each preprocessed hand image into a pre-trained hand reconstruction network, and obtain the hand 3D geometric model reconstruction result;

[0008] The hand reconstruction network includes a feature extraction module, a multi-view feature fusion module, and a decoding module. The feature extraction module extracts hand image features from each viewpoint. The multi-view feature fusion module uses dot product fusion branch and stitching fusion branch to fuse the hand image features from each viewpoint to obtain multi-view fused features. The decoding module converts the multi-view fused features into a three-dimensional geometric model reconstruction result of the hand.

[0009] The feature extraction module includes a feature extraction network corresponding to each viewpoint. In each feature extraction network, high-resolution and low-resolution hand feature maps of the corresponding hand image are extracted. The high-resolution and low-resolution hand feature maps are then concatenated, and channel attention is calculated to obtain hand image features. In the dot product fusion branch, any hand image feature is used as a reference feature, and all other hand image features are used as source features. A dot product is calculated on the reference and source features to obtain a first fusion feature. In the concatenation fusion branch, all hand image features are concatenated, and global information is extracted to obtain a second fusion feature. The first and second fusion features are then fused to obtain a multi-view fusion feature. The decoding module obtains a vertex offset matrix based on the multi-view fusion feature and reconstructs a preset standard hand mesh based on the vertex offset matrix to obtain a 3D hand reconstruction result.

[0010] Secondly, embodiments of this application provide a three-dimensional hand reconstruction device based on multi-view feature interaction, comprising:

[0011] The acquisition module acquires hand images corresponding to the same hand from at least three different viewpoints;

[0012] The reconstruction module inputs each preprocessed hand image into a pre-trained hand reconstruction network to obtain the hand's three-dimensional geometric model reconstruction result;

[0013] The reconstruction module is loaded with a pre-trained hand feature network. The hand feature reconstruction network includes a feature extraction module, a multi-view feature fusion module, and a decoding module. The feature extraction module includes a feature extraction network corresponding to multiple different viewpoints. The multi-view feature fusion module includes a dot product fusion branch and a concatenation fusion branch. The feature extraction module extracts the hand image features of the hand image from each viewpoint. The multi-view feature fusion module uses the dot product fusion branch and the concatenation fusion branch to fuse the hand image features of each viewpoint to obtain multi-view fused features. The decoding module converts the multi-view fused features into a three-dimensional geometric model reconstruction result of the hand.

[0014] The feature extraction module includes a feature extraction network corresponding to each viewpoint. In each feature extraction network, high-resolution and low-resolution hand feature maps of the corresponding hand image are extracted. The high-resolution and low-resolution hand feature maps are then concatenated, and channel attention is calculated to obtain hand image features. In the dot product fusion branch, any hand image feature is used as a reference feature, and all other hand image features are used as source features. A dot product is calculated on the reference and source features to obtain a first fusion feature. In the concatenation fusion branch, all hand image features are concatenated, and global information is extracted to obtain a second fusion feature. The first and second fusion features are then fused to obtain a multi-view fusion feature. The decoding module obtains a vertex offset matrix based on the multi-view fusion feature and reconstructs a preset standard hand mesh based on the vertex offset matrix to obtain a 3D hand reconstruction result.

[0015] Thirdly, embodiments of this application provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform a three-dimensional hand reconstruction method based on multi-view feature interaction.

[0016] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, which, when executed by a processor, implements a three-dimensional hand reconstruction method based on multi-view feature interaction.

[0017] The main contributions and innovations of this invention are as follows:

[0018] This solution utilizes a multi-view feature fusion module to fully leverage complementary information from different perspectives, effectively addressing common issues like occlusion and viewpoint limitations in single-view reconstruction, thus significantly improving reconstruction accuracy and robustness. In the feature extraction stage, this solution introduces an attention mechanism to dynamically select and fuse effective information, significantly improving the accuracy of 3D joints. In the feature fusion stage, a dynamic feature fusion module and a multi-view feature fusion module simultaneously extract geometric features and key information, reducing interference from irrelevant features and achieving efficient fusion of multi-scale and multi-view features. Furthermore, this solution employs a joint-vertex feature interaction module and a motion decoupling regression head to deeply explore the interaction between joints and vertices, as well as the physical characteristics of hand movements. This feature interaction-based decoding and prediction mechanism accurately predicts the motion offset of hand mesh vertices, achieving precise 3D reconstruction.

[0019] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram of the structure of a hand feature reconstruction network according to an embodiment of this application;

[0022] Figure 2 This is a structural diagram of a dynamic feature extraction module according to an embodiment of this application;

[0023] Figure 3 This is a structural diagram of a multi-view feature fusion module according to an embodiment of this application;

[0024] Figure 4 This is a structural diagram of a motion decoupling regression head according to an embodiment of this application;

[0025] Figure 5 This is a structural block diagram of a three-dimensional hand reconstruction device based on multi-view feature interaction according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0029] Example 1

[0030] This application provides a 3D hand reconstruction method based on multi-view feature interaction. In the feature fusion stage, a dynamic feature fusion module and a multi-view feature fusion module are used to simultaneously extract geometric features and key information. In the decoding stage, the interaction relationships between joints and vertices, as well as the physical characteristics of hand movements, are deeply explored to complete accurate 3D reconstruction. Specifically, refer to... Figure 1 The method includes:

[0031] Acquire hand images corresponding to the same hand from at least three perspectives, input each preprocessed hand image into a pre-trained hand reconstruction network, and obtain the hand 3D geometric model reconstruction result;

[0032] The hand reconstruction network includes a feature extraction module, a multi-view feature fusion module, and a decoding module. The feature extraction module extracts hand image features from each viewpoint. The multi-view feature fusion module uses dot product fusion branch and stitching fusion branch to fuse the hand image features from each viewpoint to obtain multi-view fused features. The decoding module converts the multi-view fused features into a three-dimensional geometric model reconstruction result of the hand.

[0033] The feature extraction module includes a feature extraction network corresponding to each viewpoint. In each feature extraction network, high-resolution and low-resolution hand feature maps of the corresponding hand image are extracted. The high-resolution and low-resolution hand feature maps are then concatenated, and channel attention is calculated to obtain hand image features. In the dot product fusion branch, any hand image feature is used as a reference feature, and all other hand image features are used as source features. A dot product is calculated on the reference and source features to obtain a first fusion feature. In the concatenation fusion branch, all hand image features are concatenated, and global information is extracted to obtain a second fusion feature. The first and second fusion features are then fused to obtain a multi-view fusion feature. The decoding module obtains a vertex offset matrix based on the multi-view fusion feature and reconstructs a preset standard hand mesh based on the vertex offset matrix to obtain a 3D hand reconstruction result.

[0034] In this scheme, hand images are acquired from 8 different perspectives. During the preprocessing of the hand images, the resolution of each hand image is unified to 256×256 pixels, and the pixel values ​​in the hand images are normalized to the range [0,1].

[0035] In this scheme, the hand feature reconstruction network is trained on the DexYCB dataset, which contains 582K sets of images, each set of images is acquired from the perspective of 8 cameras, and is used for multi-view 3D hand reconstruction tasks.

[0036] Specifically, during the pre-training process, the DexYCB dataset is imported, and each image is cropped and scaled to unify its resolution to 256×256 pixels. The pixel value range of each image is adjusted to between [0,1], thus completing the preprocessing of the data.

[0037] In the current embodiment, the feature extraction network includes a backbone module and a dynamic feature extraction module. The backbone module is used to extract features from the hand image to obtain a high-resolution hand feature map, and to downsample the high-resolution hand feature map multiple times to obtain a low-resolution hand feature map. The dynamic feature extraction module is used to perform channel attention calculation and spatial information modeling on the high-resolution hand feature map and the low-resolution hand feature map to obtain a spatial-channel joint weighted feature, and to perform nonlinear output on the spatial-channel joint weighted feature to obtain a hand feature reconstruction network.

[0038] In other words, this scheme generates a corresponding hand feature reconstruction network for each viewpoint of the hand image, thereby effectively capturing information about the hand joints and overall contour from different viewpoints and reducing interference from irrelevant features.

[0039] Furthermore, since the feature extraction network performs channel attention calculations to focus on important information in each hand image, a heatmap of the hand image from each viewpoint can be output as a heatmap, thus providing a more intuitive view of the important information in each hand image.

[0040] Specifically, in the feature extraction step of the backbone module for hand images, an attention mechanism is used in the backbone module to perform pixel-by-pixel and channel-by-channel saliency calculation on the hand images, generating spatial-channel joint weights. The spatial-channel joint weights are then used to reweight the hand images so that the model automatically focuses on key areas such as knuckles and metacarpophalangeal joints, while suppressing background and redundant textures, and outputting enhanced single-view features. This provides high-discrimination and low-noise initial feature input for subsequent multi-view feature fusion and 3D joint prediction.

[0041] Specifically, high-resolution hand feature maps are obtained by extracting features from hand images. Since hand images have high resolution, high-resolution hand feature maps focus more on local details such as knuckle creases and fingertip edges. In contrast, low-resolution hand feature maps, obtained by downsampling high-resolution hand feature maps multiple times, focus more on global information such as palm contours, palm posture, and the relative positions of each finger.

[0042] Specifically, the formula for nonlinear output of the space-channel joint weighted features is expressed as:

[0043]

[0044] in, For hand feature reconstruction networks, GELU is a non-linear activation function. For space-channel joint weighted features, This is a 3×3 convolution operation.

[0045] Furthermore, the structure diagram of the dynamic feature extraction module is as follows: Figure 2 As shown, the dynamic feature extraction module is used to stitch together a high-resolution hand feature map and a low-resolution hand feature map to obtain a stitched feature map. Channel attention is calculated on the stitched feature map to obtain channel attention weighted features. At the same time, the high-resolution hand feature map and the low-resolution hand feature map are added element-wise to obtain spatial weight coefficients. The spatial weight coefficients are then multiplied element-wise with the channel attention weighted features to obtain spatial-channel joint weighted features.

[0046] Specifically, since the low-resolution hand feature map and the high-resolution hand feature map have different resolutions, they cannot be directly stitched together. Therefore, this scheme uses deconvolution to make the resolution of the low-resolution hand feature map consistent with that of the high-resolution hand feature map. Then, the deconvolutioned low-resolution hand feature map is stitched together with the high-resolution feature map. Deconvolution can recover the spatial information lost during downsampling, and the deconvolutioned low-resolution hand feature map still retains global information for the network to understand.

[0047] Specifically, in calculating channel attention on the concatenated feature map, global average pooling is performed on the concatenated feature map to obtain a global channel descriptor. The global channel descriptor is then input into a multilayer perceptron, and channel weight coefficients are generated using the Sigmoid function. The formula is as follows:

[0048]

[0049] in, Here are the channel weight coefficients, AVP is the global average pooling operation, and [;] indicates the feature concatenation operation. This is a high-resolution hand feature map. This is a low-resolution hand feature map after deconvolution. MLP is a multilayer perceptron, and Sigmoid is the activation function.

[0050] Specifically, after obtaining the channel weight coefficients, a 1×1 convolution is used to fuse the channel weight coefficients with the concatenated feature map to complete the channel attention calculation and obtain the channel attention weighted features. The formula is expressed as:

[0051]

[0052] in, Channel attention-weighted features, This is a 1×1 convolution operation. For element-wise multiplication, This is a high-resolution hand feature map. This is a low-resolution hand feature map after deconvolution. The channel weight coefficients are used. The 1×1 convolution operation is used to compress the number of channels back to the usable dimensions, retaining important information and discarding less informative features.

[0053] Specifically, in the step of obtaining the spatial weight coefficients, a 1×1 convolution operation is performed on both the high-resolution hand feature map and the deconvolutioned low-resolution hand feature map. The convolution results are then summed element-wise and the spatial weight coefficients are obtained through the Sigmoid activation function. The formula is as follows:

[0054]

[0055] in, Spatial weighting coefficient, This is a high-resolution hand feature map. This is a low-resolution hand feature map after deconvolution. This is a 1×1 convolution operation, with Sigmoid as the activation function. This is an element-wise addition operation.

[0056] The formula for obtaining the spatial-channel joint weighted features is expressed as:

[0057]

[0058] in, Spatial weighting coefficient, Channel attention-weighted features, For space-channel joint weighted features, This is for element-wise multiplication.

[0059] In the current embodiment, different hand feature reconstruction networks share weights, and the dimensions of all hand feature reconstruction networks are unified before the hand feature reconstruction networks are input into the multi-view feature fusion module.

[0060] Specifically, by using linear layer downsampling and 1×1 convolution, the channels of each hand feature reconstruction network are unified to the same size, thereby outputting a hand feature reconstruction network with consistent dimensions, thus reducing the computational load.

[0061] In the current embodiment, the reference feature is multiplied by each source feature to obtain a similarity feature map between multiple viewpoints. The average value of the mixed features of the similarity feature maps between multiple viewpoints is taken to obtain a first fusion feature. The intermediate feature map is used to extract global information through the Sigmoid activation function to obtain a second fusion feature.

[0062] Specifically, the similarity feature maps between viewpoints correspond one-to-one with the source features, and the channels of each similarity feature map between viewpoints are compressed. The number of channels in the compressed similarity feature map between viewpoints is the same as the number of channels in the source features. For example, a 1×1 convolution operation is used to perform channel compression.

[0063] Specifically, this scheme uses the hand feature reconstruction network corresponding to the hand image obtained from the front as the reference feature. The similarity feature map between each view contains the spatial correspondence of the same hand structure under the two view. The average value of the mixed features of the similarity feature maps between multiple view effectively preserves the cross-view geometric consistency and complementary information.

[0064] Before extracting global information from the intermediate feature map, linear downsampling is performed on the intermediate feature map, and nonlinearity is introduced through the GELU activation function. Then, deconvolution upsampling is used to restore the spatial resolution to the same level as the hand feature reconstruction network.

[0065] Specifically, the global information extracted by the Sigmoid activation function includes the common hand contour, hand pose, and relative positional relationships of each finger across all viewpoints.

[0066] In the current embodiment, the structure of the multi-view feature fusion module is as follows: Figure 3 As shown, in the step of obtaining multi-view fusion features, the first fusion feature and the second fusion feature are added element-wise to obtain the multi-view feature sum, and the first fusion feature and the second fusion feature are multiplied element-wise to obtain the multi-view feature multiplication result. The multi-view feature sum and the multi-view feature multiplication result are then concatenated to obtain the multi-view fusion feature. The formula is expressed as:

[0067]

[0068] Where F represents the multi-view fusion feature. The first fusion feature, This is the second fusion feature. For element-wise multiplication, [;] indicates concatenation. This is a 1×1 convolution operation used to unify dimensions.

[0069] Specifically, element-wise addition can preserve the difference information between features from different perspectives, while element-wise multiplication can highlight the common information between features from different perspectives. Therefore, this scheme integrates multi-perspective information by concatenating the results of multi-perspective feature addition and multiplication.

[0070] In the current embodiment, the decoding module includes a decoding unit, a dual attention unit, and a motion decoupling regression head. The dual attention unit includes a first attention layer and a second attention layer. The decoding unit decodes the multi-view fusion features to obtain joint features and vertex features. The first attention layer performs cross-attention calculation on the joint features and vertex features to obtain cross-enhanced vertex features. The second attention layer performs self-attention calculation on the cross-enhanced vertex features to obtain predicted vertex features. The motion decoupling regression head obtains a vertex offset matrix based on the predicted vertex features. The vertex offset matrix contains the offset of each vertex in the predicted vertex features from the corresponding vertex in the standard hand mesh.

[0071] Specifically, the decoding unit obtains joint features and vertex features by decoding the multi-view fusion features. The joint features are features representing all hand joints, and the vertex features are features representing all vertices in the hand network.

[0072] Specifically, in the cross-attention calculation of the first attention layer, joint features are used as a guide to capture the driving relationship between joint movement and vertex features, thereby clarifying the mapping logic between the skeletal structure and the mesh shape, and integrating vertex features into joint movement constraints. That is, clarifying the basic logic of joint movement driving vertex displacement, the calculation formula for cross-attention is as follows:

[0073]

[0074] in, The query vector represents the vertex features. For input vertex features, The key vector represents the joint features. This is a value vector of joint features. Input joint features, For learnable weight matrix, To enhance vertex features through cross-referencing, softmax is used as the activation function. Let be the vertex-joint similarity matrix, and d be the feature dimension.

[0075] Specifically, in the self-attention calculation of the second attention layer, the interaction relationships between different vertices within the vertex features are modeled to further explore the spatial neighborhood relationships and motion consistency between vertices, thereby strengthening the rationality of the local topology of the mesh and avoiding unreasonable results such as finger vertices detaching from the palm. The formula for self-attention calculation is expressed as follows:

[0076]

[0077] in, The query vector represents the vertex features. The key vector represents the vertex features. This is a value vector of joint features. To enhance vertex features through cross crossings, For learnable weight matrix, To predict vertex features, This is the vertex-vertex similarity matrix.

[0078] Furthermore, the structure of the motion decoupling regression head is as follows: Figure 4As shown, the motion decoupling regression head decomposes hand movements into multiple basic motion units, and constructs a motion weight matrix, a basic motion matrix, and a motion bias matrix based on each basic motion unit. The predicted vertex features are substituted into the motion weight matrix, the basic motion matrix, and the motion bias matrix to obtain the vertex offset matrix. The motion weight matrix includes the weight of each vertex for each motion unit, the basic motion matrix includes the displacement of each vertex corresponding to each basic motion unit, and the motion bias matrix is ​​used to compensate for the offset of individual hand size differences.

[0079] Specifically, the hand mesh motion is decomposed into m basic motion units. The vertex offset is constructed by predicting the basic motions and then aggregated to generate the vertex offset. This ensures that the motion modeling conforms to the physiological structure and physical motion laws of the hand, reducing prediction complexity and improving accuracy.

[0080] Specifically, the optimal parameters of the motion weight matrix, basic motion matrix, and motion bias matrix are obtained during the pre-training process of the hand feature reconstruction network.

[0081] Specifically, the formula for generating the vertex offset matrix is:

[0082]

[0083] in, This is the vertex offset matrix. Here is the motion weight matrix. The basic motion matrix, Let V be the motion deviation matrix, and let V be the vertex set.

[0084] In the current embodiment, this method obtains the three-dimensional hand reconstruction result by superimposing the vertex offset matrix onto the vertex coordinates of the standard hand mesh.

[0085] Example 2

[0086] Based on the same concept, referencing Figure 5 This application also proposes a three-dimensional hand reconstruction device based on multi-view feature interaction, comprising:

[0087] The acquisition module is used to acquire hand images corresponding to the same hand from at least three different viewpoints;

[0088] The reconstruction module inputs each preprocessed hand image into a pre-trained hand reconstruction network to obtain the hand 3D geometric model reconstruction result. The hand reconstruction network includes a feature extraction module, a multi-view feature fusion module, and a decoding module. The feature extraction module extracts the hand image features of each viewpoint hand image. The multi-view feature fusion module uses dot product fusion branch and stitching fusion branch to fuse the hand image features of each viewpoint to obtain multi-view fusion features. The decoding module converts the multi-view fusion features into the hand 3D geometric model reconstruction result.

[0089] The feature extraction module includes a feature extraction network corresponding to each viewpoint. In each feature extraction network, high-resolution and low-resolution hand feature maps of the corresponding hand image are extracted. The high-resolution and low-resolution hand feature maps are then concatenated, and channel attention is calculated to obtain hand image features. In the dot product fusion branch, any hand image feature is used as a reference feature, and all other hand image features are used as source features. A dot product is calculated on the reference and source features to obtain a first fusion feature. In the concatenation fusion branch, all hand image features are concatenated, and global information is extracted to obtain a second fusion feature. The first and second fusion features are then fused to obtain a multi-view fusion feature. The decoding module obtains a vertex offset matrix based on the multi-view fusion feature and reconstructs a preset standard hand mesh based on the vertex offset matrix to obtain a 3D hand reconstruction result.

[0090] Example 3

[0091] This embodiment also provides an electronic device, see reference. Figure 6 It includes a memory 404 and a processor 402, the memory 404 storing a computer program and the processor 402 being configured to run the computer program to perform the steps in any of the above method embodiments.

[0092] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0093] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0094] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0095] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the three-dimensional hand reconstruction methods based on multi-view feature interaction in the above embodiments.

[0096] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0097] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0098] The input / output device 408 is used to input or output information. In this embodiment, the input information may be images of a hand from different perspectives, and the output information may be the results of a three-dimensional hand reconstruction.

[0099] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program:

[0100] Acquire hand images corresponding to the same hand from at least three perspectives, input each preprocessed hand image into a pre-trained hand reconstruction network, and obtain the hand 3D geometric model reconstruction result;

[0101] The hand reconstruction network includes a feature extraction module, a multi-view feature fusion module, and a decoding module. The feature extraction module extracts hand image features from each viewpoint. The multi-view feature fusion module uses dot product fusion branch and stitching fusion branch to fuse the hand image features from each viewpoint to obtain multi-view fused features. The decoding module converts the multi-view fused features into a three-dimensional geometric model reconstruction result of the hand.

[0102] The feature extraction module includes a feature extraction network corresponding to each viewpoint. In each feature extraction network, high-resolution and low-resolution hand feature maps of the corresponding hand image are extracted. The high-resolution and low-resolution hand feature maps are then concatenated, and channel attention is calculated to obtain hand image features. In the dot product fusion branch, any hand image feature is used as a reference feature, and all other hand image features are used as source features. A dot product is calculated on the reference and source features to obtain a first fusion feature. In the concatenation fusion branch, all hand image features are concatenated, and global information is extracted to obtain a second fusion feature. The first and second fusion features are then fused to obtain a multi-view fusion feature. The decoding module obtains a vertex offset matrix based on the multi-view fusion feature and reconstructs a preset standard hand mesh based on the vertex offset matrix to obtain a 3D hand reconstruction result.

[0103] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0104] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0105] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 6Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0106] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for 3D hand reconstruction based on multi-view feature interaction, characterized in that, include: Acquire hand images corresponding to the same hand from at least three perspectives, input each preprocessed hand image into a pre-trained hand reconstruction network, and obtain the hand 3D geometric model reconstruction result; The hand reconstruction network includes a feature extraction module, a multi-view feature fusion module, and a decoding module. The feature extraction module extracts hand image features from each viewpoint. The multi-view feature fusion module uses dot product fusion branch and stitching fusion branch to fuse the hand image features from each viewpoint to obtain multi-view fused features. The decoding module converts the multi-view fused features into a three-dimensional geometric model reconstruction result of the hand. The feature extraction module includes a feature extraction network corresponding to each viewpoint. In each feature extraction network, a high-resolution hand feature map and a low-resolution hand feature map of the corresponding hand image are extracted. The high-resolution hand feature map and the low-resolution hand feature map are concatenated and channel attention is calculated to obtain the hand image features. In the dot product fusion branch, any hand image feature is used as a reference feature, and the hand image features other than the reference feature are used as source features. The reference feature and the source features are dot product calculated to obtain the first fusion feature. In the concatenation fusion branch, all hand image features are concatenated and global information is extracted to obtain the second fusion feature. The first fusion feature and the second fusion feature are fused to obtain the multi-view fusion feature. The decoding module obtains the vertex offset matrix based on the multi-view fusion features, and reconstructs the preset standard hand mesh based on the vertex offset matrix to obtain the three-dimensional hand reconstruction result.

2. The three-dimensional hand reconstruction method based on multi-view feature interaction according to claim 1, characterized in that, The feature extraction network includes a backbone module and a dynamic feature extraction module. The backbone module is used to extract features from the hand image to obtain a high-resolution hand feature map, and to downsample the high-resolution hand feature map multiple times to obtain a low-resolution hand feature map. The dynamic feature extraction module is used to perform channel attention calculation and spatial information modeling on the high-resolution hand feature map and the low-resolution hand feature map to obtain spatial-channel joint weighted features, and to perform nonlinear output on the spatial-channel joint weighted features to obtain hand image features.

3. The three-dimensional hand reconstruction method based on multi-view feature interaction according to claim 2, characterized in that, The dynamic feature extraction module is used to stitch together a high-resolution hand feature map and a low-resolution hand feature map to obtain a stitched feature map. Channel attention is calculated on the stitched feature map to obtain channel attention weighted features. At the same time, the high-resolution hand feature map and the low-resolution hand feature map are added element-wise to obtain spatial weight coefficients. The spatial weight coefficients are then multiplied element-wise with the channel attention weighted features to obtain spatial-channel joint weighted features.

4. The three-dimensional hand reconstruction method based on multi-view feature interaction according to claim 1, characterized in that, The first fusion feature and the second fusion feature are added element by element to obtain the multi-view feature addition result. The first fusion feature and the second fusion feature are multiplied element by element to obtain the multi-view feature multiplication result. The multi-view feature addition result and the multi-view feature multiplication result are concatenated to obtain the multi-view fusion feature.

5. The three-dimensional hand reconstruction method based on multi-view feature interaction according to claim 1, characterized in that, The decoding module includes a decoding unit, a dual attention unit, and a motion decoupling regression head. The dual attention unit includes a first attention layer and a second attention layer. The decoding unit decodes the multi-view fusion features to obtain joint features and vertex features. The first attention layer performs cross-attention calculation on the joint features and vertex features to obtain cross-enhanced vertex features. The second attention layer performs self-attention calculation on the cross-enhanced vertex features to obtain predicted vertex features. The motion decoupling regression head obtains a vertex offset matrix based on the predicted vertex features. The vertex offset matrix contains the offset of each vertex in the predicted vertex features from the corresponding vertex in the standard hand mesh.

6. The three-dimensional hand reconstruction method based on multi-view feature interaction according to claim 5, characterized in that, The motion decoupling regression head decomposes hand movements into multiple basic motion units and constructs a motion weight matrix, a basic motion matrix, and a motion bias matrix based on each basic motion unit. The predicted vertex features are substituted into the motion weight matrix, the basic motion matrix, and the motion bias matrix to obtain the vertex offset matrix. The motion weight matrix includes the weight of each vertex for each motion unit, the basic motion matrix includes the displacement of each vertex corresponding to each basic motion unit, and the motion bias matrix is ​​used to compensate for the offset of individual hand size differences.

7. A three-dimensional hand reconstruction device based on multi-view feature interaction, characterized in that, include: The acquisition module is used to acquire hand images corresponding to the same hand from at least three different viewpoints; The reconstruction module inputs each preprocessed hand image into a pre-trained hand reconstruction network to obtain the hand 3D geometric model reconstruction result. The hand reconstruction network includes a feature extraction module, a multi-view feature fusion module, and a decoding module. The feature extraction module extracts the hand image features of each viewpoint hand image. The multi-view feature fusion module uses dot product fusion branch and stitching fusion branch to fuse the hand image features of each viewpoint to obtain multi-view fusion features. The decoding module converts the multi-view fusion features into the hand 3D geometric model reconstruction result. The feature extraction module includes a feature extraction network corresponding to each viewpoint. In each feature extraction network, a high-resolution hand feature map and a low-resolution hand feature map of the corresponding hand image are extracted. The high-resolution hand feature map and the low-resolution hand feature map are concatenated and channel attention is calculated to obtain the hand image features. In the dot product fusion branch, any hand image feature is used as a reference feature, and the hand image features other than the reference feature are used as source features. The reference feature and the source features are dot product calculated to obtain the first fusion feature. In the concatenation fusion branch, all hand image features are concatenated and global information is extracted to obtain the second fusion feature. The first fusion feature and the second fusion feature are fused to obtain the multi-view fusion feature. The decoding module obtains the vertex offset matrix based on the multi-view fusion features, and reconstructs the preset standard hand mesh based on the vertex offset matrix to obtain the three-dimensional hand reconstruction result.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform a three-dimensional hand reconstruction method based on multi-view feature interaction as described in any one of claims 1-6.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements a three-dimensional hand reconstruction method based on multi-view feature interaction as described in any one of claims 1-6.

Images