Hand three-dimensional reconstruction method and device suitable for complex illumination condition

Abstract

The invention discloses a hand three-dimensional reconstruction method and device suitable for a complex illumination condition, and the method comprises the steps: obtaining hand event stream data and an LDR image # imgabs0 # which are collected in a time-space synchronization manner, carrying out the preprocessing of the event stream data, and inputting the preprocessed event stream data and an LDR image # imgabs1 # into a pre-trained cross-modal adaptive fusion network; and inputting the HDR image # imgabs3 # and the LDR image # imgabs4 # into a pre-trained cascaded hand three-dimensional reconstruction network to obtain a hand three-dimensional model Mhand, wherein the HDR image # imgabs3 # and the LDR image # imgabs4 # are input into the pre-trained cascaded hand three-dimensional reconstruction network. According to the hand three-dimensional reconstruction method and device suitable for the complex illumination condition, the robustness of an imaging network in a real scene containing extreme illumination is improved, noise is suppressed, meanwhile, the loss of HDR image details is avoided, and the three-dimensional coordinates of the three-dimensional hand grid vertex are effectively inferred.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a method and device for three-dimensional reconstruction of a hand under complex lighting conditions. Background Art

[0002] 3D reconstruction of the hand is a basic computer vision task that can be applied to a variety of scenarios, such as human-computer interaction, virtual reality, augmented reality, autonomous driving, and robot control. Thanks to the development of deep learning in recent years, 3D reconstruction methods for the hand have made significant progress. Existing studies on traditional frame cameras have proposed and developed many 3D reconstruction methods for the hand using deep neural networks. However, such methods have limitations in applicable scenarios and inherit the essential limitations of frame cameras. They are not sufficient to deal with severely missing image details and textures in backlit and overexposed scenes. Although event-based methods can effectively deal with backlit and overexposed situations, due to the sparsity of the event stream, such methods are difficult to reconstruct the details of the hand, and in static scenes, since event cameras cannot generate events, such methods are difficult to work properly. Therefore, research on hand posture estimation that fuses RGB images and events is of great significance. Summary of the invention

[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a method and device for three-dimensional reconstruction of a hand suitable for complex lighting conditions.

[0004] The present invention proposes a method for 3D reconstruction of a hand under complex lighting conditions, comprising the following steps:

[0005] S1. Acquiring event stream data and LDR images of the hand collected synchronously in time and space

[0006] S2, after preprocessing the event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0007] S3. HDR image LDR Image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0008] Preferably, the preprocessing specifically includes:

[0009] Perform event compression on the event stream data to obtain the corresponding event voxels

[0010] Preferably, the cross-modal adaptive fusion network includes a feature alignment module, a multi-scale feature extraction module, a cross-modal feature fusion module, and an image reconstruction module; step S2 specifically includes:

[0011] Event voxel and LDR image Input to the feature alignment module to obtain event features and LDR image features

[0012] The event features and LDR image features Input to the multi-scale feature extraction module to obtain multi-scale event features and multi-scale image features

[0013] Multi-scale event features and multi-scale image features Input to the cross-modal feature fusion module, and use the exposure mask M to extract the corresponding exposure mask features for multi-scale image features Perform feature weighting of different exposure areas to obtain exposure-weighted LDR image features

[0014] Exposure-weighted LDR image features Input to the image reconstruction module to organically integrate features of different scales to obtain HDR images

[0015] Preferably, the feature alignment module includes an adversarial feature discriminator D F , event feature encoder T ε , image feature encoder T L , the event voxel and LDR image Input to the feature alignment module to obtain event features and LDR image features Specifically include:

[0016] Event voxel Input event feature encoder T ε , using the event feature encoder T ε For event voxels Encode to obtain event characteristics

[0017] And the LDR image Input image feature encoder T L , using the image feature encoder T L For LDR images Encode to obtain LDR image features

[0018] Using adversarial feature discriminator D F The event features and LDR image features Alignment is performed on potentially shared domains.

[0019] Preferably, the cascaded hand 3D reconstruction network includes: a hand detection network D MP , 2D pose estimation network P MP , a three-dimensional posture estimation network PoseNet, and a hand three-dimensional model reconstruction network MeshNet; step S3 specifically includes:

[0020] Convert HDR image Input hand detection network D MP , through the hand detection network D MP Get the hand position coordinates to detect the hand image I hand ;

[0021] The hand image I hand Input 2D pose estimation network P MP , through the 2D pose estimation network P MP For the hand image I hand The coordinates of the 21 2D skeleton points are used to locate the corresponding 2D posture of the hand.

[0022] The 2D hand gesture Input the 3D posture estimation network PoseNet, and use the 3D posture estimation network PoseNet to transform the 2D posture of the hand Convert it into a 4096-dimensional feature vector, then convert the 4096-dimensional feature vector into a 3J-dimensional vector, and use the 3J-dimensional vector as the three-dimensional posture of the hand Output;

[0023] The 2D hand gesture 3D hand gestures Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand .

[0024] Preferably, the hand three-dimensional model reconstruction network MeshNet includes a fusion posture estimation network and an adaptive graph convolution network; 3D hand gestures Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand , including:

[0025] The 2D hand gesture Corresponding 2D bone points and 3D hand postures The corresponding 3D skeleton points are input into the fusion pose estimation network to generate the fused pose estimate P∈R J×5 ;

[0026] The fused pose estimate P∈R J×5 Input the adaptive graph convolutional network to obtain the 3D hand model M hand .

[0027] Preferably, the training process of the cross-modal adaptive fusion network specifically includes:

[0028] Collect multiple groups of hand images and mark the hand joints to form a training sample set;

[0029] With the hand image as input and the three-dimensional coordinates of each joint as output, the preset cross-modal adaptive fusion network is weakly supervised trained based on the training sample set to obtain a trained cross-modal adaptive fusion network;

[0030] The loss function used during training is as follows:

[0031]

[0032] in, is the loss function for feature alignment; is the content consistency loss function; is the perceptual loss function; is the generator level loss function; is the noise suppression loss function; ω fa ,ω pix ,ω per ,ω ce ,ω de All are weights.

[0033] Preferably, the loss function for feature alignment is Specifically:

[0034]

[0035] in, is the mathematical expectation; is the discriminator D F Determine LDR image features and event characteristics The same credibility is distributed;

[0036] Content consistency loss function Specifically:

[0037]

[0038] in, To restore the image; is the reference image; || ||1 is the L1 loss function;

[0039] Perceptual Loss Function Specifically:

[0040]

[0041] in, is the j-th layer feature map extracted from the image using the pre-trained VGG-19 network; C j ×H j ×W j is the scale; α j is the weight of the j-th layer feature map; is the L2 loss function;

[0042] Generator Level Loss Function Specifically:

[0043]

[0044] in, Represents the discriminator D I judge is the probability of the true outcome;

[0045] The noise suppression loss function is specifically:

[0046]

[0047] in, For The result of finding the absolute value; is the weight map after denoising.

[0048] The present invention proposes a three-dimensional hand reconstruction device suitable for use under complex lighting conditions, comprising:

[0049] Data acquisition module, used to obtain event stream data and LDR images of the hand acquired synchronously in time and space

[0050] The first processing module is used to pre-process the event stream data and the LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0051] The second processing module is used to convert the HDR image LDR Image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0052] In the present invention, the proposed method and device for three-dimensional reconstruction of hands under complex lighting conditions greatly enhances the ability to cope with complex lighting factors by introducing event cameras as a supplement to traditional cameras. At the same time, due to the characteristics of the event camera's high-speed perception of moving targets, the network pays more attention to the target's motion information and better extracts its semantic features, enhancing the network model's ability to handle dynamic gestures and laying the foundation for three-dimensional reconstruction of hands in motion. The cross-modal adaptive fusion network based on the exposure attention mechanism establishes an adaptive fusion mechanism for event streams and images under complex lighting conditions by designing an implicit feature alignment strategy and an adaptive fusion method based on the exposure attention mechanism, significantly improving the robustness of the imaging network in real scenes containing extreme lighting. The model optimization strategy based on image detail enhancement and noise suppression enables the network to effectively cope with multi-source mixed noise from LDR images and event streams under extreme lighting conditions, and uses the distribution difference of image content and noise to construct a gradient penalty term, which suppresses noise while avoiding the loss of HDR image details. The cascaded 3D hand reconstruction network uses the adjacency matrix to describe the topological structure of the hand skeleton points, and through multiple layers of graph convolution operations, gradually propagates and aggregates the information of the skeleton points, thereby effectively inferring the 3D coordinates of the vertices of the 3D hand mesh. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the processing flow structure of a method for three-dimensional reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0054] Figure 2 A schematic diagram of the distribution structure of acquisition equipment for a method for three-dimensional reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0055] Figure 3 A schematic diagram of the structure of an HDR image generation process of a method for 3D reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0056] Figure 4 A schematic diagram of a feature extraction structure of a method for 3D reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0057] Figure 5 A schematic diagram of an exposure fusion structure of a method for 3D reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0058] Figure 6A schematic diagram of an image reconstruction structure of a method for three-dimensional reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0059] Figure 7 A schematic diagram of the structure of a cascaded 3D hand reconstruction process of a 3D hand reconstruction method suitable for complex lighting conditions proposed by the present invention;

[0060] Figure 8 This is a schematic diagram of the overall process structure of MeshNet, a method for 3D reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0061] Fig. 9 This is a schematic diagram of the implementation optimization process structure of a method for three-dimensional reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0062] Fig.10 A schematic diagram of the processing steps of a method for three-dimensional reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0063] Fig.11 Schematic diagram of the system architecture of a three-dimensional hand reconstruction device suitable for complex lighting conditions proposed by the present invention. DETAILED DESCRIPTION

[0064] Reference Figure 1-11 The present invention proposes a method for 3D reconstruction of a hand under complex lighting conditions, comprising the following steps:

[0065] S1. Acquiring event stream data and LDR images of the hand collected synchronously in time and space

[0066] In this embodiment, the preprocessing specifically includes:

[0067] Perform event compression on the event stream data to obtain the corresponding event voxels

[0068] In this embodiment, a hybrid imaging device is constructed and a hand three-dimensional reconstruction data set under different lighting conditions is captured. The hybrid imaging device constructed is composed of a traditional RGB camera and an event camera. On this basis, a time synchronization technology based on hardware triggering and a spatial calibration method based on spatial homography transformation are implemented to ensure the spatiotemporal matching of the collected cross-modal data. The present invention designs a hardware clock trigger synchronization method to achieve accurate time alignment when the traditional camera and the event camera are collected. STM32F103C8T6 is selected as the main control to construct a timer-triggered dual camera. In practical applications, the timer configuration can be flexibly adjusted to accurately control the exposure time, mode and frame rate to ensure data quality. In view of the different voltage reference signals caused by the differences in camera types, the system design distinguishes and configures the reference voltage to ensure the stability and reliability of the camera synchronization trigger. Once the synchronization trigger signal is received, the two cameras will shoot synchronously, and the imaging information and the trigger timestamp will be transmitted to the computer together. In order to ensure that the field of view of the two cameras in the hybrid imaging device is consistent, the present invention adopts a spatial calibration method based on spatial homography transformation. Given the internal and external parameters of the event camera and the RGB camera, the spatial alignment between the camera outputs can be completed by calculating the homography transformation matrix. In order to facilitate the homography estimation of event stream and RGB image frame, the present invention utilizes E2VID algorithm to reconstruct intensity image from event stream, converts event stream into visual representation similar to RGB image frame, and then ensures that scene targets captured by camera are spatially consistent by performing homography estimation.

[0069] In this embodiment, equipment is set up: the acquisition equipment is as follows: Figure 2As shown, a beam splitter is used to divide the light under the same field of view into two beams, which are collected by a traditional RGB camera and an event camera respectively. Time synchronization: A hardware clock trigger synchronization method is designed to ensure that the traditional camera and the event camera can achieve accurate time alignment during data acquisition. The present invention selects STM32F103C8T6 as the main control unit and constructs a timer for triggering two cameras. In practical applications, according to the needs of specific scenarios, the configuration of the timer can be flexibly adjusted to achieve accurate control of exposure time, exposure mode and frame rate to ensure the quality of collected data. When a synchronization trigger signal is received, the two cameras will shoot synchronously, and the imaging information and the corresponding trigger timestamp will be transmitted to the acquisition device together. Spatial synchronization: The present invention introduces a calibration method based on spatial homography transformation, and the homography transformation matrix is ​​obtained by accurately calculating the internal and external parameters of the event camera and the RGB camera, so as to achieve accurate spatial alignment between camera outputs. In order to simplify the homography estimation process of the event stream and the RGB image frame, the present invention uses the E2VID algorithm to convert the event stream into a visual form similar to the RGB image frame. After this conversion, homography estimation is performed to ensure that the scene targets captured by the camera remain highly consistent in the spatial dimension. A cross-modal adaptive fusion network based on the exposure attention mechanism is constructed. The network includes "feature alignment and multi-scale feature extraction module based on domain adaptation" and "adaptive fusion reconstruction module based on exposure attention mechanism". The network block diagram is shown in the figure below. Figure 3 shown.

[0070] In this embodiment, the event is first evenly divided into 5 time intervals on the time axis, and then the corresponding event points are accumulated in each time interval according to the polarity of the event. The frame compression process of each polarity event in a specific time interval can be expressed as:

[0071]

[0072] in, represents the event frame compression result of the scale 2B×H×W, 2 represents the positive and negative polarity of the event, and H and W represent the spatial resolution of the event. i ,t i ) represents t i The pixel point p(x i ,t i ), m∈{0,1,…,B-1} represents the mth time interval, and t i It means that in the time interval The sum of events in B is set to 5.

[0073] S2, after preprocessing the event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0074] In this embodiment, the cross-modal adaptive fusion network includes a feature alignment module, a multi-scale feature extraction module, a cross-modal feature fusion module, and an image reconstruction module; step S2 specifically includes: converting event voxels and LDR image Input to the feature alignment module to obtain event features and LDR image features The event features and LDR image features Input to the multi-scale feature extraction module to obtain multi-scale event features and multi-scale image features Multi-scale event features and multi-scale image features Input to the cross-modal feature fusion module, and use the exposure mask M to extract the corresponding exposure mask features for multi-scale image features Perform feature weighting of different exposure areas to obtain exposure-weighted LDR image features Exposure-weighted LDR image features Input to the image reconstruction module to organically integrate features of different scales to obtain HDR images

[0075] In this embodiment, a feature discriminator is first introduced to shorten the distribution distance between LDR image features and event features, so as to realize the alignment of cross-modal features in the shared domain; then a multi-scale event feature extractor and a multi-scale image feature extractor are constructed to mine the dynamic range information contained in LDR images and events. The latter first extracts an exposure mask that can reflect the brightness saturation of different regions from the LDR image, and then effectively realizes the reconstruction of HDR images under different lighting conditions by implementing cross-modal feature fusion and multi-scale feature fusion guided by the exposure mask.

[0076] In this embodiment, cross-modal feature alignment: the present invention encodes event voxels and LDR images into a potential shared domain, and introduces an adversarial feature discriminator to bring the feature distributions of the two closer, such as Figure 4 As shown. Then the event feature encoder T is used ε Encode event features and use image feature encoder T L Encode the features of LDR images. Event features and LDR image features It can be expressed as:

[0077]

[0078] In this embodiment, the feature alignment module includes an adversarial feature discriminator D F , event feature encoder T ε , image feature encoder T L , the event voxel and LDR image Input to the feature alignment module to obtain event features and LDR image features Specifically include: Input event feature encoder T ε , using the event feature encoder T ε For event voxels Encode to obtain event characteristics And the LDR image Input image feature encoder T L , using the image feature encoder T L For LDR images Encode to obtain LDR image features Using adversarial feature discriminator D F The event features and LDR image features Alignment is performed on potentially shared domains.

[0079] In this embodiment, multi-scale feature extraction: The present invention constructs a multi-scale event feature extraction module and a multi-scale image feature extraction module to realize the extraction of deep and shallow features of RGB images and event voxels. The multi-scale feature extraction module is mainly composed of a convolution module (Conv Block), a downsampling layer (Downsample), a residual group (Residual Group) and a dense fusion module (Dense Fusion). The convolution module and the downsampling layer are responsible for mapping the LDR image and events to different depths and scales, and then the residual group extracts the feature representation of each depth and scale. In order to compensate for the spatial information that may be lost during the downsampling process, the present invention uses a dense fusion module to ensure effective connection and information exchange between non-adjacent level features. The multi-scale feature extraction process can be expressed as:

[0080]

[0081] in, Represents multi-scale event characteristics, Represents multi-scale image features.

[0082] In this embodiment, the exposure mask is generated: the present invention introduces an exposure mask M, which dynamically adjusts the weights of different regions in the fusion process according to the brightness information of the LDR image. The exposure mask is generated by a segmented mapping function, which takes the brightness value of the LDR image as input and outputs the corresponding mask value. The mapping function is as follows: Figure 5 (a), the calculation formula is as follows:

[0083]

[0084] in, represents the brightness channel of the LDR image, and δ is set to 0.7.

[0085] In this embodiment, exposure mask guided cross-modal feature fusion: The present invention designs an exposure mask guided cross-modal feature fusion module (EAF) for cross-modal feature fusion. Figure 5 As shown in (b), this module uses LDR multi-scale image features and multi-scale event characteristics As input, the exposure mask is used to promote The adaptive transformation of effectively realizes the feature weighting of different exposure areas. Specifically, given the LDR image feature The exposure mask is downsampled to match The size of the image is obtained by extracting the corresponding exposure mask features, and then using the exposure mask features to Scaling and shifting operations are performed to generate exposure-weighted LDR image features. The process can be expressed as:

[0086]

[0087] In this embodiment, image reconstruction: The image reconstruction module of the present invention is composed of an exposure mask-guided cross-modal feature fusion module (EAF), a residual group, a convolution module, a deconvolution module, and a dense fusion module. Figure 6 As shown. The role of the EAF module is to use the exposure mask to adaptively realize the fusion of LDR image features and event features. The residual module plays the role of extracting and connecting information of the same scale, effectively enhancing the reconstruction capability of HDR features at the spatial level. At the same time, the deconvolution operation gradually restores the spatial resolution of the feature map, gradually revealing the details and structural information of the image. In addition, the dense fusion module transfers deep semantic features to shallow texture features as an error correction feedback mechanism to refine and enhance image texture details for better reconstruction effects. Through the synergistic effect of this series of modules, the organic integration of features of different scales is effectively realized, and finally the convolution module is used to generate information-rich and pleasing HDR images. The process can be expressed as:

[0088]

[0089] S3. HDR image LDR Image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0090] In this embodiment, the cascaded hand 3D reconstruction network includes: a hand detection network D MP , 2D pose estimation network P MP , a three-dimensional posture estimation network PoseNet, and a hand three-dimensional model reconstruction network MeshNet; step S3 specifically includes:

[0091] Convert HDR image Input hand detection network D MP , through the hand detection network D MP Get the hand position coordinates to detect the hand image I hand ;

[0092] The hand image I hand Input 2D pose estimation network P MP , through the 2D pose estimation network P MP For the hand image I hand The coordinates of the 21 2D skeleton points are used to locate the corresponding 2D posture of the hand.

[0093] The 2D hand gesture Input the 3D posture estimation network PoseNet, and use the 3D posture estimation network PoseNet to transform the 2D posture of the hand Convert it into a 4096-dimensional feature vector, then convert the 4096-dimensional feature vector into a 3J-dimensional vector, and use the 3J-dimensional vector as the three-dimensional posture of the hand Output;

[0094] The 2D hand gesture 3D hand gestures Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand .

[0095] In this embodiment, the hand 3D model reconstruction network MeshNet includes a fusion posture estimation network and an adaptive graph convolution network; 3D hand gestures Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand , including:

[0096] The 2D hand gesture Corresponding 2D bone points and 3D hand postures The corresponding 3D skeleton points are input into the fusion pose estimation network to generate the fused pose estimate P∈R J×5 ;

[0097] The fused pose estimate P∈R J×5 Input the adaptive graph convolutional network to obtain the 3D hand model M hand .

[0098] In this embodiment, the training process of the cross-modal adaptive fusion network specifically includes: collecting multiple groups of hand images and marking the hand joints to form a training sample set; taking the hand images as input and the three-dimensional coordinates of each joint as output, performing weakly supervised training on the preset cross-modal adaptive fusion network based on the training sample set to obtain a trained cross-modal adaptive fusion network.

[0099] In this embodiment, the loss function used during training is as follows:

[0100]

[0101] in, is the loss function for feature alignment; is the content consistency loss function; is the perceptual loss function; is the generator level loss function; is the noise suppression loss function; ω fa ,ω pix ,ω per ,ω ce ,ω de All are weights.

[0102] In this embodiment, a model optimization strategy based on image detail enhancement and noise suppression is constructed. In order to optimize the above network architecture, the present invention designs feature alignment constraints, content consistency constraints and noise suppression constraints from three perspectives: cross-domain distribution closeness, image content reconstruction and noise suppression.

[0103] Among them, the feature alignment constraint introduces a feature discriminator to judge the distribution similarity between LDR image features and event features. Here, the generator and the discriminator perform adversarial learning. As the training progresses, the semantic information contained in the features generated by the generator gradually becomes consistent.

[0104] Content consistency constraints are used to reduce the distance between the output HDR image and the real HDR image. In view of the importance of pixel-level consistency in image reconstruction tasks, the present invention uses Manhattan distance (L1 loss) to constrain this process at the pixel level. At the same time, in order to enhance the content and structural information of the image and generate results that are more in line with human eye perception, the present invention further uses perceptual loss to optimize the network. In order to alleviate the problem that the restored result is not natural enough in color, the present invention designs an image discriminator and introduces pixel-level adversarial loss to effectively improve the color fidelity and realism of the final HDR image, and obtains a more attractive visual result. Reconstructed HDR images are often affected by multi-source mixed noise from LDR images and event streams. In order to effectively suppress these noises, the present invention proposes noise suppression loss, which further constrains the output image at the gradient level. This loss uses the distribution difference of image content and noise to construct a gradient penalty term, aiming to suppress noise while avoiding the loss of HDR image details.

[0105] In this embodiment, the loss function of feature alignment is Specifically:

[0106]

[0107] in, is the mathematical expectation; is the discriminator D F Determine LDR image features and event characteristics The same credibility is distributed;

[0108] Content consistency loss function Specifically:

[0109]

[0110] in, To restore the image; is the reference image; || ||1 is the L1 loss function;

[0111] Perceptual Loss Function Specifically:

[0112]

[0113] in, is the j-th layer feature map extracted from the image using the pre-trained VGG-19 network; C j ×H j ×W j is the scale; α j is the weight of the j-th layer feature map; is the L2 loss function;

[0114] Generator Level Loss Function Specifically:

[0115]

[0116] in, Represents the discriminator D I judge is the probability of the true outcome;

[0117] The noise suppression loss function is specifically:

[0118]

[0119] in, For The result of finding the absolute value; is the weight map after denoising.

[0120] In this embodiment, the cascaded 3D hand reconstruction network extracts the 3D key bone points of the hand from the enhanced HDR image in a cascade manner and constructs a 3D hand model. First, a hand region detection model is constructed to obtain the palm image from the input image, and then a 2D bone point extraction model is constructed to obtain the coordinates of the key bone points. The 3D bone points are obtained by upgrading the 2D bone points, and finally the 2D and 3D bone points are used to construct a 3D hand model. The cascaded 3D hand reconstruction network includes three core components: MediaPipe Hands, PoseNet, and MeshNet. MediaPipe Hands is used to obtain the hand position and estimate the 2D hand posture from the input RGB image; PoseNet upgrades these 2D bone points to 3D hand bone points; and MeshNet uses a combination of 2D and 3D bone point information of the hand to finely estimate the 3D mesh of the hand in a coarse-to-fine manner. The overall process is as follows: Figure 7 shown.

[0121] In this embodiment, the hand 2D posture estimation: The present invention uses the MediaPipe Hands hand detection network D MP To obtain the hand position coordinates, the detector model uses a feature extractor based on the encoding and decoding structure to achieve greater scene context awareness. The process can be expressed as:

[0122]

[0123] in, For the enhanced HDR image, I hand is the hand image detected.

[0124] Using MediaPipe Hands 2D pose estimation network PMP The process of accurately locating the coordinates of the 21 two-dimensional bone points of the detected hand image can be described as follows:

[0125]

[0126] In this embodiment, the two-dimensional hand skeleton point is upgraded: the present invention uses the PoseNet two-dimensional hand skeleton point upgrade network to convert the normalized two-dimensional input posture Then, through two residual blocks and a fully connected layer, the feature vector is converted into a 3J-dimensional vector, expressed as Where j represents the number of hand joints. The process can be expressed as:

[0127]

[0128] In this embodiment, the hand 3D model is reconstructed: the present invention uses the MeshNet hand 3D model reconstruction network to estimate the 2D posture and 3D pose estimation Specifically, the 2D skeleton points and 3D skeleton points are concatenated to generate a fused pose estimate P∈R J×5 , and then estimate the root joint-related 3D hand mesh M∈R from P V×3 , where V represents the number of mesh vertices. The overall process of MeshNet is as follows Figure 8 As shown in the figure, it is mainly composed of an adaptive graph convolutional network, which uses the adjacency matrix to describe the topological structure of the skeleton points, and through multi-layer graph convolution operations, gradually propagates and aggregates the information of the hand skeleton points, thereby reconstructing the 3D model of the hand. hand The reconstruction process is expressed as:

[0129]

[0130] In this embodiment, the main purpose is to implement the transplantation and acceleration of the algorithm on the edge device to ensure the application efficiency. The overall solution is as follows: Fig. 9 As shown in the figure, it specifically includes: implementing algorithm model optimization and conversion, optimizing and accelerating the model under the pytorch framework, and transplanting the model to the AI ​​chip through model conversion. Implementing algorithm hardware reasoning, sending the pre-processed data to the AI ​​chip for algorithm forward reasoning, and obtaining the final result through post-processing.

[0131] Reference Figure 1-11 The present invention proposes a three-dimensional hand reconstruction device suitable for complex lighting conditions, comprising:

[0132] Data acquisition module, used to obtain event stream data and LDR images of the hand acquired synchronously in time and space

[0133] The first processing module is used to pre-process the event stream data and the LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0134] The second processing module is used to convert the HDR image LDR Image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0135] In the specific working process of the method and device for 3D reconstruction of a hand under complex lighting conditions of this embodiment,

[0136] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for 3D reconstruction of a hand under complex lighting conditions, characterized in that: The following steps are involved: S1. Acquiring event stream data and LDR images of the hand collected synchronously in time and space S2, after preprocessing the event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand S3. HDR image LDR Image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

2. The method for 3D reconstruction of a hand under complex lighting conditions according to claim 1, characterized in that: The pre-processing specifically includes: Perform event compression on the event stream data to obtain the corresponding event voxels 3. The method for 3D reconstruction of a hand under complex lighting conditions according to claim 2, characterized in that: The cross-modal adaptive fusion network includes a feature alignment module, a multi-scale feature extraction module, a cross-modal feature fusion module, and an image reconstruction module; Step S2 specifically includes: Event voxel and LDR image Input to the feature alignment module to obtain event features and LDR image features The event features and LDR image features Input to the multi-scale feature extraction module to obtain multi-scale event features and multi-scale image features Multi-scale event features and multi-scale image features Input to the cross-modal feature fusion module, and use the exposure mask M to extract the corresponding exposure mask features for multi-scale image features Perform feature weighting of different exposure areas to obtain exposure-weighted LDR image features Exposure-weighted LDR image features Input to the image reconstruction module to organically integrate features of different scales to obtain HDR images 4. The method for 3D reconstruction of a hand under complex lighting conditions according to claim 3, characterized in that: The feature alignment module includes an adversarial feature discriminator D F , event feature encoder T ε , image feature encoder T L , the event voxel and LDR image Input to the feature alignment module to obtain event features and LDR image features Specifically include: Event voxel Input event feature encoder T ε , using the event feature encoder T ε For event voxels Encode to obtain event characteristics And the LDR image Input image feature encoder T L , using the image feature encoder T L For LDR images Encode to obtain LDR image features Using adversarial feature discriminator D F The event features and LDR image features Alignment is performed on potentially shared domains.

5. The method for 3D reconstruction of a hand under complex lighting conditions according to claim 1, characterized in that: The cascaded 3D hand reconstruction network includes: hand detection network D MP , 2D pose estimation network P MP , a three-dimensional posture estimation network PoseNet, and a hand three-dimensional model reconstruction network MeshNet; step S3 specifically includes: Convert HDR image Input hand detection network D MP , through the hand detection network D MP Get the hand position coordinates to detect the hand image I hand ; The hand image I hand Input 2D pose estimation network P MP , through the 2D pose estimation network P MP For the hand image I hand The coordinates of the 21 2D skeleton points are used to locate the corresponding 2D posture of the hand. The 2D hand gesture Input the 3D posture estimation network PoseNet, and use the 3D posture estimation network PoseNet to transform the 2D posture of the hand Convert it into a 4096-dimensional feature vector, then convert the 4096-dimensional feature vector into a 3J-dimensional vector, and use the 3J-dimensional vector as the three-dimensional posture of the hand Output; The 2D hand gesture 3D hand gestures Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand .

6. The method for 3D reconstruction of a hand under complex lighting conditions according to claim 5, characterized in that: The hand three-dimensional model reconstruction network MeshNet includes a fusion posture estimation network and an adaptive graph convolution network; the two-dimensional posture of the hand 3D hand gestures Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand , specifically including: The 2D hand gesture Corresponding 2D bone points and 3D hand postures The corresponding 3D skeleton points are input into the fusion pose estimation network to generate the fused pose estimate P∈R J×5 ; The fused pose estimate P∈R J×5 Input the adaptive graph convolutional network to obtain the 3D hand model M hand .

7. The method for 3D reconstruction of a hand under complex lighting conditions according to claim 4, characterized in that: The training process of the cross-modal adaptive fusion network specifically includes: Collect multiple groups of hand images and mark the hand joints to form a training sample set; With the hand image as input and the three-dimensional coordinates of each joint as output, the preset cross-modal adaptive fusion network is weakly supervised trained based on the training sample set to obtain a trained cross-modal adaptive fusion network; The loss function used during training as follows: in, is the loss function for feature alignment; is the content consistency loss function; is the perceptual loss function; is the generator level loss function; is the noise suppression loss function; ω fa ,ω pix ,ω per ,ω ce ,ω de All are weights.

8. The method for 3D reconstruction of a hand under complex lighting conditions according to claim 7, characterized in that: Loss function for feature alignment Specifically: in, is the mathematical expectation; is the discriminator D F Determine LDR image features and event characteristics The same credibility is distributed; Content consistency loss function Specifically: in, To restore the image; is the reference image; || ||1 is the L1 loss function; Perceptual Loss Function Specifically: in, is the j-th layer feature map extracted from the image using the pre-trained VGG-19 network; C j ×H j ×W j is the scale; α j is the weight of the j-th layer feature map; is the L2 loss function; Generator Level Loss Function Specifically: in, Represents the discriminator D I judge is the probability of the true outcome; The noise suppression loss function is specifically: in, For The result of finding the absolute value; is the weight map after denoising.

9. A 3D hand reconstruction device suitable for use under complex lighting conditions, characterized in that: include: Data acquisition module, used to obtain event stream data and LDR images of the hand acquired synchronously in time and space The first processing module is used to pre-process the event stream data and the LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand The second processing module is used to convert the HDR image LDR Image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

Images