A 3D hand reconstruction method and device applicable to complex lighting conditions

Abstract

The present invention discloses a method and apparatus for 3D reconstruction of a hand suitable for complex lighting conditions, comprising: obtaining event stream data and an LDR image #imgabs0# of a hand collected synchronously in time and space; preprocessing the event stream data; and inputting the LDR image #imgabs1# and the event stream data into a pre-trained cross-modal adaptive fusion network to obtain an HDR image #imgabs2# of the hand; and inputting the HDR image #imgabs3# and the LDR image #imgabs4# into a pre-trained cascaded 3D reconstruction network of the hand to obtain a 3D hand model M. hand This application is applicable to a method and apparatus for 3D reconstruction of a hand under complex lighting conditions, which improves the robustness of the imaging network in real scenes with extreme lighting, suppresses noise while avoiding the loss of HDR image details, and effectively infers the 3D coordinates of the vertices of the 3D hand mesh.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing technology, and in particular to a method and device for three-dimensional reconstruction of a hand under complex lighting conditions. Background Art

[0002] Three-dimensional hand reconstruction is a fundamental computer vision task with applications in a wide range of scenarios, including human-computer interaction, virtual reality, augmented reality, autonomous driving, and robotic control. Thanks to the recent development of deep learning, 3D hand reconstruction methods have made significant progress. Existing research on traditional frame cameras has proposed and developed numerous 3D hand reconstruction methods using deep neural networks. However, these methods are limited in their applicable scenarios, inheriting the inherent limitations of frame cameras and failing to address severe loss of image detail and texture in backlit and overexposed scenes. While event-based methods can effectively address backlit and overexposed scenes, they struggle to reconstruct hand details due to the sparsity of the event stream. Furthermore, in static scenes, event cameras are unable to generate events, making these methods difficult to work properly. Therefore, research on hand pose estimation that fuses RGB images and events is of great significance. Summary of the Invention

[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a three-dimensional hand reconstruction method and device suitable for complex lighting conditions.

[0004] The present invention proposes a method for 3D reconstruction of a hand suitable for complex lighting conditions, comprising the following steps:

[0005] S1. Acquire event stream data and LDR images of the hand collected synchronously in time and space

[0006] S2. Preprocessing of event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0007] S3, HDR image LDR image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0008] Preferably, the preprocessing specifically includes:

[0009] Perform event compression on the event stream data to obtain the corresponding event voxels

[0010] Preferably, the cross-modal adaptive fusion network includes a feature alignment module, a multi-scale feature extraction module, a cross-modal feature fusion module, and an image reconstruction module; step S2 specifically includes:

[0011] Event voxel and LDR images Input to the feature alignment module to obtain event features and LDR image features

[0012] The event characteristics and LDR image features Input to the multi-scale feature extraction module to obtain multi-scale event features and multi-scale image features

[0013] Multi-scale event features and multi-scale image features Input to the cross-modal feature fusion module, and use the exposure mask M to extract the corresponding exposure mask features for multi-scale image features Perform feature weighting of different exposure areas to obtain exposure-weighted LDR image features

[0014] Exposure-weighted LDR image features Input to the image reconstruction module to organically integrate features of different scales to obtain HDR images

[0015] Preferably, the feature alignment module includes an adversarial feature discriminator D F , event feature encoder T ε , image feature encoder T L , the event voxel and LDR images Input to the feature alignment module to obtain event features and LDR image features Specifically include:

[0016] Event voxel Input event feature encoder T ε , using event feature encoder T ε For event voxels Encode to obtain event characteristics

[0017] And the LDR image Input image feature encoder T L , using the image feature encoder T L For LDR images Encode to obtain LDR image features

[0018] Using adversarial feature discriminator D F The event characteristics and LDR image features Alignment is performed on potentially shared domains.

[0019] Preferably, the cascaded hand 3D reconstruction network includes: a hand detection network D MP , 2D pose estimation network P MP , 3D posture estimation network PoseNet, hand 3D model reconstruction network MeshNet; step S3 specifically includes:

[0020] HDR image Input hand detection network D MP , through the hand detection network D MP Get the hand position coordinates to detect the hand image I hand ;

[0021] The hand image I hand Input 2D pose estimation network P MP , through the 2D pose estimation network P MP For the detected hand image I hand The coordinates of 21 2D skeleton points are used to locate the corresponding 2D posture of the hand

[0022] The 2D posture of the hand Input the 3D pose estimation network PoseNet, and use the 3D pose estimation network PoseNet to transform the 2D pose of the hand Convert it into a 4096-dimensional feature vector, then convert the 4096-dimensional feature vector into a 3J-dimensional vector, and use the 3J-dimensional vector as the three-dimensional posture of the hand Output;

[0023] The 2D posture of the hand 3D hand posture Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand .

[0024] Preferably, the hand three-dimensional model reconstruction network MeshNet includes a fusion posture estimation network and an adaptive graph convolution network; the two-dimensional posture of the hand 3D hand posture Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand , specifically including:

[0025] The 2D posture of the hand Corresponding 2D bone points and 3D hand postures The corresponding 3D skeleton points are input into the fusion pose estimation network to generate the fused pose estimate P∈R J×5 ;

[0026] The fused pose estimate P∈R J×5 Input the adaptive graph convolutional network to obtain the hand 3D model M hand .

[0027] Preferably, the training process of the cross-modal adaptive fusion network specifically includes:

[0028] Collect multiple sets of hand images and mark the hand joints to form a training sample set;

[0029] With the hand image as input and the three-dimensional coordinates of each joint as output, the preset cross-modal adaptive fusion network is weakly supervised trained based on the training sample set to obtain the trained cross-modal adaptive fusion network;

[0030] The loss function used during training as follows:

[0031]

[0032] in, is the loss function for feature alignment; is the content consistency loss function; is the perceptual loss function; is the generator level loss function; is the noise suppression loss function; ω fa 、ω pix 、ω per 、ω ce 、ω de All are weights.

[0033] Preferably, the loss function for feature alignment is Specifically:

[0034]

[0035] in, is the mathematical expectation; is the discriminator D F Determining LDR image features and event characteristics Distribute the same credibility;

[0036] Content consistency loss function Specifically:

[0037]

[0038] in, To restore the image; is the reference image; || ||1 is the L1 loss function;

[0039] Perceptual loss function Specifically:

[0040]

[0041] in, is the j-th layer feature map extracted from the image using the pre-trained VGG-19 network; C j ×H j ×W j is the scale; α j is the weight of the j-th layer feature map; is the L2 loss function;

[0042] Generator-level loss function Specifically:

[0043]

[0044] in, Denotes the discriminator D I judge is the probability of the true outcome;

[0045] The noise suppression loss function is specifically:

[0046]

[0047] in, For The result of finding the absolute value; is the weight map after denoising.

[0048] The present invention proposes a three-dimensional hand reconstruction device suitable for complex lighting conditions, comprising:

[0049] Data acquisition module, used to obtain event stream data and LDR images of the hand collected synchronously in time and space

[0050] The first processing module is used to pre-process the event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0051] The second processing module is used to convert the HDR image LDR image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0052] In the present invention, the proposed method and device for three-dimensional reconstruction of hands under complex lighting conditions greatly enhances the ability to cope with complex lighting factors by introducing an event camera as a supplement to the traditional camera. At the same time, due to the characteristics of the event camera's high-speed perception of moving targets, the network pays more attention to the target's motion information and better extracts its semantic features, enhancing the network model's ability to process dynamic gestures and laying the foundation for three-dimensional reconstruction of hands in motion. The cross-modal adaptive fusion network based on the exposure attention mechanism establishes an adaptive fusion mechanism for event streams and images under complex lighting conditions by designing an implicit feature alignment strategy and an adaptive fusion method based on the exposure attention mechanism, significantly improving the robustness of the imaging network in real scenes with extreme lighting. The model optimization strategy based on image detail enhancement and noise suppression enables the network to effectively cope with multi-source mixed noise from LDR images and event streams under extreme lighting conditions. The gradient penalty term is constructed by using the distribution difference between image content and noise, which suppresses noise while avoiding the loss of HDR image details. The cascaded 3D hand reconstruction network uses the adjacency matrix to describe the topological structure of the hand skeleton points, and through multiple layers of graph convolution operations, gradually propagates and aggregates the information of the skeleton points, thereby effectively inferring the 3D coordinates of the vertices of the 3D hand mesh. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of the processing flow structure of a method for 3D reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0054] Figure 2 This is a schematic diagram of the distribution structure of acquisition equipment for a method of 3D hand reconstruction under complex lighting conditions proposed by the present invention;

[0055] Figure 3 This is a schematic diagram of the HDR image generation process structure of a method for 3D reconstruction of a hand under complex lighting conditions proposed by the present invention;

[0056] Figure 4 This is a schematic diagram of the feature extraction structure of a 3D hand reconstruction method suitable for complex lighting conditions proposed by the present invention;

[0057] Figure 5 This is a schematic diagram of an exposure fusion structure of a method for 3D hand reconstruction under complex lighting conditions proposed by the present invention;

[0058] Figure 6This is a schematic diagram of the image reconstruction structure of a 3D hand reconstruction method suitable for complex lighting conditions proposed by the present invention;

[0059] Figure 7 This is a schematic diagram of the cascaded 3D hand reconstruction process structure of a 3D hand reconstruction method suitable for complex lighting conditions proposed by the present invention;

[0060] Figure 8 This is a schematic diagram of the overall MeshNet process structure of a 3D hand reconstruction method suitable for complex lighting conditions proposed by the present invention;

[0061] Figure 9 This is a schematic diagram of the implementation optimization process structure of a method for 3D hand reconstruction suitable for complex lighting conditions proposed by the present invention;

[0062] Figure 10 This is a schematic diagram of the processing steps of a three-dimensional hand reconstruction method suitable for complex lighting conditions proposed by the present invention;

[0063] Figure 11 This is a schematic diagram of the system architecture of a three-dimensional hand reconstruction device proposed by the present invention, which is suitable for use under complex lighting conditions. DETAILED DESCRIPTION

[0064] Reference Figure 1-11 The present invention proposes a method for 3D reconstruction of a hand under complex lighting conditions, comprising the following steps:

[0065] S1. Acquire event stream data and LDR images of the hand collected synchronously in time and space

[0066] In this embodiment, the preprocessing specifically includes:

[0067] Perform event compression on the event stream data to obtain the corresponding event voxels

[0068] In this embodiment, a hybrid imaging device is constructed to capture a 3D hand reconstruction dataset under different lighting conditions. The hybrid imaging device consists of a traditional RGB camera and an event camera. Hardware-triggered time synchronization technology and a spatial calibration method based on spatial homography transformation are implemented to ensure spatiotemporal matching of the collected cross-modal data. The present invention designs a hardware clock-triggered synchronization method to achieve precise time alignment during data acquisition between the traditional camera and the event camera. The STM32F103C8T6 is used as the master controller to construct a timer-triggered dual-camera system. In practical applications, the timer configuration can be flexibly adjusted to precisely control the exposure time, mode, and frame rate to ensure data quality. Given the different voltage reference signals caused by different camera types, the system design differentiates the reference voltage configuration to ensure stable and reliable camera synchronization triggering. Upon receiving the synchronization trigger signal, the two cameras will capture synchronously and transmit the imaging information and trigger timestamp to a computer. To ensure consistent fields of view between the two cameras in the hybrid imaging device, the present invention utilizes a spatial calibration method based on spatial homography transformation. Given the internal and external parameters of the event camera and the RGB camera, spatial alignment between the camera outputs can be achieved by calculating the homography transformation matrix. To facilitate the homography estimation between event streams and RGB image frames, the present invention utilizes the E2VID algorithm to reconstruct an intensity image from the event stream, converts the event stream into a visual representation similar to the RGB image frame, and then performs homography estimation to ensure that the scene targets captured by the camera are spatially consistent.

[0069] In this embodiment, equipment is set up: the acquisition equipment is as follows: Figure 2As shown, a spectroscope is used to split the light in the same field of view into two beams, which are collected by an RGB traditional camera and an event camera respectively. Time synchronization: A hardware clock trigger synchronization method is designed to ensure that the traditional camera and the event camera can achieve precise time alignment during data acquisition. The present invention uses STM32F103C8T6 as the main control unit and constructs a timer to trigger the two cameras. In practical applications, according to the needs of specific scenarios, the configuration of the timer can be flexibly adjusted to achieve precise control of exposure time, exposure mode and frame rate to ensure the quality of collected data. When the synchronization trigger signal is received, the two cameras will shoot synchronously and transmit the imaging information and the corresponding trigger timestamp to the acquisition device. Spatial synchronization: The present invention introduces a calibration method based on spatial homography transformation. By accurately calculating the internal and external parameters of the event camera and the RGB camera, the homography transformation matrix is obtained to achieve precise spatial alignment between the camera outputs. In order to simplify the homography estimation process of the event stream and the RGB image frame, the present invention uses the E2VID algorithm to convert the event stream into a visual form similar to the RGB image frame. After this conversion, homography estimation is performed to ensure that the scene targets captured by the camera maintain a high degree of consistency in spatial dimensions. A cross-modal adaptive fusion network based on the exposure attention mechanism is constructed. The network includes a "feature alignment and multi-scale feature extraction module based on domain adaptation" and an "adaptive fusion reconstruction module based on the exposure attention mechanism". The network block diagram is shown below. Figure 3 shown.

[0070] In this embodiment, the event is first evenly divided into five time intervals on the time axis, and then the corresponding event points are accumulated in each time interval according to the polarity of the event. The frame compression process of each polarity event in a specific time interval can be expressed as:

[0071]

[0072] in, represents the event frame compression result of the scale 2B×H×W, 2 represents the positive and negative polarity of the event, and H and W represent the spatial resolution of the event. i ,t i ) represents t i The pixel point p(x i ,t i ), m∈{0,1,…,B-1} represents the mth time interval, and t i It means that in the time interval The sum of events in B is set to 5.

[0073] S2. Preprocessing of event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0074] In this embodiment, the cross-modal adaptive fusion network includes a feature alignment module, a multi-scale feature extraction module, a cross-modal feature fusion module, and an image reconstruction module; Step S2 specifically includes: and LDR images Input to the feature alignment module to obtain event features and LDR image features The event characteristics and LDR image features Input to the multi-scale feature extraction module to obtain multi-scale event features and multi-scale image features Multi-scale event features and multi-scale image features Input to the cross-modal feature fusion module, and use the exposure mask M to extract the corresponding exposure mask features for multi-scale image features Perform feature weighting of different exposure areas to obtain exposure-weighted LDR image features Exposure-weighted LDR image features Input to the image reconstruction module to organically integrate features of different scales to obtain HDR images

[0075] In this example, a feature discriminator is first introduced to narrow the distribution distance between LDR image features and event features, achieving cross-modal feature alignment in a shared domain. A multi-scale event feature extractor and a multi-scale image feature extractor are then constructed to mine the dynamic range information contained in LDR images and events. The latter first extracts an exposure mask from the LDR image that reflects the brightness saturation of different regions. Then, through exposure mask-guided cross-modal feature fusion and multi-scale feature fusion, HDR images can be effectively reconstructed under different lighting conditions.

[0076] In this embodiment, cross-modal feature alignment: the present invention encodes event voxels and LDR images into a potential shared domain, and introduces an adversarial feature discriminator to bring the feature distributions of the two closer, such as Figure 4 Then the event feature encoder T is used ε Encode event features and use image feature encoder T L Encode the features of LDR images. Event features and LDR image features It can be expressed as:

[0077]

[0078] In this embodiment, the feature alignment module includes an adversarial feature discriminator D F , event feature encoder T ε , image feature encoder T L , the event voxel and LDR images Input to the feature alignment module to obtain event features and LDR image features Specifically include: event voxels Input event feature encoder T ε , using event feature encoder T ε For event voxels Encode to obtain event characteristics And the LDR image Input image feature encoder T L , using the image feature encoder T L For LDR images Encode to obtain LDR image features Using adversarial feature discriminator D F The event characteristics and LDR image features Alignment is performed on potentially shared domains.

[0079] In this embodiment, multi-scale feature extraction: the present invention constructs a multi-scale event feature extraction module and a multi-scale image feature extraction module to realize the extraction of deep and shallow features of RGB images and event voxels. The multi-scale feature extraction module is mainly composed of a convolution module (Conv Block), a downsampling layer (Downsample), a residual group (Residual Group) and a dense fusion module (Dense Fusion). The convolution module and the downsampling layer are responsible for mapping the LDR image and events to different depths and scales, and then the residual group extracts the feature representation of each depth and scale. In order to compensate for the spatial information that may be lost in the downsampling process, the present invention uses a dense fusion module to ensure effective connection and information exchange between non-adjacent level features. The multi-scale feature extraction process can be expressed as:

[0080]

[0081] in, Represents multi-scale event characteristics, Represents multi-scale image features.

[0082] In this embodiment, exposure mask generation: The present invention introduces an exposure mask M, which dynamically adjusts the weights of different regions in the fusion process according to the brightness information of the LDR image. The exposure mask is generated by a segmented mapping function, which takes the brightness value of the LDR image as input and outputs the corresponding mask value. The mapping function is as follows: Figure 5 (a) is shown, and its calculation formula is as follows:

[0083]

[0084] in, represents the luminance channel of the LDR image, and δ is set to 0.7.

[0085] In this embodiment, exposure mask guided cross-modal feature fusion: The present invention designs an exposure mask guided cross-modal feature fusion module (EAF) for cross-modal feature fusion. Figure 5 As shown in (b), this module uses LDR multi-scale image features and multi-scale event characteristics As input, the exposure mask is used to promote The adaptive transformation of effectively realizes the feature weighting of different exposure areas. Specifically, given the LDR image feature The exposure mask is downsampled to match The size of the exposure mask is extracted and the corresponding exposure mask features are then used to Scaling and shifting operations are performed to generate exposure-weighted LDR image features. The process can be expressed as:

[0086]

[0087] In this embodiment, image reconstruction: The image reconstruction module of the present invention is composed of an exposure mask guided cross-modal feature fusion module (EAF), a residual group, a convolution module, a deconvolution module and a dense fusion module, such as Figure 6 As shown. The role of the EAF module is to use the exposure mask to adaptively realize the fusion of LDR image features and event features. The residual module plays the role of extracting and connecting information of the same scale, effectively enhancing the reconstruction ability of HDR features at the spatial level. At the same time, the deconvolution operation gradually restores the spatial resolution of the feature map, gradually revealing the details and structural information of the image. In addition, the dense fusion module transfers deep semantic features to shallow texture features as an error correction feedback mechanism to refine and enhance image texture details to obtain better reconstruction effects. Through the synergistic effect of this series of modules, the organic integration of features of different scales is effectively achieved, and finally the convolution module is used to generate information-rich and pleasing HDR images. The process can be expressed as:

[0088]

[0089] S3, HDR image LDR image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0090] In this embodiment, the cascaded hand 3D reconstruction network includes: hand detection network D MP , 2D pose estimation network P MP , 3D posture estimation network PoseNet, hand 3D model reconstruction network MeshNet; step S3 specifically includes:

[0091] Convert HDR image Input hand detection network D MP , through the hand detection network D MP Get the hand position coordinates to detect the hand image I hand ;

[0092] The hand image I hand Input 2D pose estimation network P MP , through the 2D pose estimation network P MP For the detected hand image I hand The coordinates of 21 2D skeleton points are used to locate the corresponding 2D posture of the hand

[0093] The 2D posture of the hand Input the 3D pose estimation network PoseNet, and use the 3D pose estimation network PoseNet to transform the 2D pose of the hand Convert it into a 4096-dimensional feature vector, then convert the 4096-dimensional feature vector into a 3J-dimensional vector, and use the 3J-dimensional vector as the three-dimensional posture of the hand Output;

[0094] The 2D posture of the hand 3D hand posture Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand .

[0095] In this embodiment, the hand 3D model reconstruction network MeshNet includes a fusion posture estimation network and an adaptive graph convolution network; the 2D posture of the hand 3D hand posture Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand , specifically including:

[0096] The 2D posture of the hand Corresponding 2D bone points and 3D hand postures The corresponding 3D skeleton points are input into the fusion pose estimation network to generate the fused pose estimate P∈R J×5 ;

[0097] The fused pose estimate P∈R J×5 Input the adaptive graph convolutional network to obtain the hand 3D model M hand .

[0098] In this embodiment, the training process of the cross-modal adaptive fusion network specifically includes: collecting multiple groups of hand images and marking the hand joints to form a training sample set; using the hand images as input and the three-dimensional coordinates of each joint as output, weakly supervised training is performed on the preset cross-modal adaptive fusion network based on the training sample set to obtain a trained cross-modal adaptive fusion network.

[0099] In this embodiment, the loss function used in the training process is as follows:

[0100]

[0101] in, is the loss function for feature alignment; is the content consistency loss function; is the perceptual loss function; is the generator level loss function; is the noise suppression loss function; ω fa 、ω pix 、ω per 、ω ce 、ω de All are weights.

[0102] In this embodiment, a model optimization strategy based on image detail enhancement and noise suppression is constructed. To optimize the above network architecture, the present invention designs feature alignment constraints, content consistency constraints, and noise suppression constraints from three perspectives: cross-domain distribution closeness, image content reconstruction, and noise suppression.

[0103] The feature alignment constraint introduces a feature discriminator to determine the distributional similarity between LDR image features and event features. Here, the generator and the discriminator engage in adversarial learning. As training progresses, the semantic information contained in the features generated by the generator gradually converges.

[0104] The content consistency constraint is used to reduce the distance between the output HDR image and the real HDR image. In view of the importance of pixel-level consistency in the image reconstruction task, the present invention adopts Manhattan distance (L1 loss) to constrain this process at the pixel level. At the same time, in order to enhance the content and structural information of the image and generate results that are more in line with human eye perception, the present invention further adopts perceptual loss to optimize the network. In order to alleviate the problem that the restoration result is not natural enough in color, the present invention designs an image discriminator and introduces pixel-level adversarial loss to effectively improve the color fidelity and realism of the final HDR image, thereby obtaining a more attractive visual result. The reconstructed HDR image is often affected by multi-source mixed noise from LDR images and event streams. In order to effectively suppress these noises, the present invention proposes noise suppression loss to further constrain the output image at the gradient level. This loss uses the difference in image content and noise distribution to construct a gradient penalty term, aiming to suppress noise while avoiding the loss of HDR image details.

[0105] In this embodiment, the loss function of feature alignment is Specifically:

[0106]

[0107] in, is the mathematical expectation; is the discriminator D F Determining LDR image features and event characteristics Distribute the same credibility;

[0108] Content consistency loss function Specifically:

[0109]

[0110] in, To restore the image; is the reference image; || ||1 is the L1 loss function;

[0111] Perceptual loss function Specifically:

[0112]

[0113] in, is the j-th layer feature map extracted from the image using the pre-trained VGG-19 network; C j ×H j ×W j is the scale; α j is the weight of the j-th layer feature map; is the L2 loss function;

[0114] Generator-level loss function Specifically:

[0115]

[0116] in, Denotes the discriminator D I judge is the probability of the true outcome;

[0117] The noise suppression loss function is specifically:

[0118]

[0119] in, For The result of finding the absolute value; is the weight map after denoising.

[0120] In this embodiment, the cascaded 3D hand reconstruction network extracts the 3D key bone points of the hand from the enhanced HDR image in a cascade manner and constructs a 3D hand model. First, a hand area detection model is constructed to obtain the palm image from the input image, and then a 2D bone point extraction model is constructed to obtain the coordinates of the key bone points. The 3D bone points are obtained by upgrading the 2D bone points, and finally the 2D and 3D bone points are used to construct a 3D hand model. The cascaded 3D hand reconstruction network includes three core components: MediaPipe Hands, PoseNet, and MeshNet. MediaPipe Hands is used to obtain the hand position and estimate the 2D hand posture from the input RGB image; PoseNet upgrades these 2D bone points to 3D hand bone points; and MeshNet uses a combination of 2D and 3D bone point information of the hand to finely estimate the 3D mesh of the hand from coarse to fine. The overall process is as follows: Figure 7 shown.

[0121] In this embodiment, the hand 2D posture estimation: the present invention uses the MediaPipe Hands hand detection network D MP After obtaining the hand position coordinates, the detector model uses a feature extractor based on the encoding / decoding structure to achieve greater scene context awareness. The process can be expressed as:

[0122]

[0123] in, For the enhanced HDR image, I hand is the hand image detected.

[0124] Using MediaPipe Hands 2D pose estimation network PMP The process of accurately locating the coordinates of the 21 two-dimensional skeleton points of the detected hand image can be described as follows:

[0125]

[0126] In this embodiment, the two-dimensional hand skeleton point is upgraded: the present invention uses the PoseNet two-dimensional hand skeleton point upgrade network to convert the normalized two-dimensional input posture Converted to a 4096-dimensional feature vector. Then, through two residual blocks and a fully connected layer, the feature vector is converted to a 3J-dimensional vector, represented as Where j represents the number of hand joints. The process can be expressed as:

[0127]

[0128] In this embodiment, the hand 3D model is reconstructed: the present invention uses the MeshNet hand 3D model reconstruction network to estimate the hand 3D posture from the 2D posture. and 3D pose estimation Specifically, the 2D skeleton points and 3D skeleton points are concatenated to generate the fused pose estimate P∈R J×5 , then estimate the root joint related 3D hand mesh M∈R from P V×3 , where V represents the number of mesh vertices. The overall process of MeshNet is as follows Figure 8 As shown in the figure, it is mainly composed of an adaptive graph convolutional network, which uses the adjacency matrix to describe the topological structure of the skeleton points, and through multi-layer graph convolution operations, gradually propagates and aggregates the information of the hand skeleton points, thereby reconstructing the 3D hand model. hand The reconstruction process is expressed as:

[0129]

[0130] In this embodiment, the main purpose is to realize the transplantation and acceleration of the algorithm on the edge device to ensure the application efficiency. The overall solution is as follows: Figure 9 As shown in the figure, this specifically includes: implementing algorithm model optimization and conversion, optimizing and accelerating the model within the PyTorch framework, and porting the model to the AI chip through model conversion. Implementing algorithm hardware inference, feeding pre-processed data into the AI chip for algorithm forward inference, and obtaining the final result through post-processing.

[0131] Reference Figure 1-11 The present invention proposes a three-dimensional hand reconstruction device suitable for complex lighting conditions, comprising:

[0132] Data acquisition module, used to obtain event stream data and LDR images of the hand collected synchronously in time and space

[0133] The first processing module is used to pre-process the event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand

[0134] The second processing module is used to convert the HDR image LDR image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand .

[0135] In the specific working process of the method and device for 3D reconstruction of a hand under complex lighting conditions of this embodiment,

[0136] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A 3D hand reconstruction method suitable for complex lighting conditions, characterized by: The following steps are involved: S1. Acquire event stream data and LDR images of the hand collected synchronously in time and space S2. Preprocessing of event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand S3, HDR image LDR image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand ; The cross-modal adaptive fusion network includes a feature alignment module, a multi-scale feature extraction module, a cross-modal feature fusion module, and an image reconstruction module; Step S2 specifically includes: Event voxel and LDR images Input to the feature alignment module to obtain event features and LDR image features The event characteristics and LDR image features Input to the multi-scale feature extraction module to obtain multi-scale event features and multi-scale image features Multi-scale event features and multi-scale image features Input to the cross-modal feature fusion module, and use the exposure mask M to extract the corresponding exposure mask features for multi-scale image features Perform feature weighting of different exposure areas to obtain exposure-weighted LDR image features Exposure-weighted LDR image features Input to the image reconstruction module to organically integrate features of different scales to obtain HDR images The feature alignment module includes the adversarial feature discriminator D F , event feature encoder T ε , image feature encoder T L , the event voxel and LDR images Input to the feature alignment module to obtain event features and LDR image features Specifically include: Event voxel Input event feature encoder T ε , using event feature encoder T ε For event voxels Encode to obtain event characteristics And the LDR image Input image feature encoder T L , using the image feature encoder T L For LDR images Encode to obtain LDR image features Using adversarial feature discriminator D F The event characteristics and LDR image features Alignment is performed on potentially shared domains.

2. The method for 3D hand reconstruction suitable for complex lighting conditions according to claim 1, characterized in that: The pretreatment specifically includes: Perform event compression on the event stream data to obtain the corresponding event voxels 3. The method for 3D hand reconstruction suitable for complex lighting conditions according to claim 1, characterized in that: The cascaded hand 3D reconstruction network includes: hand detection network D MP , 2D pose estimation network P MP , 3D posture estimation network PoseNet, hand 3D model reconstruction network MeshNet; step S3 specifically includes: Convert HDR image Input hand detection network D MP , through the hand detection network D MP Get the hand position coordinates to detect the hand image I hand ; The hand image I hand Input 2D pose estimation network P MP , through the 2D pose estimation network P MP For the detected hand image I hand The coordinates of 21 2D skeleton points are used to locate the corresponding 2D posture of the hand The 2D posture of the hand Input the 3D pose estimation network PoseNet, and use the 3D pose estimation network PoseNet to transform the 2D pose of the hand Convert it into a 4096-dimensional feature vector, then convert the 4096-dimensional feature vector into a 3J-dimensional vector, and use the 3J-dimensional vector as the three-dimensional posture of the hand Output; The 2D posture of the hand 3D hand posture Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand .

4. The method for 3D hand reconstruction suitable for complex lighting conditions according to claim 3, characterized in that: The hand three-dimensional model reconstruction network MeshNet includes a fusion posture estimation network and an adaptive graph convolution network; the two-dimensional posture of the hand 3D hand posture Input the hand 3D model reconstruction network MeshNet to obtain the hand 3D model M hand , specifically including: The 2D posture of the hand Corresponding 2D bone points and 3D hand postures The corresponding 3D skeleton points are input into the fusion pose estimation network to generate the fused pose estimate P∈R J×5 ; The fused pose estimate P∈R J×5 Input the adaptive graph convolutional network to obtain the hand 3D model M hand .

5. The method for 3D reconstruction of a hand suitable for complex lighting conditions according to claim 1, characterized in that: The training process of the cross-modal adaptive fusion network specifically includes: Collect multiple sets of hand images and mark the hand joints to form a training sample set; With the hand image as input and the three-dimensional coordinates of each joint as output, the preset cross-modal adaptive fusion network is weakly supervised trained based on the training sample set to obtain the trained cross-modal adaptive fusion network; The loss function used during training as follows: in, is the loss function for feature alignment; is the content consistency loss function; is the perceptual loss function; is the generator level loss function; is the noise suppression loss function; ω fa 、ω pix 、ω per 、ω ce 、ω de All are weights.

6. The method for 3D hand reconstruction suitable for complex lighting conditions according to claim 5, characterized in that: Loss function for feature alignment Specifically: in, is the mathematical expectation; is the discriminator D F Determining LDR image features and event characteristics Distribute the same credibility; Content consistency loss function Specifically: in, To restore the image; is the reference image; || ||1 is the L1 loss function; Perceptual loss function Specifically: in, is the j-th layer feature map extracted from the image using the pre-trained VGG-19 network; C j ×H j ×W j is the scale; α j is the weight of the j-th layer feature map; is the L2 loss function; Generator-level loss function Specifically: in, Denotes the discriminator D I judge is the probability of the true outcome; The noise suppression loss function is specifically: in, For The result of finding the absolute value; is the weight map after denoising.

7. A 3D hand reconstruction device suitable for use under complex lighting conditions, characterized in that: include: Data acquisition module, used to obtain event stream data and LDR images of the hand collected synchronously in time and space The first processing module is used to pre-process the event stream data and LDR image Input the pre-trained cross-modal adaptive fusion network to obtain the HDR image of the hand The second processing module is used to convert the HDR image LDR image Input the pre-trained cascaded hand 3D reconstruction network to obtain the hand 3D model M hand ; In the first processing module, the cross-modal adaptive fusion network includes a feature alignment module, a multi-scale feature extraction module, a cross-modal feature fusion module, and an image reconstruction module; The processing process of the first processing module specifically includes: Event voxel and LDR images Input to the feature alignment module to obtain event features and LDR image features The event characteristics and LDR image features Input to the multi-scale feature extraction module to obtain multi-scale event features and multi-scale image features Multi-scale event features and multi-scale image features Input to the cross-modal feature fusion module, and use the exposure mask M to extract the corresponding exposure mask features for multi-scale image features Perform feature weighting of different exposure areas to obtain exposure-weighted LDR image features Exposure-weighted LDR image features Input to the image reconstruction module to organically integrate features of different scales to obtain HDR images Among them, the feature alignment module includes the adversarial feature discriminator D F , event feature encoder T ε , image feature encoder T L , the event voxel and LDR images Input to the feature alignment module to obtain event features and LDR image features Specifically include: Event voxel Input event feature encoder T ε , using event feature encoder T ε For event voxels Encode to obtain event characteristics And the LDR image Input image feature encoder T L , using the image feature encoder T L For LDR images Encode to obtain LDR image features Using adversarial feature discriminator D F The event characteristics and LDR image features Alignment is performed on potentially shared domains.

Images