Target object three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The invention provides a target object three-dimensional reconstruction method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing, and the method comprises the steps: training a Gaussian model based on a first sample set, and obtaining a first Gaussian network; the first sample set comprises a plurality of first RGB images under a dark light condition; training a neural network model based on the second sample set to obtain a dark light enhancement network; inputting the first RGB image and a first heat map corresponding to the first RGB image into a dark light enhancement network, and inputting the output of the dark light enhancement network and the first heat map into a first Gaussian network for joint training to obtain a second Gaussian network; and performing three-dimensional reconstruction on the target object based on the second Gaussian network. According to the target object three-dimensional reconstruction method and device, the electronic equipment and the storage medium provided by the invention, the precision of three-dimensional reconstruction of the target object under the dark light condition can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and more specifically, relates to a method and device for three-dimensional reconstruction of a target object, an electronic device, and a storage medium. Background Art

[0002] Visible light-based scene reconstruction and novel view synthesis technologies have achieved remarkable results, and their application scope has penetrated into many fields, bringing great convenience to people's work and life. In the field of virtual reality (VR), this technology can create an immersive experience, making users feel as if they are in the scene; in the field of autonomous driving, it helps vehicles achieve accurate environmental perception and path planning to ensure driving safety; in the field of robotics, it helps robots better understand the surrounding environment and achieve efficient interaction with the environment; in the construction industry, it can be used for building design and construction monitoring to improve design quality and construction efficiency; in addition, in the field of digital twins, by building digital copies of real scenes, it provides strong support for various decisions.

[0003] 3D Gaussian Speckle (3DGS) By using a large number of 3D Gaussians to represent the scene, real-time rendering can be achieved while maintaining high visual fidelity, making a qualitative leap in the efficiency of 3D reconstruction and view synthesis. However, most current 3DGS methods mainly process RGB image data under ideal lighting conditions. In actual application scenarios, such as at night, in dark indoor environments, or in the presence of severe occlusion, RGB images face many challenges. Due to the photon starvation effect, the performance will be significantly reduced under low light conditions, resulting in increased image noise and reduced contrast, making the outline of the object blurred, and even unable to obtain sufficient effective image information. This greatly limits the application effect and reconstruction accuracy of 3DGS technology in these special scenarios. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method and device for three-dimensional reconstruction of a target object, an electronic device, and a storage medium to improve the accuracy of three-dimensional reconstruction of a target object under dark light conditions.

[0005] A first aspect of an embodiment of the present disclosure provides a method for three-dimensional reconstruction of a target object, comprising: Training a Gaussian model based on a first sample set to obtain a first Gaussian network; the first sample set includes a plurality of first RGB images under dark light conditions; Training a neural network model based on a second sample set to obtain a dark-light enhancement network; the second sample set includes a plurality of groups of image pairs, any group of image pairs including a second RGB image of a first object under dark-light conditions, a third RGB image of the first object under ideal light conditions, and a thermal map of the first object; Inputting the first RGB image and a first heat map corresponding to the first RGB image into the dark light enhancement network, and inputting the output of the dark light enhancement network and the first heat map into the first Gaussian network for joint training to obtain a second Gaussian network; The target object is reconstructed in three dimensions based on the second Gaussian network.

[0006] A second aspect of the embodiments of the present disclosure provides a device for three-dimensional reconstruction of a target object, comprising: A first training module is used to train a Gaussian model based on a first sample set to obtain a first Gaussian network; the first sample set includes a plurality of first RGB images under dark light conditions; A second training module is used to train a neural network model based on a second sample set to obtain a dark-light enhancement network; the second sample set includes a plurality of groups of image pairs, any group of image pairs includes a second RGB image of the first object under dark light conditions, a third RGB image of the first object under ideal light conditions, and a thermal map of the first object; a joint training module, configured to input the first RGB image and a first heat map corresponding to the first RGB image into the dark light enhancement network, and input the output of the dark light enhancement network and the first heat map into the first Gaussian network for joint training to obtain a second Gaussian network; A three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the target object based on the second Gaussian network.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned method for three-dimensional reconstruction of a target object when executing the computer program.

[0008] In a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for three-dimensional reconstruction of a target object are implemented.

[0009] The beneficial effects of the target object 3D reconstruction method and device, electronic device, and storage medium provided by the embodiments of the present disclosure are: In the disclosed embodiment, a Gaussian model is trained using a thermal map and an RGB image to perform three-dimensional reconstruction of a target object based on the Gaussian model. This can fully utilize the characteristic of thermal imaging technology that is not affected by lighting conditions, help identify contour information of objects in low-light environments, and thus improve the accuracy of three-dimensional reconstruction in low-light environments.

[0010] The training process is divided into a pre-training stage and an end-to-end training stage. In the pre-training stage, the Gaussian model is pre-trained to obtain the first Gaussian network, and the neural network model is pre-trained to obtain the dark light enhancement network. The dark light enhancement network can enhance the dark light image and make it closer to the image quality under ideal lighting. The above pre-training steps can reduce the convergence time in the subsequent end-to-end training process. In the end-to-end training stage, the dark light enhancement network and the first Gaussian network are jointly trained. In this process, the dark light enhancement network and the first Gaussian network are optimized through multiple constraints to obtain a second Gaussian network with better performance.

[0011] Therefore, the embodiment of the present disclosure improves the Gaussian model's ability to understand the scene through the dark light enhancement network and the first heat map, and can generate higher quality mesh results, thereby achieving higher precision three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A schematic diagram of a flow chart of a method for three-dimensional reconstruction of a target object provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of an end-to-end training process provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of a dark light enhancement network provided in an embodiment of the present disclosure; Figure 4 A structural block diagram of a target object three-dimensional reconstruction device provided by an embodiment of the present disclosure; Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0015] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0016] Please refer to Figure 1 , Figure 1 A schematic diagram of a process for three-dimensional reconstruction of a target object provided by an embodiment of the present disclosure, the method comprising: S101: Training a Gaussian model based on a first sample set to obtain a first Gaussian network; the first sample set includes a plurality of first RGB images under dark light conditions.

[0017] In this embodiment, the Gaussian model is trained using the heat map and the RGB image to perform three-dimensional reconstruction of the target object based on the trained Gaussian model. The training process is divided into two stages, namely, a pre-training stage and an end-to-end training stage. In the pre-training stage, the first Gaussian network and the dark light enhancement network are pre-trained to reduce the convergence time of the subsequent end-to-end training process; in the end-to-end training stage, the heat map and the RGB image that has passed the dark light enhancement network are used to fine-tune the pre-trained first Gaussian network to obtain the second Gaussian network.

[0018] Specifically, multiple first RGB images under dark light conditions can be collected in advance as the first sample set, and the Gaussian model can be trained based on the first sample set to enable the Gaussian model to learn the characteristics and distribution rules of the RGB images under dark light conditions. For example, the Gaussian model can learn the approximate shape and contour information of objects under dark light, which can be used as the basic information for building a mesh. Among them, mesh is a geometric structure that discretizes objects or spaces. It consists of a series of vertices, edges, and faces, and is used to accurately describe the shape and surface of an object. A high-quality mesh can accurately present the details and complex geometric features of an object, and is the basic information for 3D modeling.

[0019] Considering that it is not practical to achieve the effect of ideal illumination images with dark-light images when the ideal illumination images are known, this embodiment only uses heat maps and dark-light RGB images, and does not use ideal illumination images, in the end-to-end training stage. Correspondingly, in the pre-training stage, the Gaussian model is trained using a first sample set consisting of multiple dark-light RGB images. Although this may result in the lack of texture details and inaccurate depth estimation during the training process, initialization data can be established based on some observable information, and the dark-light enhanced RGB images have better perspective consistency than the end-to-end training stage.

[0020] S102: training a neural network model based on a second sample set to obtain a dark-light enhancement network; the second sample set includes a plurality of groups of image pairs, any group of image pairs including a second RGB image of a first object under dark-light conditions, a third RGB image of the first object under ideal lighting conditions, and a thermal map of the first object.

[0021] In this embodiment, a second RGB image of the same object (for example, the first object) under dark light conditions, a third RGB image under ideal light conditions, and a thermal map may be collected in advance, and the above images corresponding to the same object may be used as a group of image pairs. Multiple groups of image pairs corresponding to multiple objects may be collected using the same method to obtain a second sample training set.

[0022] Based on the second sample set, the neural network model is trained to obtain a dark light enhancement network. The neural network model here can be implemented using a retinal cortex network (Retinex-Net), a latent decoupled enhancement network (LDE-Net), a lightweight YUVTransformer network (LYT-Net), etc.

[0023] During the training process, the characteristic of heat map that can obtain the outline of objects under dark light conditions is used to integrate it into the dark light enhancement network to improve the effect of dark light enhancement and make the output image of the dark light enhancement network close to the third RGB image.

[0024] S103: Input the first RGB image and the first heat map corresponding to the first RGB image into a dark light enhancement network, input the output of the dark light enhancement network and the first heat map into a first Gaussian network for joint training, and obtain a second Gaussian network.

[0025] Please refer to Figure 2 , Figure 2 A schematic diagram of an end-to-end training process provided for an embodiment of the present disclosure. In the pre-training stage, we use the first RGB image to train the first Gaussian network in advance. However, the first Gaussian network is a Gaussian model that inputs a dark-light RGB image and outputs a dark-light RGB image, and our goal is to be able to input a dark-light RGB image and a heat map, and output a dark-light enhanced image and a heat map. Therefore, in the end-to-end training stage, the first RGB image and the first heat map corresponding to the first RGB image are input into the dark-light enhanced network, and the output of the dark-light enhanced network and the first heat map are input into the first Gaussian network for joint training. The dark-light enhanced network is used to make more RGB information input into the first Gaussian network, and the three-dimensional Gaussian sphere generated by the first Gaussian network is adjusted in combination with the information of the first heat map.

[0026] During the end-to-end training phase, the dark light enhancement network and the first heat map are used to improve the Gaussian model's ability to understand the scene, which can generate higher quality mesh results and achieve higher precision 3D reconstruction.

[0027] S104: Perform three-dimensional reconstruction of the target object based on the second Gaussian network.

[0028] In this embodiment, after end-to-end joint training, the second Gaussian network has learned richer and more accurate image feature information, including the features of the dark light image enhancement and the auxiliary information provided by the heat map. Using this information, the second Gaussian network can more accurately obtain information such as the shape, position and texture of the target object, thereby performing 3D reconstruction of the target object.

[0029] From the above, it can be concluded that the present embodiment uses the thermal map and RGB image to train the Gaussian model to perform three-dimensional reconstruction of the target object based on the Gaussian model, which can make full use of the characteristic that the thermal imaging technology is not affected by the lighting conditions, and help identify the contour information of the object in the low-light environment, thereby improving the accuracy of the three-dimensional reconstruction in the low-light environment.

[0030] The training process is divided into a pre-training stage and an end-to-end training stage. In the pre-training stage, the Gaussian model is pre-trained to obtain the first Gaussian network, and the neural network model is pre-trained to obtain the dark light enhancement network. The dark light enhancement network can enhance the dark light image and make it closer to the image quality under ideal lighting. The above pre-training steps can reduce the convergence time of the subsequent end-to-end training process. In the end-to-end training stage, the dark light enhancement network and the first Gaussian network are jointly trained. In this process, the dark light enhancement network and the first Gaussian network are optimized through multiple constraints to obtain a second Gaussian network with better performance.

[0031] Therefore, this embodiment improves the Gaussian model's ability to understand the scene through the dark light enhancement network and the first heat map, and can generate higher quality mesh results, thereby achieving higher precision three-dimensional reconstruction.

[0032] In one embodiment of the present disclosure, the first RGB image is different from the second RGB image.

[0033] In this embodiment, a supervised learning method is used to train the dark light enhancement network, that is, the third RGB image under ideal lighting conditions is used as the true value of the enhanced second RGB image under dark light conditions to supervise the dark light enhancement effect.

[0034] Considering that in the end-to-end training stage, the goal of this embodiment is to perform self-supervision, that is, to achieve high-quality enhancement of dark-light images without ideal illumination images supervising dark-light images, this embodiment will not use ideal illumination images at this stage. In the pre-training stage, it is only to train a dark-light enhancement network in advance, so the paired second RGB image under dark-light conditions and the third RGB image under ideal illumination conditions are used for constraints.

[0035] It can be concluded from the above that in the pre-training stage, this embodiment uses different scenes (that is, different RGB images) from those in the end-to-end training stage for supervised training, so that a better dark light enhancement network can be obtained.

[0036] In one embodiment of the present disclosure, training a neural network model based on the second sample set includes: Execute the parameter adjustment operation multiple times until the stop condition is met; Parameter adjustment operations include: enhancing the second RGB image of the first object under dark light conditions to obtain a first enhanced image; fusing the thermal map of the first object and the first enhanced image to obtain a second enhanced image; determining a first loss function based on the second enhanced image and a third RGB image of the first object under ideal lighting conditions, and adjusting parameters of the neural network model based on the first loss function; The stopping condition is: the first loss function is less than the first threshold, or the number of iterations reaches the set number.

[0037] Please refer to Figure 3 , Figure 3 A schematic diagram of a dark light enhancement network provided for an embodiment of the present disclosure. In this embodiment, the second RGB image under dark light conditions can be enhanced based on a neural network model such as a retinal cortex network (Retinex-Net), a latent decoupled enhancement network (LDE-Net), or a lightweight YUV Transformer network (LYT-Net) to obtain a first enhanced image; then the heat map is integrated into the first enhanced image to improve the dark light enhancement effect, so that the obtained second enhanced image approaches the third RGB image under ideal lighting conditions.

[0038] Among them, the fusion of the heat map and the first enhanced image can be achieved by using networks such as a generative adversarial network (GAN) or a variational autoencoder (VAE).

[0039] Specifically, the training process repeatedly performs parameter adjustment operations to continuously iterate and optimize the parameters of the neural network model to achieve a better image enhancement effect. In each iteration, the first loss function is calculated based on the difference between the output of the neural network model (i.e., the second enhanced image) and the true value (i.e., the third RGB image under ideal lighting conditions). The condition for stopping the iteration can be that the first loss function is less than a pre-set first threshold, or the number of iterations reaches a set number.

[0040] From the above, we can conclude that this example optimizes the image enhancement effect step by step by iteratively adjusting the parameters of the neural network model and comprehensively utilizing the information of the dark-light RGB image and the heat map, so that the model can generate high-quality images under close-to-ideal lighting conditions when facing new dark-light images.

[0041] In one embodiment of the present disclosure, enhancing a second RGB image of a first object under dark light conditions to obtain a first enhanced image includes: Decomposing a second RGB image of the first object under dark light conditions into a reflectance map and an illumination map; Enhance the reflection map and illumination map separately; The enhanced reflection map and the enhanced illumination map are synthesized into a first enhanced image.

[0042] In this embodiment, a specific implementation method for enhancing the second RGB image under dark light conditions is provided. Based on the retinal cortex network (Retinex-Net), the second RGB image can be decomposed into a reflection map and an illumination map, and then the reflection map and the illumination map from the second RGB image are enhanced respectively, and then the enhanced images are merged into the first enhanced image.

[0043] Among them, enhancing the decomposed reflection map can highlight the texture, edge and other detailed information of the object, and enhancing the decomposed illumination map can adjust the problem of uneven illumination and make the details in the dark appear.

[0044] It can be concluded from the above that this embodiment decomposes the second RGB image into a reflection map and an illumination map, and then enhances the reflection map and the illumination map respectively, so that the enhanced image can contain more details and more accurate illumination information.

[0045] On this basis, the first loss function of the dark light enhancement network includes reconstruction loss, reflection invariance loss, lighting smoothness loss, self-feature preservation loss and enhancement loss. The calculation process of each loss function is as follows: (1) Reconstruction loss: According to the Retinex theory, the decomposed reflection map R and illumination map I can be recombined into the original image. , which ensures the integrity of the information and thus the reconstruction loss The calculation formula is as follows:

[0046] In the formula, is the reflection map of the dark light image, is the reflectance map of the ideal lighting image, is the illumination map of the dark-light image, is the illumination map of the ideal illumination image, is the dark light image (i.e. the second RGB image), is the ideal illumination image (that is, the third RGB image).

[0047] (2) Reflection invariance loss: According to the Retinex theory, reflectivity reflects the inherent characteristics of the object itself and is not affected by lighting and imaging equipment. In other words, the reflectance maps obtained after decomposing images captured under different lighting conditions in the same scene should be as consistent as possible, that is, the reflectance map of the dark light image obtained by network decomposition The reflection map should be as close to the ideal lighting image as possible. and reflectance maps of dark-light-enhanced images According to this, the reflection invariant loss The calculation formula is as follows:

[0048] In the formula, about the ideal illumination image reflection map This part is only used in the pre-training stage, because in the end-to-end training, we will not use the ideal lighting images for supervision. Dark light enhanced images used in the pre-training stage , represents the first enhanced image obtained by the dark light enhancement network in the end-to-end stage.

[0049] (3) Illumination smoothness loss: In order to better guide the network decomposition to obtain the illumination map, we use the loss in the darkness-free infrared and visible image fusion (DIVFusion) method. and loss :

[0050] Where ∇ is the Sobel operator, including the x and y directions, Indicates the exponential calculation with natural logarithm e as base; is a small positive constant to prevent division by zero; is a parameter that controls the shape of the mutual consistency loss, aiming to increase the medium gradient part of the image.

[0051] (4) Self-feature preservation loss: To ensure better constraints on end-to-end training when there are no ideal illumination images, we also use the self-feature preservation loss of the deep light enhancement generative adversarial network (EnlightenGAN) without paired supervision to preserve the input dark light images. And the output dark light enhanced image Constrain the VGG feature distance.

[0052]

[0053] Where ϕ represents the feature map extracted by the pre-trained VGG19 network, i represents the i-th maximum pooling layer, j represents the j-th convolutional layer after the i-th maximum pooling layer, i = 5, j = 1; and is the size of the extracted feature map.

[0054] (4) Enhancement loss: For the output second enhanced image (i.e. Figure 3 The final enhanced image in ) and the real ideal lighting image, we have enhanced the loss, which is only used in the pre-training stage, because in the end-to-end training, we will not use the ideal lighting image for supervision. Including perceptual loss , Loss. Perceived loss Through the pre-trained VGG19 network To estimate the output and truth value The similarities between.

[0055]

[0056] In the formula, and represents the weighted hyperparameter, represents the L2 norm.

[0057] In one embodiment of the present disclosure, the target object three-dimensional reconstruction method further includes: Performing a super-resolution operation on the second heat map output by the second Gaussian network to obtain a third heat map; A super-resolution operation is performed on the fourth RGB image output by the second Gaussian network to obtain a fifth RGB image.

[0058] In this embodiment, based on the RGB image and heat map of the target object, the second Gaussian network can generate a second heat map and a fourth RGB from a new perspective. Furthermore, by super-resolution operation on the second heat map, a high-resolution third heat map can be obtained, and by super-resolution operation on the fourth RGB image, a high-resolution fifth RGB image can be obtained, providing a high-quality data foundation for subsequent image analysis and processing.

[0059] Specifically, the super-resolution operations on the second heat map and the fourth RGB image can be achieved through the existing super-resolution network. In the end-to-end training stage, the dark light enhancement network, the first Gaussian network and the two super-resolution networks are jointly trained to form an overall network architecture. In this process, various constraints are used to optimize each part.

[0060] Among them, the dark light enhancement network is connected before the first Gaussian network to make the RGB image input to the first Gaussian network have more texture details, which can better help build depth information to generate better mesh effects. The super-resolution network is connected after the first Gaussian network to reduce the memory consumption required for training the first Gaussian network, increase the training speed, and achieve a more economical and feasible training process, avoiding unnecessary consumption caused by inputting high-resolution images into the first Gaussian network. The first Gaussian network can constrain the perspective consistency of the images generated by the dark light enhancement network and the super-resolution network, so that the images are in a three-dimensional scene, rather than discrete, to avoid problems such as artifacts generated by new perspectives and deterioration of mesh quality.

[0061] From the above, it can be concluded that this embodiment forms an overall network architecture by jointly training the dark light enhancement network, the first Gaussian network and the two super-resolution networks, which can realize the use of thermal maps and RGB images in dark light scenes to obtain high-resolution images and high-resolution thermal maps with new perspectives, and thus obtain high-quality mesh results.

[0062] In one embodiment of the present disclosure, the target object three-dimensional reconstruction method further includes: A super-resolution operation of guiding the second heat map based on the fourth RGB image; A super-resolution operation of the fourth RGB image is guided based on the second heat map.

[0063] In this embodiment, the RGB image has rich visual detail information such as texture and color. The super-resolution operation of the second heat map is guided by the fourth RGB image, which can provide more accurate structure and edge information guidance for the heat map super-resolution. At the same time, the second heat map can clearly identify the contour information of objects in a dark environment. The super-resolution operation of the fourth RGB image is guided by the second heat map, which can provide auxiliary information for the super-resolution operation of the fourth RGB image to supplement the details.

[0064] It can be concluded from the above that in the super-resolution operation on the second heat map and the super-resolution operation on the fourth RGB image, this embodiment uses another modality to supplement the detail information that the modality itself has not captured, which is beneficial to improving the accuracy of the super-resolution operation.

[0065] In one embodiment of the present disclosure, the loss function of the first Gaussian network is specifically:

[0066] in, represents the loss function of the first Gaussian network, represents the inverse tone curve, represents the pixel value of the true value image, represents the pixel value of the output image, represents the L1 norm, represents a hyperparameter.

[0067] In this embodiment, considering that when calculating the loss function of the Gaussian model based on the existing MSE loss function or L1 loss function, the weight of the pixels in the dark is often too small, while the weight of the bright part is larger, which will cause some deviations in the training of the dark light image. Therefore, before calculating the loss function of the first Gaussian network, this embodiment first passes the image through an inverse tone curve to rebalance the pixel weights, and finally obtains an improved loss function. Among them, .

[0068] From the above, it can be concluded that this embodiment avoids the problem of too small weights of dark-light pixels by rebalancing the pixel weights through the inverse tone curve, so that the model can more fully utilize the information of dark-light areas during training and reduce the deviation of dark-light image training, thereby more accurately learning the features and patterns in dark-light images and improving the modeling ability of dark-light scenes.

[0069] Corresponding to the target object three-dimensional reconstruction method of the above embodiment, Figure 4 This is a block diagram of a target object 3D reconstruction device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The target object three-dimensional reconstruction device 20 includes: a first training module 21, a second training module 22, a joint training module 23 and a three-dimensional reconstruction module 24.

[0070] The first training module 21 is used to train a Gaussian model based on a first sample set to obtain a first Gaussian network; the first sample set includes a plurality of first RGB images under dark light conditions; A second training module 22 is used to train a neural network model based on a second sample set to obtain a dark-light enhancement network; the second sample set includes a plurality of groups of image pairs, any group of image pairs includes a second RGB image of the first object under dark light conditions, a third RGB image of the first object under ideal light conditions, and a thermal map of the first object; A joint training module 23 is used to input the first RGB image and the first heat map corresponding to the first RGB image into the dark light enhancement network, and input the output of the dark light enhancement network and the first heat map into the first Gaussian network for joint training to obtain a second Gaussian network; The three-dimensional reconstruction module 24 is used to perform three-dimensional reconstruction of the target object based on the second Gaussian network.

[0071] In one embodiment of the present disclosure, the first RGB image is different from the second RGB image.

[0072] In one embodiment of the present disclosure, the second training module 22 is specifically used for: Execute the parameter adjustment operation multiple times until the stop condition is met; Parameter adjustment operations include: enhancing the second RGB image of the first object under dark light conditions to obtain a first enhanced image; fusing the thermal map of the first object and the first enhanced image to obtain a second enhanced image; determining a first loss function based on the second enhanced image and a third RGB image of the first object under ideal lighting conditions, and adjusting parameters of the neural network model based on the first loss function; The stopping condition is: the first loss function is less than the first threshold, or the number of iterations reaches the set number.

[0073] In one embodiment of the present disclosure, the second training module 22 is further configured to: Decomposing a second RGB image of the first object under dark light conditions into a reflectance map and an illumination map; Enhance the reflection map and illumination map separately; The enhanced reflection map and the enhanced illumination map are synthesized into a first enhanced image.

[0074] In one embodiment of the present disclosure, the joint training module 23 is specifically used to: Performing a super-resolution operation on the second heat map output by the second Gaussian network to obtain a third heat map; A super-resolution operation is performed on the fourth RGB image output by the second Gaussian network to obtain a fifth RGB image.

[0075] In one embodiment of the present disclosure, the joint training module 23 is further used to: A super-resolution operation of guiding the second heat map based on the fourth RGB image; A super-resolution operation of the fourth RGB image is guided based on the second heat map.

[0076] In one embodiment of the present disclosure, the loss function of the first Gaussian network is specifically:

[0077] in, represents the loss function of the first Gaussian network, represents the inverse tone curve, represents the pixel value of the true value image, represents the pixel value of the output image, represents the L1 norm, represents a hyperparameter.

[0078] See also Figure 5 , Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 5 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned device embodiments, such as Figure 4 The functions of modules 21 to 24 are shown.

[0079] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0080] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0081] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0082] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the target object three-dimensional reconstruction method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0083] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0084] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0085] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0089] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for three-dimensional reconstruction of a target object, characterized in that: include: Training a Gaussian model based on the first sample set to obtain a first Gaussian network; The first sample set includes a plurality of first RGB images under dark light conditions; Training a neural network model based on a second sample set to obtain a dark-light enhancement network; the second sample set includes a plurality of groups of image pairs, any group of image pairs including a second RGB image of a first object under dark-light conditions, a third RGB image of the first object under ideal light conditions, and a thermal map of the first object; Inputting the first RGB image and a first heat map corresponding to the first RGB image into the dark light enhancement network, and inputting the output of the dark light enhancement network and the first heat map into the first Gaussian network for joint training to obtain a second Gaussian network; The target object is reconstructed in three dimensions based on the second Gaussian network.

2. The method for three-dimensional reconstruction of a target object according to claim 1, characterized in that: The first RGB image is different from the second RGB image.

3. The method for three-dimensional reconstruction of a target object according to claim 1, wherein: The step of training the neural network model based on the second sample set includes: Execute the parameter adjustment operation multiple times until the stop condition is met; The parameter adjustment operation includes: enhancing the second RGB image of the first object under dark light conditions to obtain a first enhanced image; fusing the thermal map of the first object and the first enhanced image to obtain a second enhanced image; determining a first loss function based on the second enhanced image and a third RGB image of the first object under ideal lighting conditions, and adjusting parameters of the neural network model based on the first loss function; The stopping condition is: the first loss function is less than a first threshold, or the number of iterations reaches a set number.

4. The method for three-dimensional reconstruction of a target object according to claim 3, characterized in that: Enhancing the second RGB image of the first object under dark light conditions to obtain a first enhanced image includes: Decomposing a second RGB image of the first object under dark light conditions into a reflectance image and an illumination image; enhancing the reflection map and the illumination map respectively; The enhanced reflection map and the enhanced illumination map are synthesized into a first enhanced image.

5. The method for three-dimensional reconstruction of a target object according to claim 1, characterized in that: Also includes: Performing a super-resolution operation on the second heat map output by the second Gaussian network to obtain a third heat map; Perform a super-resolution operation on the fourth RGB image output by the second Gaussian network to obtain a fifth RGB image.

6. The method for three-dimensional reconstruction of a target object according to claim 5, characterized in that: Also includes: Guiding a super-resolution operation of the second heat map based on the fourth RGB image; A super-resolution operation of the fourth RGB image is guided based on the second heat map.

7. The method for three-dimensional reconstruction of a target object according to claim 1, characterized in that: The loss function of the first Gaussian network is specifically: in, represents the loss function of the first Gaussian network, represents the inverse tone curve, represents the pixel value of the true value image, represents the pixel value of the output image, represents the L1 norm, Represents a hyperparameter.

8. A device for three-dimensional reconstruction of a target object, characterized in that: include: A first training module, used for training a Gaussian model based on a first sample set to obtain a first Gaussian network; The first sample set includes a plurality of first RGB images under dark light conditions; A second training module is used to train a neural network model based on a second sample set to obtain a dark-light enhancement network; the second sample set includes a plurality of groups of image pairs, any group of image pairs includes a second RGB image of the first object under dark light conditions, a third RGB image of the first object under ideal light conditions, and a thermal map of the first object; a joint training module, configured to input the first RGB image and a first heat map corresponding to the first RGB image into the dark light enhancement network, and input the output of the dark light enhancement network and the first heat map into the first Gaussian network for joint training to obtain a second Gaussian network; A three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the target object based on the second Gaussian network.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images