Method and device for reconstructing three-dimensional scene image based on multi-scale and joint optimization

Abstract

The application provides a three-dimensional scene image reconstruction method and device based on multi-scale and joint optimization, which comprises the following steps: generating an original rendering image and a depth map based on an explicit representation of a three-dimensional scene; constructing a multi-scale bilateral mesh with a coarse-to-fine structure and including a plurality of bilateral mesh layers with different resolutions; sequentially applying each layer of the multi-scale bilateral mesh to the original rendering image in a cascading manner to generate a final corrected image; and training the explicit representation of the three-dimensional scene and the multi-scale bilateral mesh by using a joint optimization objective function, wherein the joint optimization objective function comprises a photometric reconstruction loss for constraining the difference between the final corrected image and a ground truth image, and a geometric supervision loss for constraining the difference between the original depth map and external geometric measurement data. By introducing multi-scale mesh color correction and joint optimization of photometric loss and geometric loss, the application realizes fine color correction, decouples color and geometric modeling, and suppresses rendering defects.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for three-dimensional scene image reconstruction based on multi-scale and joint optimization. Background Technology

[0002] The 3D Gaussian Splash (3DGS) method can achieve high-quality real-time rendering of scenes in fields such as autonomous driving and virtual reality by optimizing a large number of 3D Gaussian primitives. However, in real-world scenarios such as autonomous driving, the image sequences captured by vehicle cameras often have photometric inconsistencies, such as differences in exposure and white balance parameters between different cameras, or color drift caused by changes in lighting and automatic adjustments by the ISP (Image Signal Processor) at different times for the same camera.

[0003] To address these issues, existing solutions typically employ global appearance coding for overall color correction, which fails to handle local lighting variations. Furthermore, they mostly utilize single-layer high-resolution bilateral meshes, resulting in a massive number of parameters and a susceptibility to overfitting. In addition, these existing solutions cannot decouple photometric correction from geometric reconstruction, leading to photometric errors potentially being incorrectly compensated for in the geometric structure, resulting in rendering artifacts such as floating objects, which in turn affects the accuracy of scene understanding.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] This invention provides a method and device for three-dimensional scene image reconstruction based on multi-scale and joint optimization. The main purpose of this invention is to solve the technical problems mentioned in the background art of the prior art.

[0006] The first aspect of this invention provides a method for reconstructing three-dimensional scene images based on multi-scale and joint optimization, comprising: Based on the explicit representation of the 3D scene, the rendering generates the original rendered image and the original depth map. A multi-scale bilateral mesh, ranging from coarse to fine, is constructed for color correction of the original rendered image. The multi-scale bilateral mesh includes multiple bilateral mesh levels with different resolutions. The various levels of the multi-scale bilateral mesh are applied sequentially to the original rendered image in a cascaded manner to generate the final corrected image; The explicit representation of the 3D scene and the multi-scale bilateral mesh are trained using a joint optimization objective function. The joint optimization objective function includes a photometric reconstruction loss to constrain the difference between the final corrected image and the ground truth image, and a geometric supervision loss to constrain the difference between the original depth map and external geometric measurement data.

[0007] In an optional embodiment of the first aspect of the present invention, the cascading method includes: using the pixel position of the original rendered image and its corresponding brightness value as an index, interpolating to find the corresponding color transformation parameters in each level of the multi-scale bilateral grid.

[0008] In an optional embodiment of the first aspect of the present invention, the multi-scale bilateral mesh includes: a coarse bilateral mesh layer for correcting global color shift, a medium bilateral mesh layer for correcting regional luminance differences, and a fine bilateral mesh layer for correcting local pixel-level residuals.

[0009] In an optional embodiment of the first aspect of the present invention, the joint optimization objective function further includes: a smoothing regularization loss for the multi-scale bilateral grid, used to constrain the transformation parameters of adjacent grid points in each grid level to change smoothly.

[0010] In an optional embodiment of the first aspect of the present invention, the three-dimensional scene image reconstruction method further includes a confidence gating mechanism, which is used to estimate the confidence of luminance anomalies based on pixels and adjust the correction intensity of the fine-level bilateral mesh layer according to the confidence, so as to suppress the correction effect of the fine-level bilateral mesh layer in regions with low confidence.

[0011] In an optional embodiment of the first aspect of the present invention, when processing a temporal image sequence, the joint optimization objective function further includes a temporal stability loss, which is used to constrain the parameters of at least one bilateral grid level corresponding to the same camera at adjacent timestamps to maintain consistency, so as to reduce color jitter during video playback.

[0012] In an optional embodiment of the first aspect of the present invention, during the inference phase, for a new timestamp not seen during training, color correction parameters for the new timestamp are generated by linearly interpolating the coarse-level bilateral grid layer parameters or the intermediate-level bilateral grid layer parameters corresponding to the two nearest training timestamps before and after the new timestamp.

[0013] A second aspect of the present invention provides a three-dimensional scene image reconstruction system based on multi-scale and joint optimization, wherein the three-dimensional scene real-time rendering system includes: The raw image rendering module is used to render raw images and raw depth maps based on the explicit representation of the 3D scene. A multi-scale mesh construction module is used to construct a multi-scale bilateral mesh from coarse to fine and to perform color correction on the original rendered image. The multi-scale bilateral mesh includes multiple bilateral mesh levels with different resolutions. The image color correction module is used to apply the various levels of the multi-scale bilateral grid to the original rendered image in a cascade manner to generate the final corrected image. The joint optimization module is used to train the explicit representation of the 3D scene and the multi-scale bilateral mesh using a joint optimization objective function. The joint optimization objective function includes a photometric reconstruction loss to constrain the difference between the final corrected image and the ground truth image, and a geometric supervision loss to constrain the difference between the original depth map and external geometric measurement data.

[0014] A third aspect of the present invention provides a three-dimensional scene image reconstruction device based on multi-scale and joint optimization, the three-dimensional scene image reconstruction device based on multi-scale and joint optimization includes: a memory and at least one processor, the memory storing instructions, and the memory and the at least one processor being interconnected by a circuit; The at least one processor invokes the instructions in the memory to cause the multi-scale and joint optimization-based 3D scene image reconstruction device to perform the multi-scale and joint optimization-based 3D scene image reconstruction method as described in any one of the first aspects of the present invention.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the three-dimensional scene image reconstruction method based on multi-scale and joint optimization as described in any one of the first aspects of the present invention.

[0016] Beneficial Effects: This invention provides a method and apparatus for 3D scene image reconstruction based on multi-scale and joint optimization. The method includes generating an original rendered image and a depth map based on an explicit representation of the 3D scene; constructing a multi-scale bilateral mesh with multiple layers of bilateral meshes of different resolutions, from coarse to fine; sequentially applying each layer of the multi-scale bilateral mesh to the original rendered image in a cascaded manner to generate a final corrected image; and training the explicit representation of the 3D scene and the multi-scale bilateral mesh using a joint optimization objective function, which includes a photometric reconstruction loss to constrain the difference between the final corrected image and the ground truth image, and a geometric supervision loss to constrain the difference between the original depth map and external geometric measurement data. This invention achieves refined color correction, decouples color geometric modeling, and suppresses rendering artifacts by introducing multi-scale mesh color correction and joint optimization of photometric and geometric losses. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an embodiment of a three-dimensional scene image reconstruction method based on multi-scale and joint optimization according to the present invention; Figure 2 This is a schematic diagram of an embodiment of a multi-scale bilateral grid cascade structure from coarse to fine according to the present invention; Figure 3This is a schematic diagram illustrating an embodiment of the present invention comparing the effects of photometric correction and floatation suppression before and after; Figure 4 This is a schematic diagram of an embodiment comparing the technical effects that the present invention can achieve in terms of various indicators; Figure 5 This is a schematic diagram of an embodiment of a three-dimensional scene image reconstruction system based on multi-scale and joint optimization according to the present invention; Figure 6 This is a schematic diagram of an embodiment of a three-dimensional scene image reconstruction device based on multi-scale and joint optimization according to the present invention. Detailed Implementation

[0018] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] The first aspect of this invention provides a 3D scene image reconstruction method based on multi-scale and joint optimization. The core of this method is to introduce a multi-scale bilateral grid (MS-BG) from coarse to fine at the end of the 3DGS (3D Gaussian Splash) rendering pipeline for color correction. Furthermore, a joint optimization objective function that includes photometric reconstruction and geometric supervision is used to collaboratively optimize the 3D geometric representation of the scene and the photometric appearance of the image. (See [link to relevant documentation]). Figure 1 The 3D scene image reconstruction method based on multi-scale and joint optimization includes: S100. Based on the explicit representation of the 3D scene, the original rendered image and the original depth map are generated. Specifically, in this invention, the scene is divided into sky (environment map), static background (3D Gaussian set), and dynamic objects (vehicles are transformed according to rigid body pose, and pedestrians are adjusted using a deformation network). This invention uses 3D Gaussian primitives as the explicit representation of the scene. Each point cloud or feature point in the scene is initialized as a 3D Gaussian, with attributes including position, covariance matrix, color, and opacity. During training, these parameters are optimized. In each training iteration, for a given camera pose, all 3D Gaussian primitives are projected onto the 2D image plane through a differentiable rasterizer and alpha-blending is performed to forward render an original rendered image and a corresponding original depth map. This rendering process is differentiable and allows gradient backpropagation to the parameters of the 3D Gaussian.

[0020] S200. Construct a multi-scale bilateral mesh, from coarse to fine, for color correction of the original rendered image. The multi-scale bilateral mesh includes multiple bilateral mesh levels with different resolutions. See also... Figure 2 For example, the multi-scale bilateral mesh specifically includes: a coarse bilateral mesh layer for correcting global color shift, a medium bilateral mesh layer for correcting regional luminance differences, and a fine bilateral mesh layer for correcting local pixel-level residuals.

[0021] Specifically, to perform fine and robust color correction on the original rendered image, this invention constructs a multi-scale bilateral grid (MS-BG) pyramid structure from coarse to fine. This multi-scale bilateral grid can contain three levels: a coarse-level bilateral grid layer with the lowest spatial and luminance resolution, such as 16×16×8, which is mainly responsible for capturing and correcting global color shifts in the entire image, such as overall color temperature and exposure deviation; a medium-level bilateral grid layer with medium resolution, such as 32×32×16, which is mainly responsible for correcting regional luminance differences, such as local color changes caused by large shadows or headlight illumination; and a fine-level bilateral grid layer with the highest spatial and luminance resolution, such as 64×64×32, which is mainly used to correct the remaining local high-frequency residuals after the first two levels of correction, such as color noise on fine textures or minor lens flare.

[0022] In this invention, each bilateral grid is a three-dimensional lookup table, with the three dimensions representing the spatial location of the pixels. and brightness Each grid point in the grid can store a 3×4 affine transformation matrix, which can linearly rotate, scale, and add a bias term to the RGB color vector.

[0023] S300. The layers of the multi-scale bilateral mesh are sequentially applied to the original rendered image in a cascaded manner to generate the final corrected image. For example, the formula for calculating the brightness guide map in this invention can be as follows: It can serve as the "brightness axis" index for the bilateral grid, generating three-level guide maps according to different downsampling factors, corresponding to the subsequent three-level grids.

[0024] Specifically, in this invention, the color correction process of the original rendered image is performed in a cascade manner, and the process can be as follows: First, the original rendered image is converted into a grayscale image to obtain a brightness guide map. For any pixel in the image, color correction is performed. Coarse correction uses the pixel coordinates and its luminance value in the luminance guide map as indices. In a coarse-level bilateral grid layer (e.g., between 2×2×2=8 adjacent grid points in each level), trilinear interpolation is used to obtain the coarse affine transformation operator corresponding to that pixel. This affine transformation operator is applied to the original pixel color to obtain the color after the first correction. Intermediate correction uses the same indexing method as coarse correction. Interpolation is performed in a intermediate-level bilateral grid layer to obtain the intermediate affine transformation operator, which is then applied to the color after the first correction to obtain the color after the second correction. Similarly, for fine-level correction, fine-level affine transformation operators are obtained through interpolation in the fine-level bilateral mesh layer and applied to the color after secondary correction to obtain the pixel color in the final corrected image (i.e., the cascaded method includes using the pixel position and its corresponding brightness value of the original rendered image as an index to interpolate and find the corresponding color transformation parameters in each level of the multi-scale bilateral mesh). This invention, through this coarse-to-fine cascaded correction method, allows the high-resolution fine-level bilateral mesh layer to learn only local residuals, thereby effectively reducing the number of parameters (compared to a single-layer high-resolution mesh) and the risk of overfitting. For example, the formula corresponding to the above transformation process of this invention can be as follows: ,in, This is the original rendered image. This is a coarse affine transformation (global offset correction). This is an intermediate affine transformation (regional difference correction). This is a fine-level affine transformation (local high-frequency residual correction). This is the final calibration diagram.

[0025] S400. The explicit representation of the 3D scene and the multi-scale bilateral mesh are trained using a joint optimization objective function. The joint optimization objective function includes a photometric reconstruction loss to constrain the difference between the final corrected image and the ground truth image, and a geometric supervision loss to constrain the difference between the original depth map and external geometric measurement data.

[0026] Specifically, in this invention, to ensure the accuracy of photometric correction without damaging the geometric structure, a joint optimization objective function is designed, which is composed of a weighted sum of multiple loss terms: Photometric reconstruction loss: This is the main monitoring signal in this invention, used to constrain the final corrected image to approximate the ground truth image at the same time. The photometric reconstruction loss used in this invention can be derived from... Loss and (Structural similarity) loss components: ,in, For reconstruction losses; The mean of absolute differences per pixel (i.e. loss), For the corrected image, This corresponds to the ground truth image of the camera at the corresponding time. For structural similarity loss, i.e. It also measures differences in brightness, contrast, and structure.

[0027] Geometric Supervision Loss: To prevent the photometric correction module from incorrectly "correcting" geometric imperfections, this invention also introduces a loss from... Sparse depth point clouds from external sensors such as LiDAR are used as geometric supervision to... Point cloud is projected onto the current camera view to form a sparse depth map. The geometric loss is calculated based on the original rendered depth map. and Mean square error of the squared L2 norm over effective pixels: ,in, Deep monitoring loss, A pixel-by-pixel depth map obtained from 3DGS rendering. for The point cloud is projected onto the sparse depth map formed by the current camera viewpoint, and the loss term is calculated only at the effective projected pixel locations.

[0028] In some alternative embodiments of the first aspect of the invention, to prevent color transformation from being too abrupt, a smoothing constraint can be applied to each layer of the multi-scale bilateral grid to penalize the difference in affine transformation parameters of adjacent grid points (i.e., the joint optimization objective function also includes a smoothing regularization loss for the multi-scale bilateral grid to constrain the transformation parameters of adjacent grid points in each grid level to change gradually). The weight of this smoothing regularization loss term is proportional to the grid resolution, i.e., a stronger smoothing penalty is applied to fine-level bilateral grid layers. Of course, a reversibility constraint can also be further applied to each layer of the multi-scale network, i.e., the ground truth image can be restored to the rendered image after being reversed by the inverse transformation to prevent the color transformation from being too aggressive.

[0029] In some alternative embodiments of the first aspect of the invention, a confidence gating mechanism may also be introduced. Specifically, the invention predicts a pixel-level photometric anomaly confidence map in parallel. This confidence level is used to adjust the correction strength of the fine-level bilateral mesh layer. Specifically, a term is added to the loss function when... At lower levels, the transformation operator of the fine-level bilateral mesh layer is forced to approach the identity matrix (without any transformation). This ensures that the fine-level bilateral mesh layer only works in areas where there are indeed photometric anomalies, avoiding overcorrection of normal areas. (That is, the 3D scene image reconstruction method also includes a confidence gating mechanism, which is used to estimate the confidence of photometric anomalies based on pixels and adjust the correction intensity of the fine-level bilateral mesh layer according to the confidence, so as to suppress the correction effect of the fine-level bilateral mesh layer in areas with low confidence.)

[0030] In some alternative embodiments of the first aspect of the invention, in order to eliminate inter-frame color jitter when processing video sequences, the same camera at adjacent timestamps can also be used. and Parameter matrices of coarse and medium-level bilateral mesh layers Apply L2 constraints: , For time-stability loss, For timestamps The corresponding frame's coarse / medium level bilateral mesh layer contains all affine transformation coefficient matrices. The constraint is the same-level grid coefficient matrix of the previous frame. This constraint requires that the grid coefficients of adjacent frames change as little as possible to suppress frame-by-frame color jitter during video playback. It is only applied to coarse / medium-level bilateral grid layers. Fine-level bilateral grid layers are not subject to this constraint to allow normal changes in high-frequency details (i.e., when processing temporal image sequences, the joint optimization objective function also includes a temporal stability loss, which is used to constrain the parameters of at least one bilateral grid layer corresponding to the same camera at adjacent timestamps to maintain consistency, so as to reduce color jitter during video playback). In this invention, after obtaining the above-mentioned losses, the gradient of the total loss is backpropagated, and the parameters of the three-dimensional Gaussian primitives and all affine transformation parameters of the multi-scale bilateral grid are updated simultaneously to complete the photometric-geometric joint optimization training.

[0031] Furthermore, in this invention, after training is complete, for a new timestamp that has not appeared in the training set... To generate smooth video results, this invention can also employ temporal interpolation (i.e., during the inference phase, for a new timestamp not seen during training, linear interpolation is performed on the coarse-level or intermediate-level bilateral grid layer parameters corresponding to the two nearest training timestamps before and after the new timestamp to generate color correction parameters for the new timestamp). The specific process can be as follows: First, find the two nearest training timestamps before and after the new timestamp. and Then, linear interpolation is performed only on the parameters of the coarse and medium-level bilateral grid layers to obtain the grid parameters corresponding to the new timestamp, while the fine-level bilateral grid layer directly reuses the most recent previous timestamp. The parameters are adjusted to preserve the clarity of high-frequency details. Finally, the MS-BG obtained by interpolation is used to perform color correction on the rendered image with the new timestamp. The formula used for the corresponding time-linear interpolation can be as follows: , ,in, The bilateral grid coefficients corresponding to the target timestamp obtained by interpolation during inference. For linear interpolation weights (scalars, ranging from [0, 1]), For the most recent training timestamp The corresponding coarse / medium level bilateral grid layer coefficients, For the most recent training timestamp The corresponding coarse / medium level bilateral grid layer coefficients, The target timestamp input for the inference phase (not seen during training); To meet The most recent training timestamp To meet The most recent training timestamp The closer it is to 1, the closer it is to the target time. Fine-grained bilateral mesh layers are maintained The training values ​​of the frames remain unchanged and are not used for interpolation.

[0032] In summary, the 3D scene image reconstruction scheme based on multi-scale and joint optimization of this invention includes at least the following technical points: Technical point 1: The color transformation pyramid structure of the multi-scale bilateral grid (MS-BG) from coarse to fine consists of three levels of small-sized grids: the coarse level is responsible for the overall color shift at the whole image level, the medium level is responsible for the regional luminance difference, and the fine level is responsible for the local pixel-level residual; after the three levels are stacked, the total number of parameters is about one-seventh of that of a single-layer high-resolution bilateral grid.

[0033] Technical point 2: The multi-scale color correction method of "luminance guidance + cascade correction" uses the grayscale image of the rendered image itself as the luminance index. The corresponding pixel-level color transformation is obtained by interpolation on each level of the grid. Then, it is applied in the order of coarse, medium and fine, so that the high-resolution grid mainly undertakes the local residual correction, thereby reducing the risk of overfitting.

[0034] Technical point 3: The training objective of "photometric-geometric joint optimization" uses "the difference between the corrected image and the ground truth" and "the difference between the rendering depth and the LiDAR depth" as supervision at the same time, and applies a smoothing regularization proportional to the mesh size to each level of the mesh (the penalty is heavier for fine meshes) to avoid the color module learning noise and to prevent photometric inconsistencies from polluting the geometric reconstruction.

[0035] Technical Point 4: The MS-BG enhancement mechanism of "confidence gating + temporal stability" Confidence gating: It estimates whether the luminance is abnormal on a pixel basis and only allows strong color correction by the fine layer on pixels that "actually have luminance abnormalities". For other pixels, the fine layer is forced to remain close to unchanged, avoiding the substitution of geometric error correction with color correction; Temporal stability: It applies the "adjacent frame proximity" constraint to the coarse / medium level bilateral grid layers of adjacent frames of the same camera to eliminate color jitter during video playback.

[0036] Cooperate Figure 3 and Figure 4 The beneficial effects that the multi-scale and joint optimization-based 3D scene image reconstruction scheme of this invention can achieve are as follows: 1. The number of parameters is only about 3.97M (1 / 7 of a single-layer bilateral grid), and the training time increases by only about 9% compared to appearance encoding.

[0037] 2. On autonomous driving datasets such as Waymo, NuScenes, Argoverse2, and PandaSet, the PSNR, SSIM, and Chamfer Distance metrics are generally better than common baseline methods, with more significant improvements in scenarios such as nighttime, glare, and rain.

[0038] 3. The joint optimization of photometric correction and geometric depth helps to reduce "floating objects" and improve the stability of downstream obstacle detection and trajectory planning.

[0039] 4. The solution of this invention can be integrated into 3DGS-driven scene frameworks such as OmniRe, StreetGS, and ChatSim, and can be deployed without changing the basic structure of the main rendering process.

[0040] 5. Confidence gating can reduce the possibility of misclassifying photometric problems as geometric problems, and temporal stability constraints help reduce color flicker in video playback.

[0041] 6. The present invention can complete training on a single L40-level GPU, and maintain a real-time level of 42–54 FPS during the inference phase.

[0042] See Figure 5 The second aspect of the present invention provides a three-dimensional scene image reconstruction system based on multi-scale and joint optimization, wherein the three-dimensional scene real-time rendering system includes: The original image rendering module 10 is used to render and generate original rendered images and original depth maps based on the explicit representation of the 3D scene. The multi-scale mesh construction module 20 is used to construct a multi-scale bilateral mesh from coarse to fine and to perform color correction on the original rendered image. The multi-scale bilateral mesh includes multiple bilateral mesh levels with different resolutions. The image color correction module 30 is used to apply the various levels of the multi-scale bilateral grid to the original rendered image in a cascade manner to generate the final corrected image. The joint optimization module 40 is used to train the explicit representation of the 3D scene and the multi-scale bilateral mesh using a joint optimization objective function. The joint optimization objective function includes a photometric reconstruction loss to constrain the difference between the final corrected image and the ground truth image, and a geometric supervision loss to constrain the difference between the original depth map and external geometric measurement data.

[0043] In an optional embodiment of the second aspect of the present invention, the cascading method includes: using the pixel position of the original rendered image and its corresponding brightness value as an index, interpolating to find the corresponding color transformation parameters in each level of the multi-scale bilateral grid.

[0044] In an optional embodiment of the second aspect of the present invention, the multi-scale bilateral mesh includes: a coarse bilateral mesh layer for correcting global color shift, a medium bilateral mesh layer for correcting regional luminance differences, and a fine bilateral mesh layer for correcting local pixel-level residuals.

[0045] In an optional embodiment of the second aspect of the present invention, the joint optimization objective function further includes: a smoothing regularization loss for the multi-scale bilateral grid, used to constrain the transformation parameters of adjacent grid points in each grid level to change smoothly.

[0046] In an optional embodiment of the second aspect of the present invention, the three-dimensional scene image reconstruction method further includes a confidence gating mechanism, which is used to estimate the confidence of luminance anomalies based on pixels and adjust the correction intensity of the fine-level bilateral mesh layer according to the confidence, so as to suppress the correction effect of the fine-level bilateral mesh layer in regions with low confidence.

[0047] In an optional embodiment of the second aspect of the present invention, when processing temporal image sequences, the joint optimization objective function further includes a temporal stability loss, which is used to constrain the parameters of at least one bilateral grid level corresponding to the same camera at adjacent timestamps to maintain consistency, so as to reduce color jitter during video playback.

[0048] In an optional embodiment of the second aspect of the invention, during the inference phase, for a new timestamp not seen during training, color correction parameters for the new timestamp are generated by linearly interpolating the coarse-level bilateral grid layer parameters or the intermediate-level bilateral grid layer parameters corresponding to the two most recent training timestamps before and after the new timestamp.

[0049] Figure 6 This is a schematic diagram of a 3D scene image reconstruction device based on multi-scale and joint optimization provided by an embodiment of the present invention. This 3D scene image reconstruction device based on multi-scale and joint optimization can vary considerably due to different configurations or performance. It may include one or more processors 50 (central processing units, CPUs) (e.g., one or more processors) and memory 60, and one or more storage media 70 (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media can be temporary or persistent storage. The program stored in the storage media may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the 3D scene image reconstruction device based on multi-scale and joint optimization. Furthermore, the processor may be configured to communicate with the storage media and execute the series of instruction operations in the storage media on the 3D scene image reconstruction device based on multi-scale and joint optimization.

[0050] The 3D scene image reconstruction device based on multi-scale and joint optimization of this invention may also include one or more power supplies 80, one or more wired or wireless network interfaces 90, one or more input / output interfaces 100, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6 The illustrated structure of the 3D scene image reconstruction device based on multi-scale and joint optimization does not constitute a limitation on the 3D scene image reconstruction device based on multi-scale and joint optimization. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0051] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the three-dimensional scene image reconstruction method based on multi-scale and joint optimization.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or system / unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0053] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing 3D scene images based on multi-scale and joint optimization, characterized in that, include: Based on the explicit representation of the 3D scene, the rendering generates the original rendered image and the original depth map. A multi-scale bilateral mesh, ranging from coarse to fine, is constructed for color correction of the original rendered image. The multi-scale bilateral mesh includes multiple bilateral mesh levels with different resolutions. The various levels of the multi-scale bilateral mesh are applied sequentially to the original rendered image in a cascaded manner to generate the final corrected image; The explicit representation of the 3D scene and the multi-scale bilateral mesh are trained using a joint optimization objective function. The joint optimization objective function includes a photometric reconstruction loss to constrain the difference between the final corrected image and the ground truth image, and a geometric supervision loss to constrain the difference between the original depth map and external geometric measurement data.

2. The three-dimensional scene image reconstruction method based on multi-scale and joint optimization according to claim 1, characterized in that, The cascading method includes: using the pixel positions and corresponding brightness values ​​of the original rendered image as indexes, and interpolating to find the corresponding color transformation parameters in each level of the multi-scale bilateral grid.

3. The three-dimensional scene image reconstruction method based on multi-scale and joint optimization according to claim 1 or 2, characterized in that, The multi-scale bilateral mesh includes: a coarse bilateral mesh layer for correcting global color shift, a medium bilateral mesh layer for correcting regional luminance differences, and a fine bilateral mesh layer for correcting local pixel-level residuals.

4. The three-dimensional scene image reconstruction method based on multi-scale and joint optimization according to claim 1, characterized in that, The joint optimization objective function also includes: a smoothing regularization loss for the multi-scale bilateral grid, used to constrain the transformation parameters of adjacent grid points in each grid level to change smoothly.

5. The three-dimensional scene image reconstruction method based on multi-scale and joint optimization according to claim 3, characterized in that, The three-dimensional scene image reconstruction method also includes a confidence gating mechanism, which is used to estimate the confidence of luminance anomalies based on pixels and adjust the correction intensity of the fine-level bilateral mesh layer according to the confidence, so as to suppress the correction effect of the fine-level bilateral mesh layer in the region of low confidence.

6. The three-dimensional scene image reconstruction method based on multi-scale and joint optimization according to claim 1, characterized in that, When processing temporal image sequences, the joint optimization objective function also includes a temporal stability loss, which is used to constrain the parameters of at least one bilateral grid level corresponding to the same camera at adjacent timestamps to maintain consistency, so as to reduce color jitter during video playback.

7. The three-dimensional scene image reconstruction method based on multi-scale and joint optimization according to claim 6, characterized in that, During the inference phase, for new timestamps not seen during training, color correction parameters for the new timestamps are generated by linearly interpolating the coarse-level or intermediate-level bilateral mesh layer parameters corresponding to the two nearest training timestamps before and after the new timestamp.

8. A three-dimensional scene image reconstruction system based on multi-scale and joint optimization, characterized in that, The real-time rendering system for the three-dimensional scene includes: The raw image rendering module is used to render raw images and raw depth maps based on the explicit representation of the 3D scene. A multi-scale mesh construction module is used to construct a multi-scale bilateral mesh from coarse to fine and to perform color correction on the original rendered image. The multi-scale bilateral mesh includes multiple bilateral mesh levels with different resolutions. The image color correction module is used to apply the various levels of the multi-scale bilateral grid to the original rendered image in a cascade manner to generate the final corrected image. The joint optimization module is used to train the explicit representation of the 3D scene and the multi-scale bilateral mesh using a joint optimization objective function. The joint optimization objective function includes a photometric reconstruction loss to constrain the difference between the final corrected image and the ground truth image, and a geometric supervision loss to constrain the difference between the original depth map and external geometric measurement data.

9. A three-dimensional scene image reconstruction device based on multi-scale and joint optimization, characterized in that, The 3D scene image reconstruction device based on multi-scale and joint optimization includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via lines; The at least one processor invokes the instructions in the memory to cause the multi-scale and joint optimization-based 3D scene image reconstruction device to perform the multi-scale and joint optimization-based 3D scene image reconstruction method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional scene image reconstruction method based on multi-scale and joint optimization as described in any one of claims 1-7.

Images