Image processing methods, apparatus and equipment for scene reconstruction and rendering

Abstract

This invention provides an image processing method, apparatus, and device for scene reconstruction and rendering. The method includes: initializing a sparse point cloud corresponding to each scene image to be reconstructed to obtain multiple initial Gaussian polygons; optimizing each initial Gaussian polygon based on the inverse depth map, the rendered scene image, and the normal vector map obtained from the initial Gaussian polygons to obtain multiple first target Gaussian polygons; determining second target Gaussian polygons based on the importance of the first target Gaussian polygons, wherein the importance is determined based on multiple sub-models and the target scene image to be reconstructed that matches each sub-model, wherein the multiple sub-models are obtained by meshing the multiple first target Gaussian polygons, and the target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-models; and determining a third target Gaussian polygon for rendering the reconstructed scene image based on the spatial range of the mesh.

Description

Technical Field

[0001] This invention relates to the fields of computer science, artificial intelligence, monocular depth estimation, and 3D modeling, and specifically to image processing methods, apparatus, and devices for scene reconstruction and rendering. Background Technology

[0002] 3D scene reconstruction is a long-standing research topic in computer vision and graphics, with the core pursuit being realistic visual rendering and accurate geometric reconstruction. Currently, mainstream algorithms are based on the representation of 3D Gaussian ellipsoids (3DGS), using a set of discrete Gaussian ellipsoids expressed through a 3D Gaussian distribution to fit and reconstruct the scene, and then using a highly optimized rasterization pipeline for rendering. Due to its advantages in training convergence and rendering efficiency, this algorithm has gradually replaced Neural Radiance Fields (NeRF) as the mainstream technique in this field.

[0003] In the process of realizing the concept of this invention, the inventors discovered that in the current three-dimensional scene reconstruction, there are still problems such as low reconstruction accuracy, high cost, and low fidelity of the rendered scene map. Summary of the Invention

[0004] In view of this, the present invention provides an image processing method, apparatus, device, medium, and program product for scene reconstruction and rendering.

[0005] One aspect of the present invention provides an image processing method for scene reconstruction and rendering, comprising: initializing a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed, to obtain a plurality of initial Gaussian polygons, wherein the sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool; optimizing each initial Gaussian polygon based on an inverse depth map, a rendered scene image, and a normal vector map obtained by rendering the initial Gaussian polygons, to obtain a plurality of first target Gaussian polygons; determining a second target Gaussian polygon from the plurality of first target Gaussian polygons based on the importance of the first target Gaussian polygons, wherein the importance of the first target Gaussian polygons is determined based on a plurality of sub-models and a target scene image to be reconstructed that matches each sub-model, the plurality of sub-models being obtained by meshing the plurality of first target Gaussian polygons, each sub-model being composed of first target Gaussian polygons within each mesh, and the target scene image to be reconstructed that matches each sub-model being determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model; and determining a third target Gaussian polygon for rendering the reconstructed scene image from the second target Gaussian polygons based on the spatial extent of the mesh.

[0006] According to an embodiment of the present invention, the image processing method for scene reconstruction and rendering further includes: performing depth estimation on each scene image to be reconstructed using a monocular estimation pedestal model to obtain an initial inverse depth map corresponding to each scene image to be reconstructed; determining a target inverse depth map with the same size as each scene image to be reconstructed based on the initial inverse depth map; and determining a target normal vector map based on the target inverse depth map and the target scene image.

[0007] According to an embodiment of the present invention, based on the inverse depth map, the rendered scene image, and the normal vector map obtained by rendering the initial Gaussian surface, each initial Gaussian surface is optimized to obtain multiple first target Gaussian surface elements, including: for each initial Gaussian surface element: determining a depth estimation loss value based on the target inverse depth map and the inverse depth map; determining an image loss value based on the target scene image and the rendered scene image; determining a normal vector loss value based on the target normal vector map and the normal vector map; obtaining a total loss value based on the sum of the depth estimation loss value, the image loss value, and the normal vector loss value; iteratively optimizing the initial Gaussian surface element based on the total loss value; determining the gradient of the projection position of the imaging plane when the number of iterations meets a preset number of iterations; and cloning or splitting the optimized initial Gaussian surface element to obtain multiple first target Gaussian surface elements when the average value of the gradient of the projection position of the imaging plane is greater than an average threshold and the aspect ratio of the optimized initial Gaussian surface element is less than a preset aspect ratio.

[0008] According to an embodiment of the present invention, the image processing method for scene reconstruction and rendering further includes: dividing a plurality of first target Gaussian surface elements into a mesh based on spatial relationships to obtain a plurality of sub-models; determining a target scene image to be reconstructed that matches each sub-model from a plurality of scene images to be reconstructed based on the pose of the scene image to be reconstructed and the rendering results of the sub-models; fine-tuning the first target Gaussian surface elements in each sub-model based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed, and the normal vector map of the target scene image to be reconstructed; and determining the importance of the first target Gaussian surface elements in each sub-model when the number of fine-tunings satisfies a preset number of iterations.

[0009] According to an embodiment of the present invention, the rendering result of the sub-model includes a first rendering result obtained when the opacity of the first target Gaussian surface element contained in the sub-model is set to zero and a second rendering result obtained when the opacity is kept at the original value; based on the pose of the scene image to be reconstructed and the rendering result of the sub-model, determining the target scene image to be reconstructed that matches each sub-model from multiple scene images to be reconstructed includes: determining the scene image to be reconstructed as the target scene image to be reconstructed that matches each sub-model when it is determined that the pose of the scene image to be reconstructed is within the mesh space of the sub-model and the loss value of the first rendering result and the second rendering result exceeds a preset loss value.

[0010] According to an embodiment of the present invention, when it is determined that the number of fine-tuning steps meets the preset number of iterations, determining the importance of the first target Gaussian surface element in each sub-model includes: determining the number of pixels covered by the projection of each first target Gaussian surface element onto the target scene image to be reconstructed, when it is determined that the number of fine-tuning steps meets the preset number of iterations; determining the opacity of each first target Gaussian surface element; and determining the importance of the first target Gaussian surface element in each sub-model based on the pixels, the opacity, and the first target Gaussian surface element in each sub-model.

[0011] According to an embodiment of the present invention, determining a third target Gaussian surface element for rendering a reconstructed scene image from a second target Gaussian surface element based on the spatial range of the grid includes: determining the second target Gaussian surface element located at the center of the spatial range of the grid as the third target Gaussian surface element according to the spatial range of the grid.

[0012] According to an embodiment of the present invention, the third target Gaussian surface element includes multiple elements; the image processing method for scene reconstruction and rendering further includes: vector quantization of the spherical harmonic coefficients of the third target Gaussian surface element with the lowest importance; configuring a preset precision for the spherical harmonic coefficients of the other third target Gaussian surface elements among the multiple third target Gaussian surface elements except for the third target Gaussian surface element with the lowest importance; and compressing the multiple third target Gaussian surface elements based on an octree compression scheme.

[0013] A second aspect of the present invention provides an image processing apparatus for scene reconstruction and rendering, comprising: an initialization module for initializing a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed, thereby obtaining a plurality of initial Gaussian surface elements, wherein the sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool; an optimization module for optimizing each initial Gaussian surface element based on an inverse depth map, a rendered scene image, and a normal vector map obtained from the initial Gaussian surface elements, thereby obtaining a plurality of first target Gaussian surface elements; and a first determination module for determining, based on the importance of the first target Gaussian surface elements, a subset of the first target Gaussian surface elements. The first target Gaussian surface is determined from the first target Gaussian surface, where the importance of the first target Gaussian surface is determined based on multiple sub-models and the target scene image to be reconstructed that matches each sub-model. The multiple sub-models are obtained by meshing multiple first target Gaussian surfaces, and each sub-model consists of first target Gaussian surfaces within each mesh. The target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model. The second determination module is used to determine a third target Gaussian surface for rendering the reconstructed scene image from the second target Gaussian surface based on the spatial range of the mesh.

[0014] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the image processing method for scene reconstruction and rendering described above.

[0015] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the image processing method for scene reconstruction and rendering described above.

[0016] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the image processing method for scene reconstruction and rendering described above.

[0017] According to embodiments of the present invention, each initial Gaussian polygon is optimized based on the inverse depth map, the rendered scene image, and the normal vector map obtained from the initial Gaussian polygon rendering. This provides reliable depth supervision for multiple scene images to be reconstructed without depth annotations, accelerates the optimization of Gaussian polygons, and eliminates Gaussian polygon blur artifacts. Based on the importance of the first target Gaussian polygon, a second target Gaussian polygon is determined, and a third target Gaussian polygon is determined based on the spatial range of the mesh. This avoids redundant post-processing steps, enabling high-quality scene reconstruction and real-time rendering with high reconstruction accuracy and low reconstruction overhead. Attached Figure Description

[0018] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustration schematically depicts an application scenario of an image processing method, apparatus, device, medium, and program product for scene reconstruction and rendering according to embodiments of the present invention.

[0020] Figure 2 A flowchart illustrating an image processing method for scene reconstruction and rendering according to an embodiment of the present invention is shown schematically;

[0021] Figure 3 The diagram illustrates the overall structure of the Gaussian surface element adaptive optimization algorithm according to an embodiment of the present invention.

[0022] Figure 4 This diagram schematically illustrates the overall structure of a distributed parallel training pipeline according to an embodiment of the present invention.

[0023] Figure 5A schematic diagram illustrating the structure of an image processing apparatus for scene reconstruction and rendering according to an embodiment of the present invention is shown; and

[0024] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method for scene reconstruction and rendering according to an embodiment of the present invention. Detailed Implementation

[0025] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a method having at least one of A, B and C" should include, but is not limited to, a method having A alone, a method having B alone, a method having C alone, a method having A and B, a method having A and C, a method having B and C, and / or a method having A, B and C, etc.).

[0029] In realizing this invention, the inventors discovered that to address the ambiguity and inaccuracy of 3DGS representation in surface estimation, 2D Gaussian Splatting (2DGS) performs dimensional collapse on the Gaussian distribution used in 3DGS, employing a set of discrete Gaussian elements to represent the 3D structure of the scene. Simultaneously, 2DGS introduces constraints on the distribution of normal vectors, ensuring the smoothness and completeness of the surface reconstruction results and improving the accuracy of the geometric structure.

[0030] The pedestal-based monocular depth estimation technique draws on paradigms in computer vision and natural language processing. It is trained on a large-scale dataset containing both unlabeled and depth-annotated images. The resulting model can accurately estimate the depth of input monocular RGB images and has good generalization ability. It plays a key role in technologies such as robotics, autonomous driving, and virtual reality.

[0031] The combination and optimization of technologies from multiple fields, including scene representation based on 3D Gaussian ellipsoids, geometric reconstruction based on 2D Gaussian surface elements, and monocular depth estimation based on pedestal models, aim to improve the rendering quality, geometric accuracy, training efficiency, and rendering speed of large-scale scene reconstruction, and meet the needs of different application scenarios for high-quality, high-efficiency, and lightweight reconstruction results.

[0032] Based on this, embodiments of the present invention provide an image processing method for scene reconstruction and rendering, comprising: initializing a sparse point cloud corresponding to each of multiple scene images to be reconstructed to obtain multiple initial Gaussian elements, wherein the sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool; optimizing each initial Gaussian element based on the inverse depth map, the rendered scene image, and the normal vector map obtained by rendering the initial Gaussian elements to obtain multiple first target Gaussian elements; determining a second target Gaussian element from the multiple first target Gaussian elements based on the importance of the first target Gaussian elements, wherein the importance of the first target Gaussian elements is determined based on multiple sub-models and the target scene image to be reconstructed that matches each sub-model, the multiple sub-models are obtained by meshing the multiple first target Gaussian elements, each sub-model is composed of the first target Gaussian elements within each mesh, and the target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model; and determining a third target Gaussian element for rendering the reconstructed scene image from the second target Gaussian elements based on the spatial range of the mesh.

[0033] Figure 1 The illustration schematically depicts an application scenario of an image processing method, apparatus, device, medium, and program product for scene reconstruction and rendering according to embodiments of the present invention.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the image processing method for scene reconstruction and rendering provided in the embodiments of the present invention can generally be executed by server 105. Correspondingly, the image processing apparatus for scene reconstruction and rendering provided in the embodiments of the present invention can generally be located in server 105. The image processing method for scene reconstruction and rendering provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the image processing apparatus for scene reconstruction and rendering provided in the embodiments of the present invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The following will be based on Figure 1 The described scene, through Figures 2-4 The image processing method for scene reconstruction and rendering according to embodiments of the invention is described in detail.

[0041] Figure 2A flowchart illustrating an image processing method for scene reconstruction and rendering according to an embodiment of the present invention is shown schematically.

[0042] like Figure 2 As shown, the image processing method for scene reconstruction and rendering includes operations S210~S240.

[0043] In operation S210, the sparse point cloud corresponding to each of the multiple scene images to be reconstructed is initialized to obtain multiple initial Gaussian surface elements.

[0044] In operation S220, based on the inverse depth map, rendered scene image and normal vector map obtained from the initial Gaussian surface rendering, each initial Gaussian surface is optimized to obtain multiple first target Gaussian surfaces.

[0045] In operation S230, based on the importance of the first target Gaussian surface element, the second target Gaussian surface element is determined from multiple first target Gaussian surface elements.

[0046] In operation S240, based on the spatial extent of the mesh, a third target Gaussian surface is determined from the second target Gaussian surface for rendering the reconstructed scene image.

[0047] According to embodiments of the present invention, sparse point clouds can be obtained by processing the scene image to be reconstructed using a preset reconstruction tool. The preset reconstruction tool may, for example, include the 3D reconstruction tool COLMAP. The scene image to be reconstructed may include an RGB image. Multiple scene images to be reconstructed may be multi-view RGB images of the scene to be reconstructed.

[0048] According to embodiments of the present invention, before each reconstruction task begins, multi-view RGB images of the target scene can be collected, such as a set of images obtained by extracting frames from a video of the target scene captured by a camera, or a set of images with poses captured by a drone of the target scene. These images can be processed by COLMAP to obtain camera parameters, poses of each view, and sparse point clouds for subsequent image processing.

[0049] According to embodiments of the present invention, optimizing each initial Gaussian surface element may include optimizing the properties of each initial Gaussian surface element. Properties may include, but are not limited to, spherical harmonic coefficients, surface element position, and opacity. The initial Gaussian surface elements can be rendered to obtain an inverse depth map. The inverse depth map can be converted into a depth map, and the depth map can be used to estimate a normal vector map. Combined with the rendered scene image, each initial Gaussian surface element can be iteratively optimized to obtain multiple first target Gaussian surface elements.

[0050] According to an embodiment of the present invention, the importance of the first target Gaussian surface element is determined based on multiple sub-models and the target scene image to be reconstructed that matches each sub-model. The multiple sub-models are obtained by meshing multiple first target Gaussian surface elements. Each sub-model is composed of first target Gaussian surface elements within each mesh. The target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model.

[0051] According to an embodiment of the present invention, the importance of a first target Gaussian surface element can be compared with an importance threshold, and the first target Gaussian surface element whose importance is greater than or equal to the importance threshold can be determined as the second target Gaussian surface element.

[0052] According to an embodiment of the present invention, a second target Gaussian surface located at the center of the mesh can be determined as a third target Gaussian surface for rendering the reconstructed scene image. Based on conventional rendering methods, the third target Gaussian surface can be used to render the reconstructed scene image.

[0053] According to embodiments of the present invention, each initial Gaussian polygon is optimized based on the inverse depth map, the rendered scene image, and the normal vector map obtained from the initial Gaussian polygon rendering. This provides reliable depth supervision for multiple scene images to be reconstructed without depth annotations, accelerates the optimization of Gaussian polygons, and eliminates Gaussian polygon blur artifacts. Based on the importance of the first target Gaussian polygon, a second target Gaussian polygon is determined, and a third target Gaussian polygon is determined based on the spatial range of the mesh. This avoids redundant post-processing steps, enabling high-quality reconstruction and real-time rendering of large-scale scenes with high reconstruction accuracy and low reconstruction overhead.

[0054] According to an embodiment of the present invention, the image processing method for scene reconstruction and rendering includes, in addition to the above... Figure 2 In addition to the operations S210~S240 shown, the following operations may also be included: using a monocular estimation pedestal model to perform depth estimation on each scene image to be reconstructed to obtain an initial inverse depth map corresponding to each scene image to be reconstructed; based on the initial inverse depth map, determining a target inverse depth map with the same size as each scene image to be reconstructed; and determining a target normal vector map based on the target inverse depth map and the target scene image.

[0055] According to embodiments of the present invention, the median of the inverse depth and the average of the absolute values ​​of the deviations between the inverse depth and the median can be calculated based on multiple initial inverse depth maps. The sparse point cloud corresponding to each scene image to be reconstructed can be projected back to the viewpoint corresponding to each scene image to be reconstructed, obtaining the sparse inverse depth distribution of each scene image to be reconstructed at the actual scale. The median of the sparse inverse depth and the average of the absolute values ​​of the deviations between the sparse inverse depth and the median are calculated based on multiple sparse inverse depth distributions. Based on the initial inverse depth map, the median of the inverse depth and the average of the absolute values ​​of the deviations between the inverse depth and the median, and the median of the sparse inverse depth and the average of the absolute values ​​of the deviations between the sparse inverse depth and the median, the depth estimated by the monocular estimation pedestal model is scaled down to the actual scale to obtain the target inverse depth map.

[0056] For example, RGB images and pose information can be input into a monocular estimation pedestal model, such as Depth-Anything-V2, to obtain the depth distribution corresponding to each viewpoint for use in the next step of depth supervision.

[0057] According to an embodiment of the present invention, the target inverse depth map can be transformed into a target depth map, and the target normal vector map can be estimated from the target depth map and the target scene image.

[0058] According to an embodiment of the present invention, the depth regression technique based on a monocular estimation pedestal model estimates the depth distribution of each training view using the monocular estimation pedestal model and scales it according to the actual scale of the scene to supervise the rendering depth of the model. It can provide reliable depth supervision for multiple scene images to be reconstructed without depth annotation and is used for adaptive optimization of Gaussian polygons.

[0059] According to embodiments of the present invention, regarding the above... Figure 2 Operation S220 optimizes each initial Gaussian surface element based on the inverse depth map, rendered scene image, and normal vector map obtained from the initial Gaussian surface element rendering, resulting in multiple first target Gaussian surface elements. This operation may include the following steps for each initial Gaussian surface element: determining a depth estimation loss value based on the target inverse depth map and the inverse depth map; determining an image loss value based on the target scene image and the rendered scene image; determining a normal vector loss value based on the target normal vector map and the normal vector map; obtaining a total loss value based on the sum of the depth estimation loss value, the image loss value, and the normal vector loss value; iteratively optimizing the initial Gaussian surface element based on the total loss value; determining the gradient of the projection position of the imaging plane if the number of iterations meets a preset number of iterations; and cloning or splitting the optimized initial Gaussian surface element if the average value of the gradient of the projection position of the imaging plane is greater than an average threshold and the aspect ratio of the optimized initial Gaussian surface element is less than a preset aspect ratio, resulting in multiple first target Gaussian surface elements.

[0060] According to an embodiment of the present invention, for each initial Gaussian surface element: a depth estimation loss value can be obtained by calculating the absolute value of the difference between the inverse depth map and the target inverse depth map. An image loss value is obtained by calculating the loss between the rendered scene image and the target scene image. A normal vector loss value is obtained by calculating the absolute value of the difference between the normal vector map and the target normal vector map. Based on the total loss value, gradient backpropagation can be performed to update the properties of each initial Gaussian surface element, etc., using the gradient of the total loss value.

[0061] For example, using a monocular estimation pedestal model to perform depth estimation on each scene image to be reconstructed, the corresponding initial inverse depth map is obtained. (i.e., the reciprocal of depth), calculate the inverse depth median. and the average of the absolute values ​​of the inverse depth and the median deviation .

[0062] By projecting the sparse point cloud corresponding to each scene image to be reconstructed back to the viewpoint corresponding to each scene image to be reconstructed, the sparse inverse depth distribution of each scene image to be reconstructed at the actual scale is obtained. and the average of the absolute values ​​of the sparse inverse depth and the median deviation The depth estimated by the monocular estimation base model is scaled down to the actual scale to obtain the target inverse depth map. It can be represented by the following formula (1):

[0063] (1)

[0064] The loss function for depth estimation can be expressed as shown in equation (2):

[0065] (2)

[0066] in, This is an inverse depth map. This is used to estimate the loss value for depth.

[0067] Based on the initial Gaussian polygons and the viewpoint of the corresponding RGB image, the rasterization pipeline can render the corresponding scene image, namely, RGB image I, depth map D, and normal vector map N. Taking the reciprocal of the depth map yields the inverse depth map. The image loss function, also known as the RGB image loss function, is shown in equation (3) below:

[0068] (3)

[0069] Among them, L RGB Image loss value, Let λ be the target scene image, and λ be a parameter.

[0070] The normal vector loss function is shown in equation (4) below:

[0071] (4)

[0072] in, For the target normal vector map, This represents the loss value of the normal vector.

[0073] The overall loss function is shown in equation (5) below:

[0074] (5)

[0075] Where α is the weight of the depth estimation loss function, β is the weight of the normal vector loss function.

[0076] The loss function for the projection position of the imaging plane is shown in equation (6) below:

[0077] (6)

[0078] Accumulation between different iteration steps Obtain the gradient When the preset number of iterations is N, densification is performed, and the accumulated values ​​from these N iterations are... If the average value is greater than the average threshold and the ratio of the major axis to the minor axis of the optimized initial Gaussian surface is less than the preset ratio, the optimized initial Gaussian surface will be cloned or split.

[0079] According to embodiments of the present invention, a surface densification technique based on elongation filtering and gradient separation is used to perform gradient backpropagation based on the total loss value, iteratively optimizing the initial Gaussian surface elements to accelerate optimization. When the average gradient at the projection position on the imaging plane is greater than the average threshold, and the aspect ratio of the optimized initial Gaussian surface elements is less than a preset aspect ratio, the optimized initial Gaussian surface elements are cloned or split, avoiding excessive growth in the number of Gaussian surface elements. Combining the elongation of the Gaussian surface elements with the gradient of the SSIM loss to select candidate Gaussian surface elements for densification effectively improves the performance degradation caused by blur artifacts in the early stages of training and avoids excessive growth in the number of surface elements caused by the degradation of Gaussian surface elements during projection, ensuring stable training and effectively eliminating blur artifacts while accelerating the convergence speed of adaptive optimization.

[0080] Figure 3 The diagram illustrates the overall structure of the Gaussian surface element adaptive optimization algorithm according to an embodiment of the present invention.

[0081] Another embodiment of the present invention provides an image processing method for scene reconstruction and rendering, which includes the above-mentioned features. Figure 2In addition to the operations S210~S240 shown, the following operations may also be included: based on spatial relationships, meshing multiple first target Gaussian elements to obtain multiple sub-models; based on the pose of the scene image to be reconstructed and the rendering results of the sub-models, determining the target scene image to be reconstructed that matches each sub-model from multiple scene images to be reconstructed; fine-tuning the first target Gaussian elements in each sub-model based on the inverse depth map, the target scene image to be reconstructed, and the normal vector map of the target scene image to be reconstructed; and determining the importance of the first target Gaussian elements in each sub-model when the number of fine-tunings meets the preset number of iterations.

[0082] According to an embodiment of the present invention, each dimension of the coordinates of multiple first target Gaussian surface elements can be mapped to the [0,1] interval through nonlinear compression and uniformly divided into m×n grids, with the first target Gaussian surface elements in each grid constituting a sub-model.

[0083] According to an embodiment of the present invention, the rendering result of the sub-model may include a first rendering result obtained when the opacity of the first target Gaussian surface element contained in the sub-model is set to zero, and a second rendering result obtained when the opacity is kept at its original value.

[0084] Based on the pose of the scene image to be reconstructed and the rendering results of the sub-model, the target scene image to be reconstructed that matches each sub-model is determined from multiple scene images to be reconstructed. This may include the following operation: if the pose of the scene image to be reconstructed is determined to be within the mesh space of the sub-model, and the loss values ​​of the first rendering result and the second rendering result exceed a preset loss value, the scene image to be reconstructed is determined as the target scene image to be reconstructed that matches each sub-model.

[0085] According to an embodiment of the present invention, the first target Gaussian surface element in each sub-model is fine-tuned based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed, and the normal vector map of the target scene image to be reconstructed.

[0086] For example, a separate Graphics Processing Unit (GPU) can be allocated to each sub-model for training, and the target scene image to be reconstructed matched with each sub-model can be loaded onto the same GPU, such as... Figure 3The Gaussian surface adaptive optimization algorithm shown is used for iterative optimization. This involves projecting the first target Gaussian surface in the sub-model, such as a 2D Gaussian surface, and the camera pose of the target scene image to be reconstructed that matches the 2D Gaussian surface. Then, the rasterization pipeline is used for rendering, and the total loss value is calculated. Based on the total loss value, an adaptive density control is performed using a stretching filter to obtain the optimized 2D Gaussian surface. This iterative optimization is repeated until the number of iterations meets the preset number of iterations. Then, the importance of each Gaussian surface is calculated and pruning is performed.

[0087] The Gaussian facet adaptive optimization algorithm significantly accelerates the convergence speed of rendering and geometric quality and improves the robustness and stability of the training process by introducing depth regression technology based on monocular estimation pedestal model and facet densification technology based on elongation filtering and gradient separation.

[0088] After each sub-model has been fine-tuned, a sub-model merging operation can be performed to obtain the final scene representation, which consists of multiple first-objective Gaussian elements. When there is a need for a mesh representation of the scene, a mesh file suitable for downstream tasks can be extracted from the model using a common 3D reconstruction technique, namely TSDF (Truncated Signed Distance Function), combined with the Marching Cubes algorithm.

[0089] According to an embodiment of the present invention, when it is determined that the number of fine-tuning iterations meets the preset number of iterations, determining the importance of the first target Gaussian surface element in each sub-model may include the following operations: when it is determined that the number of fine-tuning iterations meets the preset number of iterations, determining the number of pixels covered by the projection of each first target Gaussian surface element onto the target scene image to be reconstructed; determining the opacity of each first target Gaussian surface element; and determining the importance of the first target Gaussian surface element in each sub-model based on the pixels, the opacity, and the first target Gaussian surface element in each sub-model.

[0090] For example, after each K number of fine-tuning operations, the importance C of each first target Gaussian element is calculated. n As shown in equations (7) to (8):

[0091] (7)

[0092] (8)

[0093] in, It is the set of pixels covered by the projection of the nth first Gaussian element onto the kth target scene image to be reconstructed, α n Let n(p) be the opacity of the nth first Gaussian pixel, and n(p) be the first target Gaussian pixel after intersecting the ray passing through pixel p and sorted by depth. It is the set of target scene images to be reconstructed that are matched with sub-model m.

[0094] According to an embodiment of the present invention, sub-models are divided based on spatial relationships, and training data is allocated according to the contribution of each sub-model to the rendering results of each training perspective; distributed fine-tuning, parallel iterative optimization and pruning of each sub-model, and integration of the training results of each sub-model can obtain a unified representation of the scene, thereby simplifying the sub-models, avoiding redundant post-processing, and reducing reconstruction overhead in terms of storage, GPU memory, and fine-tuning time.

[0095] According to embodiments of the present invention, regarding the above... Figure 2 Operation S240, based on the spatial extent of the grid, determines a third target Gaussian surface from the second target Gaussian surface for rendering the reconstructed scene image, and may include the operation of: determining the second target Gaussian surface located at the center of the spatial extent of the grid as the third target Gaussian surface according to the spatial extent of the grid.

[0096] According to an embodiment of the present invention, for all sub-models, a second target Gaussian surface element whose center coordinates are located within the spatial range of the grid can be extracted and determined as a third target Gaussian surface element. The union of the third target Gaussian surface elements extracted from each sub-block is used as the final scene representation model.

[0097] According to an embodiment of the present invention, the reconstructed third target Gaussian surface element has high reconstruction accuracy and can be used to render realistic reconstructed scene images.

[0098] According to an embodiment of the present invention, the third target Gaussian element may include multiple elements.

[0099] Another embodiment of the present invention provides an image processing method for scene reconstruction and rendering, which includes the above-mentioned features. Figure 2 In addition to the operations S210~S240 shown, the following operations may also be included: vector quantization of the spherical harmonic coefficients of the third target Gaussian surface element with the lowest importance; setting a preset precision for the spherical harmonic coefficients of the other third target Gaussian surface elements among the multiple third target Gaussian surface elements except for the third target Gaussian surface element with the lowest importance; and compression of the multiple third target Gaussian surface elements based on an octree compression scheme.

[0100] According to an embodiment of the present invention, the spherical harmonic coefficients of the least important third target Gaussian surface element can be vector-quantized, while the spherical harmonic coefficients of the remaining third target Gaussian surface elements, as well as other attributes including surface element position and opacity, are saved with float16 precision.

[0101] According to an embodiment of the present invention, by quantization compression, the numerical precision of the least important Gaussian elements is reduced, thereby reducing memory overhead.

[0102] According to embodiments of the present invention, after scene reconstruction and compression are completed, the scene representation can be stored and transmitted in the form of a compressed file. After decompression, the model can be used for real-time rendering and browsing, meeting the requirements of real-time rendering. Alternatively, the Gaussian surface representation of the scene can be transformed into a mesh representation using a Marching Cube algorithm based on TSDF fields, thereby enabling further interactive editing, lighting reset, and other operations.

[0103] Figure 4 The diagram illustrates the overall structure of a distributed parallel training pipeline according to an embodiment of the present invention.

[0104] According to an embodiment of the present invention, implementing an image processing method for scene reconstruction and rendering may include two key components, such as Figure 3 The Gaussian surface element adaptive optimization algorithm shown and as follows Figure 4 The diagram shows a distributed parallel training pipeline. This pipeline is used to implement distributed training, model pruning, and quantization compression, solving the problems of high training overhead and long training time in existing algorithms.

[0105] like Figure 4 As shown, this pipeline can include four stages: pre-training, sub-model and data partitioning, distributed fine-tuning, and quantization compression. Pre-training employs methods such as... Figure 3The proposed Gaussian facet adaptive optimization algorithm iteratively optimizes across all training views (multiple images of the scene to be reconstructed) to obtain a coarse representation of the global scene, namely multiple first-target Gaussian facets. In the sub-model and data partitioning stage, the coordinates of the pre-trained first-target Gaussian facets are mapped to a nonlinear space and partitioned into a series of sub-models based on spatial relationships. Further, for each sub-model, the algorithm traverses all training views and assigns those whose camera poses fall within the sub-model space or whose sub-models significantly contribute to the rendering results of that view to the training set of that sub-model. In the distributed fine-tuning stage, each sub-model and its training data are allocated to a corresponding GPU, and the proposed Gaussian facet adaptive optimization algorithm is used for iterative optimization and facet densification. Simultaneously, during fine-tuning, after each sub-model completes a preset number of iterations on the training set, its training set is traversed to calculate the importance of each Gaussian facet, and all facets with importance less than a given percentage threshold are deleted, thus pruning and compressing the model without significantly impairing its performance. After all sub-models have completed training, they are integrated to obtain the final scene representation, namely the second target Gaussian surface. During the quantization and compression phase, the algorithm traverses all training views to evaluate the importance of each second Gaussian surface in the model output from the fine-tuning phase. The spherical harmonic coefficients of the second Gaussian surface with the lowest importance are vector-quantized, while the spherical harmonic coefficients of the remaining second Gaussian surface, as well as other attributes including surface position and opacity, are saved with float16 precision.

[0106] According to embodiments of the present invention, the image processing method for scene reconstruction and rendering provided by the present invention can utilize multi-view RGB images, including aerial and street view perspectives, as input to perform high-quality reconstruction and real-time rendering of large-scale scenes, with significant improvements in surface reconstruction accuracy and reconstruction overhead. Visual verification and comparative experiments demonstrate its effectiveness and robustness in large-scale scene reconstruction tasks, indicating broad application prospects.

[0107] Figure 5 A schematic block diagram of an image processing apparatus for scene reconstruction and rendering according to an embodiment of the present invention is shown.

[0108] like Figure 5 As shown, the image processing apparatus 500 for scene reconstruction and rendering includes an initialization module 510, an optimization module 520, a first determination module 530, and a second determination module 540.

[0109] The initialization module 510 is used to initialize the sparse point cloud corresponding to each scene image to be reconstructed in multiple scene images to obtain multiple initial Gaussian elements. The sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool.

[0110] The optimization module 520 is used to optimize each initial Gaussian surface based on the inverse depth map, the rendered scene image, and the normal vector map obtained from the initial Gaussian surface rendering, to obtain multiple first target Gaussian surfaces.

[0111] The first determining module 530 is used to determine a second target Gaussian surface from multiple first target Gaussian surface elements based on the importance of the first target Gaussian surface elements. The importance of the first target Gaussian surface elements is determined based on multiple sub-models and the target scene image to be reconstructed that matches each sub-model. The multiple sub-models are obtained by meshing multiple first target Gaussian surface elements. Each sub-model consists of first target Gaussian surface elements within each mesh. The target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model.

[0112] The second determining module 540 is used to determine a third target Gaussian surface from the second target Gaussian surface for rendering the reconstructed scene image, based on the spatial extent of the grid.

[0113] According to an embodiment of the present invention, the image processing apparatus 500 for scene reconstruction and rendering further includes a depth estimation module, a third determination module, and a fourth determination module.

[0114] The depth estimation module uses a monocular estimation base model to estimate the depth of each scene image to be reconstructed, obtaining an initial inverse depth map for each image. The third determination module determines a target inverse depth map with the same size as each scene image to be reconstructed, based on the initial inverse depth map. The fourth determination module determines a target normal map based on the target inverse depth map and the target scene image.

[0115] According to an embodiment of the present invention, the image processing apparatus 500 for scene reconstruction and rendering further includes a mesh division module, a fifth determination module, a fine-tuning module, and a sixth determination module.

[0116] The mesh generation module is used to mesh multiple first target Gaussian elements based on spatial relationships, resulting in multiple sub-models. The fifth determination module is used to determine the target scene image matching each sub-model from multiple scene images, based on the pose of the scene image to be reconstructed and the rendering results of the sub-models. The fine-tuning module is used to fine-tune the first target Gaussian elements in each sub-model based on the inverse depth map, the target scene image itself, and its normal vector map. The sixth determination module is used to determine the importance of the first target Gaussian elements in each sub-model, provided that the number of fine-tuning iterations meets a preset number of iterations.

[0117] According to an embodiment of the present invention, the image processing apparatus 500 for scene reconstruction and rendering further includes a vectorization module, a configuration module, and a compression module.

[0118] The vectorization module is used to perform vector quantization on the spherical harmonic coefficients of the least important third-objective Gaussian surface element. The configuration module is used to configure the preset precision for the spherical harmonic coefficients of the other third-objective Gaussian surface elements, excluding the least important one. The compression module is used to compress the multiple third-objective Gaussian surface elements using an octree-based compression scheme.

[0119] According to embodiments of the present invention, any plurality of modules among the initialization module 510, optimization module 520, first determination module 530, and second determination module 540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the initialization module 510, optimization module 520, first determination module 530, and second determination module 540 can be at least partially implemented as hardware circuitry, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), on-chip methods, substrate-on-a-chip methods, package-on-a-package methods, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the initialization module 510, optimization module 520, first determination module 530, and second determination module 540 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0120] It should be noted that the image processing device part for scene reconstruction and rendering in the embodiments of the present invention corresponds to the image processing method part for scene reconstruction and rendering in the embodiments of the present invention. The description of the image processing device part for scene reconstruction and rendering is specifically referred to in the image processing method part for scene reconstruction and rendering, and will not be repeated here.

[0121] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method for scene reconstruction and rendering according to an embodiment of the present invention.

[0122] like Figure 6As shown, an electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0123] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0124] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0125] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0126] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0127] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0128] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0129] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0130] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0131] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0134] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. An image processing method for scene reconstruction and rendering, characterized in that, The method includes: Initialize the sparse point cloud corresponding to each of the multiple scene images to be reconstructed to obtain multiple initial Gaussian elements, wherein the sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool. Based on the inverse depth map, rendered scene image and normal vector map obtained by the initial Gaussian surface rendering, each initial Gaussian surface is optimized to obtain multiple first target Gaussian surfaces; Based on the importance of the first target Gaussian surface element, a second target Gaussian surface element is determined from multiple first target Gaussian surface elements. The importance of the first target Gaussian surface element is determined based on multiple sub-models and the target scene image to be reconstructed that matches each sub-model. The multiple sub-models are obtained by meshing multiple first target Gaussian surface elements. Each sub-model is composed of the first target Gaussian surface elements in each mesh. The target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model. Based on the spatial range of the grid, a third target Gaussian surface is determined from the second target Gaussian surface for rendering the reconstructed scene image; The method further includes: Based on spatial relationships, multiple first target Gaussian surface elements are meshed to obtain multiple sub-models; Based on the pose of the scene image to be reconstructed and the rendering result of the sub-model, the target scene image to be reconstructed that matches each sub-model is determined from multiple scene images to be reconstructed; Based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed, and the normal vector map of the target scene image to be reconstructed, the first target Gaussian surface element in each sub-model is fine-tuned; If the number of fine-tunings satisfies the preset number of iterations, determine the pixels covered by the projection of each first target Gaussian surface element onto the target scene image to be reconstructed; Determine the opacity of each first target Gaussian element; and The importance of the first target Gaussian surface in each sub-model is determined based on the pixel, the opacity, and the first target Gaussian surface in each sub-model.

2. The method according to claim 1, characterized in that, The method further includes: The depth of each of the scene images to be reconstructed is estimated using a monocular estimation base model to obtain an initial inverse depth map corresponding to each scene image to be reconstructed. Based on the initial inverse depth map, a target inverse depth map with the same size as each of the images of the scene to be reconstructed is determined; and Based on the target inverse depth map and the target scene image, the target normal vector map is determined.

3. The method according to claim 2, characterized in that, The inverse depth map, rendered scene image, and normal vector map obtained based on the initial Gaussian surface are used to optimize each initial Gaussian surface to obtain multiple first target Gaussian surfaces, including: For each of the initial Gaussian elements: Based on the target inverse depth map and the inverse depth map, determine the depth estimation loss value; Based on the target scene image and the rendered scene image, determine the image loss value; Based on the target normal vector map and the normal vector map, determine the normal vector loss value; The total loss value is obtained by summing the depth estimation loss value, the image loss value, and the normal vector loss value. Based on the total loss value, each initial Gaussian surface element is iteratively optimized; Given that the number of iterations meets the preset number of iterations, determine the gradient of the projection position on the imaging plane; and If the average gradient of the projection position of the imaging plane is greater than the average threshold, and the major-minor axis ratio of the optimized initial Gaussian surface element is less than the preset major-minor axis ratio, the optimized initial Gaussian surface element is cloned or split to obtain multiple first target Gaussian surface elements.

4. The method according to claim 1, characterized in that, The rendering result of the sub-model includes a first rendering result of the first target Gaussian surface element contained in the sub-model when the opacity is set to zero and a second rendering result when the opacity is kept at the original value; The step of determining the target scene image to be reconstructed that matches each of the sub-models from multiple scene images to be reconstructed, based on the pose of the scene image to be reconstructed and the rendering result of the sub-model, includes: If the pose of the scene image to be reconstructed is determined to be within the mesh space of the sub-model, and the loss values ​​of the first rendering result and the second rendering result exceed the preset loss value, the scene image to be reconstructed is determined to be the target scene image to be reconstructed that matches each sub-model.

5. The method according to claim 1, characterized in that, The step of determining a third target Gaussian surface for rendering the reconstructed scene image from the second target Gaussian surface based on the spatial extent of the grid includes: Based on the spatial range of the grid, the second target Gaussian surface element located at the center of the spatial range of the grid is determined as the third target Gaussian surface element.

6. The method according to claim 1, characterized in that, The third target Gaussian surface element includes multiple elements; The method further includes: The spherical harmonic coefficients of the third target Gaussian surface element, which has the lowest importance, are vector-quantized; Configure a preset precision for the spherical harmonic coefficients of the third target Gaussian surface elements, excluding the third target Gaussian surface element with the lowest importance; and An octree-based compression scheme is used to compress multiple third target Gaussian surface elements.

7. An image processing apparatus for scene reconstruction and rendering, characterized in that, The device includes: An initialization module is used to initialize the sparse point cloud corresponding to each of the multiple scene images to be reconstructed, and obtain multiple initial Gaussian surface elements. The sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool. The optimization module is used to optimize each of the initial Gaussian surfaces based on the inverse depth map, the rendered scene image, and the normal vector map obtained by rendering the initial Gaussian surfaces, so as to obtain multiple first target Gaussian surfaces. A first determining module is configured to determine a second target Gaussian surface element from a plurality of first target Gaussian surface elements based on the importance of the first target Gaussian surface element. The importance of the first target Gaussian surface element is determined based on a plurality of sub-models and a target scene image to be reconstructed that matches each sub-model. The plurality of sub-models are obtained by meshing the plurality of first target Gaussian surface elements, and each sub-model consists of the first target Gaussian surface elements within each mesh. The target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model. The second determining module is used to determine a third target Gaussian surface element for rendering the reconstructed scene image from the second target Gaussian surface element based on the spatial range of the grid. The device further includes: The mesh generation module is used to mesh multiple first target Gaussian surface elements based on spatial relationships to obtain multiple sub-models; The fifth determining module is used to determine, based on the pose of the scene image to be reconstructed and the rendering result of the sub-model, the target scene image to be reconstructed that matches each sub-model from multiple scene images to be reconstructed; The fine-tuning module is used to fine-tune the first target Gaussian surface element in each of the sub-models based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed, and the normal vector map of the target scene image to be reconstructed. The sixth determining module is used to determine, when the number of fine-tunings meets the preset number of iterations, the pixels covered by the projection of each first target Gaussian surface element on the target scene image to be reconstructed; determine the opacity of each first target Gaussian surface element; and determine the importance of the first target Gaussian surface element in each sub-model based on the pixels, the opacity, and the first target Gaussian surface element in each sub-model.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 6.

Images