Image processing method, device and equipment for scene reconstruction and rendering

Abstract

The invention provides an image processing method, device and equipment for scene reconstruction and rendering. The method comprises the following steps: initializing sparse point clouds corresponding to each scene image to be reconstructed to obtain a plurality of initial Gaussian surface elements; optimizing each initial Gaussian surface element based on an inverse depth map, a rendering scene image and a normal vector map obtained by rendering the initial Gaussian surface elements to obtain a plurality of first target Gaussian surface elements; based on the importance of the first target Gaussian surface elements, determining a second target Gaussian surface element, the importance being determined based on a plurality of sub-models and a target scene image to be reconstructed matched with each sub-model, the plurality of sub-models being obtained by performing mesh generation on the plurality of first target Gaussian surface elements, and the plurality of sub-models being obtained by performing mesh generation on the plurality of first target Gaussian surface elements; the target scene image to be reconstructed matched with each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model; and determining a third target Gaussian surface element for rendering the reconstructed scene image based on the spatial range of the grid.

Description

Technical Field

[0001] The present invention relates to the fields of computer science, artificial intelligence, monocular depth estimation and three-dimensional modeling technology, and in particular to an image processing method, device and equipment for scene reconstruction and rendering. Background Art

[0002] 3D scene reconstruction is a long-standing topic in the field of computer vision and graphics, and its core pursuit is realistic visual rendering and accurate geometric reconstruction. The current mainstream algorithm is based on the representation of 3D Gaussian Splatting (3DGS), which uses a set of discrete Gaussian ellipsoids expressed by 3D Gaussian distribution to fit and reconstruct the scene, and uses a highly optimized rasterization pipeline for rendering. Due to its advantages in training convergence and rendering efficiency, this algorithm has gradually replaced Neural Radiance Fields (NeRF) and become the mainstream technology in this field.

[0003] In the process of realizing the concept of the present invention, the inventors found that the current three-dimensional scene reconstruction still has problems such as low reconstruction accuracy, high cost, and low fidelity of the rendered scene graph. Summary of the invention

[0004] In view of this, the present invention provides an image processing method, apparatus, device, medium and program product for scene reconstruction and rendering.

[0005] One aspect of the present invention provides an image processing method for scene reconstruction and rendering, including: initializing a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed to obtain a plurality of initial Gaussian face elements, wherein the sparse point cloud is obtained by processing the scene image to be reconstructed using a preset reconstruction tool; optimizing each initial Gaussian face element based on an inverse depth map, a rendered scene image, and a normal vector map obtained by rendering the initial Gaussian face elements to obtain a plurality of first target Gaussian face elements; determining a second target Gaussian face element from a plurality of first target Gaussian face elements based on the importance of the first target Gaussian face element, wherein the importance of the first target Gaussian face element is determined based on a plurality of sub-models and a target scene image to be reconstructed matching each sub-model, the plurality of sub-models are obtained by gridding the plurality of first target Gaussian face elements, each sub-model is composed of the first target Gaussian face elements in each grid, and the target scene image to be reconstructed matching each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model; and determining a third target Gaussian face element for rendering and reconstructing the scene image from the second target Gaussian face elements based on the spatial range of the grid.

[0006] According to an embodiment of the present invention, the image processing method for scene reconstruction and rendering also includes: using a monocular estimation base model to perform depth estimation on each scene image to be reconstructed to obtain an initial inverse depth map corresponding to each scene image to be reconstructed; based on the initial inverse depth map, determining a target inverse depth map that is consistent with the size of each scene image to be reconstructed; and determining a target normal vector map based on the target inverse depth map and the target scene image.

[0007] According to an embodiment of the present invention, based on the inverse depth map, rendered scene image and normal map obtained by rendering the initial Gaussian surface element, each initial Gaussian surface element is optimized to obtain multiple first target Gaussian surface elements, including: for each initial Gaussian surface element: based on the target inverse depth map and the inverse depth map, a depth estimation loss value is determined; based on the target scene image and the rendered scene image, an image loss value is determined; based on the target normal map and the normal map, a normal loss value is determined; according to the sum of the depth estimation loss value, the image loss value and the normal loss value, a total loss value is obtained; according to the total loss value, the initial Gaussian surface element is iteratively optimized; when it is determined that the number of iterative optimizations meets a preset number of iterations, a gradient of the projection position of the imaging plane is determined; and when it is determined that the average value of the gradient of the projection position of the imaging plane is greater than an average threshold, and the major-minor axis ratio of the optimized initial Gaussian surface element is less than the preset major-minor axis ratio, the optimized initial Gaussian surface element is cloned or split to obtain multiple first target Gaussian surface elements.

[0008] According to an embodiment of the present invention, the image processing method for scene reconstruction and rendering also includes: based on the spatial relationship, meshing multiple first target Gaussian surface elements to obtain multiple sub-models; based on the pose of the scene image to be reconstructed and the rendering results of the sub-models, determining the target scene image to be reconstructed that matches each sub-model from the multiple scene images to be reconstructed; based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed and the normal vector map of the target scene image to be reconstructed, fine-tuning the first target Gaussian surface element in each sub-model; and when it is determined that the number of fine-tuning meets the preset number of iterations, determining the importance of the first target Gaussian surface element in each sub-model.

[0009] According to an embodiment of the present invention, the rendering result of the sub-model includes a first rendering result obtained when the opacity of the first target Gaussian surface element contained in the sub-model is set to zero and a second rendering result obtained when the opacity is maintained at the original value; based on the pose of the scene image to be reconstructed and the rendering result of the sub-model, a target scene image to be reconstructed that matches each sub-model is determined from multiple scene images to be reconstructed, including: when it is determined that the pose of the scene image to be reconstructed is within the grid space of the sub-model and the loss values ​​of the first rendering result and the second rendering result exceed a preset loss value, the scene image to be reconstructed is determined as the target scene image to be reconstructed that matches each sub-model.

[0010] According to an embodiment of the present invention, when it is determined that the number of fine-tuning meets the preset number of iterations, the importance of the first target Gaussian surface element in each sub-model is determined, including: when it is determined that the number of fine-tuning meets the preset number of iterations, determining the pixels covered by the projection of each first target Gaussian surface element on the target scene image to be reconstructed; determining the opacity of each first target Gaussian surface element; and determining the importance of the first target Gaussian surface element in each sub-model based on the pixels, opacity and the first target Gaussian surface element in each sub-model.

[0011] According to an embodiment of the present invention, based on the spatial range of the grid, a third target Gaussian surface element for rendering and reconstructing a scene image is determined from the second target Gaussian surface element, including: according to the spatial range of the grid, the second target Gaussian surface element located at the center of the spatial range of the grid is determined as the third target Gaussian surface element.

[0012] According to an embodiment of the present invention, the third target Gaussian surface element includes multiple third target Gaussian surface elements; the image processing method for scene reconstruction and rendering also includes: vector quantizing the spherical harmonic coefficients of the third target Gaussian surface element with the lowest importance; configuring a preset precision for the spherical harmonic coefficients of other third target Gaussian surface elements except the third target Gaussian surface element with the lowest importance among the multiple third target Gaussian surface elements; and compressing the multiple third target Gaussian surface elements based on an octree compression scheme.

[0013] The second aspect of the present invention provides an image processing device for scene reconstruction and rendering, comprising: an initialization module, for initializing a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed, to obtain a plurality of initial Gaussian face elements, wherein the sparse point cloud is obtained by processing the scene image to be reconstructed using a preset reconstruction tool; an optimization module, for optimizing each initial Gaussian face element based on an inverse depth map, a rendered scene image, and a normal vector map obtained by rendering the initial Gaussian face element, to obtain a plurality of first target Gaussian face elements; a first determination module, for optimizing, based on the importance of the first target Gaussian face element, from the plurality of first target Gaussian face elements. The invention discloses a method for determining a second target Gaussian surface element from the standard Gaussian surface element, wherein the importance of the first target Gaussian surface element is determined based on a plurality of sub-models and a target scene image to be reconstructed that matches each sub-model, the plurality of sub-models are obtained by gridding a plurality of first target Gaussian surface elements, each sub-model is composed of the first target Gaussian surface elements in each grid, and the target scene image to be reconstructed that matches each sub-model is determined based on the posture of the scene image to be reconstructed and the rendering result of the sub-model; and a second determination module is used to determine a third target Gaussian surface element for rendering and reconstructing the scene image from the second target Gaussian surface element based on the spatial range of the grid.

[0014] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the image processing method for scene reconstruction and rendering.

[0015] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implements the steps of the above-mentioned image processing method for scene reconstruction and rendering.

[0016] The fifth aspect of the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned image processing method for scene reconstruction and rendering.

[0017] According to an embodiment of the present invention, based on the inverse depth map, rendered scene image and normal map obtained by rendering the initial Gaussian surface element, each initial Gaussian surface element is optimized, which can provide reliable depth supervision for multiple scene images to be reconstructed without depth annotations, accelerate the optimization of Gaussian surface elements, and eliminate Gaussian surface element blurring artifacts. Based on the importance of the first target Gaussian surface element, the second target Gaussian surface element is determined, and the third target Gaussian surface element is determined based on the spatial range of the grid, which avoids redundant post-processing processes, enables high-quality reconstruction and real-time rendering of the scene, and has high reconstruction accuracy and low reconstruction overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0019] Figure 1 An application scenario diagram schematically illustrates an image processing method, apparatus, device, medium, and program product for scene reconstruction and rendering according to an embodiment of the present invention;

[0020] Figure 2 Schematically shows a flow chart of an image processing method for scene reconstruction and rendering according to an embodiment of the present invention;

[0021] Figure 3 The overall structure of the Gaussian facet adaptive optimization algorithm according to an embodiment of the present invention is schematically shown;

[0022] Figure 4 The overall structure diagram of the distributed parallel training pipeline according to an embodiment of the present invention is schematically shown;

[0023] Figure 5A block diagram schematically shows a structure of an image processing device for scene reconstruction and rendering according to an embodiment of the present invention; and

[0024] Figure 6 A block diagram of an electronic device suitable for implementing an image processing method for scene reconstruction and rendering according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0025] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a method having at least one of A, B and C" should include but is not limited to a method having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] In the process of realizing the concept of the present invention, the inventors found that in order to solve the ambiguity and inaccuracy of 3DGS representation in the surface estimation process, 2D Gaussian Splatting (2DGS) dimensionally collapsed the Gaussian distribution form used by 3DGS and used a set of discrete Gaussian surface pairs to represent the three-dimensional structure of the scene. At the same time, 2DGS introduced constraints on the distribution of normal vectors to ensure the smoothness and integrity of the surface reconstruction results and improve the accuracy of the geometric structure.

[0030] The monocular depth estimation technology based on the pedestal model draws on paradigms in the fields of computer vision and natural language processing. It is trained on a large-scale dataset that contains both unlabeled images and images with depth annotations. The resulting model can accurately estimate the depth of the input monocular RGB image and has good generalization ability. It plays a key role in technologies such as robotics, autonomous driving, and virtual reality.

[0031] The combination and optimization of technologies from multiple fields, such as scene representation based on three-dimensional Gaussian ellipsoids, geometric reconstruction based on two-dimensional Gaussian surfaces, and monocular depth estimation based on base models, aims to improve the rendering quality, geometric accuracy, training efficiency, and rendering speed of large-scale scene reconstruction, and meet the needs of different application scenarios for high-quality, high-efficiency, and lightweight reconstruction results.

[0032] Based on this, an embodiment of the present invention provides an image processing method for scene reconstruction and rendering, including: initializing a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed to obtain a plurality of initial Gaussian surface elements, wherein the sparse point cloud is obtained by processing the scene image to be reconstructed using a preset reconstruction tool; optimizing each initial Gaussian surface element based on an inverse depth map, a rendered scene image, and a normal vector map obtained by rendering the initial Gaussian surface element to obtain a plurality of first target Gaussian surface elements; determining a second target Gaussian surface element from a plurality of first target Gaussian surface elements based on the importance of the first target Gaussian surface element, wherein the importance of the first target Gaussian surface element is determined based on a plurality of sub-models and a target scene image to be reconstructed matching each sub-model, the plurality of sub-models are obtained by gridding a plurality of first target Gaussian surface elements, each sub-model is composed of the first target Gaussian surface elements in each grid, and the target scene image to be reconstructed matching each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model; and determining a third target Gaussian surface element for rendering and reconstructing the scene image from the second target Gaussian surface element based on the spatial range of the grid.

[0033] Figure 1 An application scenario diagram schematically illustrates an image processing method, apparatus, device, medium, and program product for scene reconstruction and rendering according to an embodiment of the present invention.

[0034] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0035] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the image processing method for scene reconstruction and rendering provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the image processing device for scene reconstruction and rendering provided in the embodiment of the present invention can generally be set in the server 105. The image processing method for scene reconstruction and rendering provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image processing device for scene reconstruction and rendering provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0040] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The image processing method for scene reconstruction and rendering according to an embodiment of the present invention is described in detail.

[0041] Figure 2A flowchart of an image processing method for scene reconstruction and rendering according to an embodiment of the present invention is schematically shown.

[0042] like Figure 2 As shown, the image processing method for scene reconstruction and rendering includes operations S210 to S240.

[0043] In operation S210, a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed is initialized to obtain a plurality of initial Gaussian facets.

[0044] In operation S220, each initial Gaussian surface is optimized based on the inverse depth map, the rendered scene image, and the normal vector map obtained by rendering the initial Gaussian surface to obtain a plurality of first target Gaussian surfaces.

[0045] In operation S230 , a second target Gaussian surface is determined from among the plurality of first target Gaussian surface elements based on the importance of the first target Gaussian surface element.

[0046] In operation S240 , a third target Gaussian surface for rendering and reconstructing the scene image is determined from the second target Gaussian surface based on the spatial range of the grid.

[0047] According to an embodiment of the present invention, the sparse point cloud may be obtained by processing the scene image to be reconstructed using a preset reconstruction tool. The preset reconstruction tool may include, for example, a three-dimensional reconstruction tool COLMAP. The scene image to be reconstructed may include an RGB image. The multiple scene images to be reconstructed may be multi-view RGB images of the scene to be reconstructed.

[0048] According to an embodiment of the present invention, before each reconstruction task begins, multi-view RGB images of the target scene may be collected, such as a set of images obtained by extracting frames from a video of the target scene taken by a camera, or a set of images with poses taken by a drone of the target scene. These images may be processed by COLMAP to obtain camera parameters, poses of each view, and sparse point clouds used in subsequent image processing.

[0049] According to an embodiment of the present invention, optimizing each initial Gaussian surface element may include optimizing the attributes of each initial Gaussian surface element. Attributes may include, for example, but are not limited to, spherical harmonic coefficients, surface element positions, opacity, etc. The initial Gaussian surface element may be rendered to obtain an inverse depth map. The inverse depth map may be converted into a depth map, and a normal vector map may be estimated from the depth map. In combination with the rendered scene image, each initial Gaussian surface element may be continuously iteratively optimized to obtain multiple first target Gaussian surfaces.

[0050] According to an embodiment of the present invention, the importance of the first target Gaussian surface element is determined based on multiple sub-models and a target scene image to be reconstructed that matches each sub-model. The multiple sub-models are obtained by gridding multiple first target Gaussian surface elements, and each sub-model is composed of the first target Gaussian surface elements in each grid. The target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model.

[0051] According to an embodiment of the present invention, the importance of the first target Gaussian surface element may be compared with an importance threshold, and the first target Gaussian surface element whose importance is greater than or equal to the importance threshold may be determined as the second target Gaussian surface element.

[0052] According to an embodiment of the present invention, the second target Gaussian surface located at the center of the grid can be determined as the third target Gaussian surface used for rendering and reconstructing the scene image. Based on a conventional rendering method, the third target Gaussian surface can be used to render and reconstruct the scene image.

[0053] According to an embodiment of the present invention, based on the inverse depth map, rendered scene image and normal map obtained by rendering the initial Gaussian facet, each initial Gaussian facet is optimized, which can provide reliable depth supervision for multiple scene images to be reconstructed without depth annotations, accelerate the optimization of Gaussian facets, and eliminate Gaussian facet blurring artifacts. Based on the importance of the first target Gaussian facet, the second target Gaussian facet is determined, and the third target Gaussian facet is determined based on the spatial range of the grid, which avoids redundant post-processing processes, enables high-quality reconstruction and real-time rendering of large-scale scenes, and has high reconstruction accuracy and low reconstruction overhead.

[0054] According to an embodiment of the present invention, the image processing method for scene reconstruction and rendering includes the above Figure 2 In addition to the operations S210 to S240 shown, the following operations may also be included: using a monocular estimation base model to perform depth estimation on each scene image to be reconstructed to obtain an initial inverse depth map corresponding to each scene image to be reconstructed; based on the initial inverse depth map, determining a target inverse depth map that is consistent with the size of each scene image to be reconstructed; and determining a target normal vector map based on the target inverse depth map and the target scene image.

[0055] According to an embodiment of the present invention, the median of the inverse depth and the average of the absolute value of the deviation between the inverse depth and the median can be calculated based on multiple initial inverse depth maps. The sparse point cloud corresponding to each scene image to be reconstructed can be projected back to the viewing angle corresponding to each scene image to be reconstructed to obtain a sparse inverse depth distribution of each scene image to be reconstructed at an actual scale. The median of the sparse inverse depth and the average of the absolute value of the deviation between the sparse inverse depth and the median are calculated based on multiple sparse inverse depth distributions. Based on the initial inverse depth map, the median of the inverse depth and the average of the absolute value of the deviation between the inverse depth and the median, the median of the sparse inverse depth and the average of the absolute value of the deviation between the sparse inverse depth and the median, the depth estimated by the monocular estimation base model is scaled to the actual scale to obtain a target inverse depth map.

[0056] For example, the RGB image and pose information can be input into a monocular estimation base model, such as Depth-Anything-V2, to obtain the depth distribution corresponding to each perspective for the next step of depth supervision.

[0057] According to the embodiment of the present invention, the target inverse depth map can be converted into a target depth map, and the target normal vector map can be estimated from the target depth map and the target scene image.

[0058] According to an embodiment of the present invention, the depth regression technology based on the monocular estimation base model utilizes the monocular estimation base model to estimate the depth distribution of each training view, and scales it according to the actual scale of the scene to supervise the rendering depth of the model. It can provide reliable depth supervision for multiple scene images to be reconstructed without depth annotations, and can be used for adaptive optimization of Gaussian face elements.

[0059] According to an embodiment of the present invention, for the above Figure 2 In operation S220, based on the inverse depth map, rendered scene image and normal map obtained by rendering the initial Gaussian surface element, each initial Gaussian surface element is optimized to obtain multiple first target Gaussian surface elements, which may include the following operations: for each initial Gaussian surface element: based on the target inverse depth map and the inverse depth map, a depth estimation loss value is determined; based on the target scene image and the rendered scene image, an image loss value is determined; based on the target normal map and the normal map, a normal loss value is determined; according to the sum of the depth estimation loss value, the image loss value and the normal loss value, a total loss value is obtained; according to the total loss value, the initial Gaussian surface element is iteratively optimized; when it is determined that the number of iterative optimizations meets the preset number of iterations, the gradient of the projection position of the imaging plane is determined; and when it is determined that the average value of the gradient of the projection position of the imaging plane is greater than the average threshold, and the major-minor axis ratio of the optimized initial Gaussian surface element is less than the preset major-minor axis ratio, the optimized initial Gaussian surface element is cloned or split to obtain multiple first target Gaussian surface elements.

[0060] According to an embodiment of the present invention, for each initial Gaussian surface element: the depth estimation loss value can be obtained by calculating the absolute value of the difference between the inverse depth map and the target inverse depth map. The image loss value is obtained by calculating the loss of the rendered scene image and the target scene image. The normal vector loss value is obtained by calculating the absolute value of the difference between the normal vector map and the target normal vector map. According to the total loss value, gradient back propagation can be performed to update the attributes of each initial Gaussian surface element with the gradient of the total loss value, etc.

[0061] For example, the monocular estimation base model is used to estimate the depth of each scene image to be reconstructed to obtain the corresponding initial inverse depth map (i.e. the inverse of the depth), calculate the median of the inverse depth and the average of the absolute values ​​of the inverse depth deviations from the median .

[0062] The sparse point cloud corresponding to each scene image to be reconstructed is projected back to the viewpoint corresponding to each scene image to be reconstructed, and the sparse inverse depth distribution of each scene image to be reconstructed at the actual scale is obtained. and the average absolute value of the deviation of the sparse inverse depth from the median . Scale the depth estimated by the monocular estimation base model to the actual scale to obtain the target inverse depth map , which can be expressed as the following formula (1):

[0063] (1)

[0064] The loss function of depth estimation can be expressed as follows:

[0065] (2)

[0066] in, is the inverse depth map, is the depth estimation loss value.

[0067] According to the initial Gaussian surface element and the viewing angle of the corresponding RGB image, the rasterization pipeline can render the corresponding rendered scene image, that is, the RGB image I, the depth map D, and the normal vector map N. The inverse of the depth map is obtained by taking the inverse of the depth map The image loss function, also known as the RGB image loss function, is shown in the following formula (3):

[0068] (3)

[0069] Among them, L RGB is the image loss value, is the target scene image, and λ is a parameter.

[0070] The normal vector loss function is shown in the following formula (4):

[0071] (4)

[0072] in, is the target normal vector map, is the normal vector loss value.

[0073] The total loss function is shown in formula (5):

[0074] (5)

[0075] Among them, α is the weight of the depth estimation loss function, and β is the weight of the normal vector loss function.

[0076] The loss function of the projection position of the imaging plane is shown in the following formula (6):

[0077] (6)

[0078] Accumulate between different iterations Get the gradient When the number of iterations is preset to N, densification is performed, and the accumulated The average is performed, and when the average value is greater than the average threshold and the major-minor axis ratio of the optimized initial Gaussian surface element is less than the preset major-minor axis ratio, the optimized initial Gaussian surface element is cloned or split.

[0079] According to an embodiment of the present invention, based on the face densification technology of elongation filtering and gradient separation, according to the total loss value, gradient back propagation is performed, and the initial Gaussian face is iteratively optimized to accelerate the optimization. When the average value of the gradient at the projection position of the imaging plane is greater than the average threshold, and the major-minor axis ratio of the optimized initial Gaussian face is less than the preset major-minor axis ratio, the optimized initial Gaussian face is cloned or split to avoid excessive growth in the number of Gaussian faces. Combining the elongation of the Gaussian face and the gradient of the SSIM loss to select the candidate Gaussian face for densification can effectively improve the performance degradation caused by blurring artifacts in the early stages of training, and avoid excessive growth in the number of faces caused by the degradation of Gaussian faces during the projection process, ensuring the stability of the training process, effectively eliminating blurring artifacts and accelerating the convergence speed of adaptive optimization.

[0080] Figure 3 The overall structure of the Gaussian facet adaptive optimization algorithm according to an embodiment of the present invention is schematically shown.

[0081] In another embodiment of the present invention, the image processing method for scene reconstruction and rendering includes the above Figure 2In addition to the operations S210~S240 shown, the following operations may also be included: based on the spatial relationship, meshing multiple first target Gaussian surface elements to obtain multiple sub-models; based on the pose of the scene image to be reconstructed and the rendering results of the sub-models, determining the target scene image to be reconstructed that matches each sub-model from the multiple scene images to be reconstructed; based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed and the normal vector map of the target scene image to be reconstructed, fine-tuning the first target Gaussian surface element in each sub-model; and when it is determined that the number of fine-tuning meets the preset number of iterations, determining the importance of the first target Gaussian surface element in each sub-model.

[0082] According to an embodiment of the present invention, each dimension of the coordinates of multiple first target Gaussian face elements can be mapped to the [0,1] interval through nonlinear compression and evenly divided into m×n grids, and the first target Gaussian face elements in each grid constitute a sub-model.

[0083] According to an embodiment of the present invention, the rendering result of the sub-model may include a first rendering result obtained when the opacity of the first target Gaussian surface element included in the sub-model is set to zero and a second rendering result obtained when the opacity remains at the original value.

[0084] Based on the pose of the scene image to be reconstructed and the rendering results of the sub-models, determining a target scene image to be reconstructed that matches each sub-model from multiple scene images to be reconstructed can include the following operations: when it is determined that the pose of the scene image to be reconstructed is within the grid space of the sub-model and the loss values ​​of the first rendering result and the second rendering result exceed a preset loss value, determining the scene image to be reconstructed as the target scene image to be reconstructed that matches each sub-model.

[0085] According to an embodiment of the present invention, the first target Gaussian surface element in each sub-model is fine-tuned based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed, and the normal vector map of the target scene image to be reconstructed.

[0086] For example, a separate graphics processing unit (GPU) can be assigned to each sub-model for training, and the target scene image to be reconstructed matching each sub-model can be loaded into the same GPU, such as Figure 3The Gaussian surface adaptive optimization algorithm shown in the figure is iteratively optimized, that is, the first target Gaussian surface in the sub-model, such as the 2D Gaussian surface, and the camera pose of the target scene image to be reconstructed that matches the 2D Gaussian surface are projected, and then the rasterization pipeline is rendered, and then the total loss value is calculated. Based on the total loss value, the stretching filter is used to perform adaptive density control to obtain the optimized 2D Gaussian surface. The optimization is iteratively optimized in this way. When it is determined that the number of iterative optimizations meets the preset number of iterations, the importance of each Gaussian surface is calculated and a pruning operation is performed.

[0087] The Gaussian facet adaptive optimization algorithm significantly accelerates the convergence of rendering and geometric quality and improves the robustness and stability of the training process by introducing deep regression technology based on a monocular estimated base model and facet densification technology based on elongation filtering and gradient separation.

[0088] After each sub-model is fine-tuned, the sub-models can be merged to obtain the final scene representation, that is, multiple first target Gaussian surfaces. When there is a need for a mesh representation of the scene, the commonly used 3D reconstruction technology, namely the TSDF (Truncated Signed Distance Function) field combined with the Marching Cubes algorithm, can be used to extract a mesh file from the model that is convenient for downstream tasks.

[0089] According to an embodiment of the present invention, when it is determined that the number of fine-tuning meets the preset number of iterations, determining the importance of the first target Gaussian surface element in each sub-model may include the following operations: when it is determined that the number of fine-tuning meets the preset number of iterations, determining the pixels covered by the projection of each first target Gaussian surface element on the target scene image to be reconstructed; determining the opacity of each first target Gaussian surface element; and determining the importance of the first target Gaussian surface element in each sub-model based on the pixels, opacity and the first target Gaussian surface element in each sub-model.

[0090] For example, after completing the fine-tuning times K, the importance C of each first target Gaussian facet is calculated. n , as shown in the following equations (7) to (8):

[0091] (7)

[0092] (8)

[0093] in, is the set of pixels covered by the projection of the nth first Gaussian surface element on the kth target scene image to be reconstructed, α n is the opacity of the nth first Gaussian surface element, n(p) is the first target Gaussian surface element intersected by the ray passing through pixel p and sorted by depth, and is the set of target scene images to be reconstructed that matches the sub-model m.

[0094] According to an embodiment of the present invention, sub-models are divided based on spatial relationships, and training data is allocated according to the contribution of each sub-model to the rendering results of each training perspective; distributed fine-tuning, iterative optimization and pruning of each sub-model in parallel, and integration of the training results of each sub-model can obtain a unified representation of the scene, thereby simplifying the sub-models, avoiding redundant post-processing processes, and reducing reconstruction overhead in terms of storage, GPU memory, and fine-tuning time.

[0095] According to an embodiment of the present invention, for the above Figure 2 In operation S240, based on the spatial range of the grid, determining a third target Gaussian surface element for rendering and reconstructing the scene image from the second target Gaussian surface element may include the following operations: according to the spatial range of the grid, determining the second target Gaussian surface element located at the center of the spatial range of the grid as the third target Gaussian surface element.

[0096] According to an embodiment of the present invention, for all sub-models, the second target Gaussian surface element whose center coordinates are within the spatial range of the grid can be extracted as the third target Gaussian surface element, and the union of the third target Gaussian surface elements extracted from each sub-block is used as the final scene representation model.

[0097] According to the embodiment of the present invention, the reconstructed third target Gaussian surface element has high reconstruction accuracy and can be used to render a realistic and lifelike reconstructed scene image.

[0098] According to an embodiment of the present invention, the third target Gaussian surface element may include a plurality of third target Gaussian surface elements.

[0099] In another embodiment of the present invention, the image processing method for scene reconstruction and rendering includes the above Figure 2 In addition to the operations S210~S240 shown, the following operations may also be included: vector quantizing the spherical harmonic coefficients of the least important third target Gaussian surface element; configuring a preset precision for the spherical harmonic coefficients of other third target Gaussian surface elements except the least important third target Gaussian surface element among multiple third target Gaussian surface elements; and compressing multiple third target Gaussian surface elements based on an octree compression scheme.

[0100] According to an embodiment of the present invention, the spherical harmonic coefficients of the least important third target Gaussian surface element may be vectorized, while the spherical harmonic coefficients of the remaining third target Gaussian surface elements and other attributes including surface element position and opacity are saved with float16 precision.

[0101] According to an embodiment of the present invention, the numerical precision of the least important Gaussian bins is reduced through quantization compression, thereby reducing memory overhead.

[0102] According to an embodiment of the present invention, after scene reconstruction and compression are completed, the scene representation can be stored and transmitted in the form of a compressed file. After decompression, the model can be used for real-time rendering and browsing to meet the needs of real-time rendering. The Gaussian surface element representation of the scene can also be converted into a grid representation through the Marching Cube algorithm based on the TSDF field, so that operations such as interactive editing and lighting reset can be further realized.

[0103] Figure 4 The overall structure of a distributed parallel training pipeline according to an embodiment of the present invention is schematically shown.

[0104] According to an embodiment of the present invention, the image processing method for scene reconstruction and rendering may include two key components, such as Figure 3 The Gaussian facet adaptive optimization algorithm shown in Figure 4 The distributed parallel training pipeline shown in Figure 1 is used to implement distributed training as well as model pruning and quantization compression, solving the problems of high training overhead and long training time of existing algorithms.

[0105] like Figure 4 As shown in Figure 2, the pipeline can include four stages: pre-training, sub-model and data partitioning, distributed fine-tuning, and quantization compression. Figure 3The proposed Gaussian facet adaptive optimization algorithm performs iterative optimization on all training views, i.e., multiple scene images to be reconstructed, so as to obtain a rough representation of the global scene, i.e., multiple first target Gaussian facets. In the sub-model and data partitioning stage, the coordinates of the first target Gaussian facets obtained by pre-training are mapped to a nonlinear space and divided into a series of sub-models according to the spatial relationship. Furthermore, for each sub-model, the algorithm will traverse all training views and assign the training views whose corresponding camera poses are within the sub-model space range or the sub-model has a significant contribution to the rendering result of the view to the training set of the sub-model. In the distributed fine-tuning stage, each sub-model and its training data will be assigned to the corresponding GPU, and the proposed Gaussian facet adaptive optimization algorithm will be used for iterative optimization and facet densification. At the same time, in the fine-tuning process, each sub-model will traverse its training set to calculate the importance of each Gaussian facet after completing the preset number of iterations on the training set, and delete all faces whose importance is less than a given percentage threshold, so as to prune and compress the model without significantly damaging the model performance. After all sub-models have completed the training operation, the sub-models will be integrated to obtain the final scene representation, namely the second target Gaussian surface. In the quantization compression stage, the algorithm traverses all training views to evaluate the importance of each second Gaussian surface in the model output by the fine-tuning stage, and vectorizes the spherical harmonic coefficients of the second Gaussian surface with the lowest importance, while the spherical harmonic coefficients of the remaining second Gaussian surfaces, as well as other attributes including surface position and opacity, are saved with float16 precision.

[0106] According to the embodiments of the present invention, based on the image processing method for scene reconstruction and rendering provided by the present invention, it is possible to use multi-view RGB images including aerial photography and street view as input to perform high-quality reconstruction and real-time rendering of large-scale scenes, and achieve significant results in terms of surface reconstruction accuracy and reconstruction overhead. Through visual verification and comparative experiments, it can prove its effectiveness and robustness in large-scale scene reconstruction tasks, and has broad application prospects.

[0107] Figure 5 The structure block diagram of the image processing device for scene reconstruction and rendering according to an embodiment of the present invention is schematically shown.

[0108] like Figure 5 As shown, the image processing device 500 for scene reconstruction and rendering includes an initialization module 510 , an optimization module 520 , a first determination module 530 and a second determination module 540 .

[0109] The initialization module 510 is used to initialize the sparse point cloud corresponding to each of the multiple scene images to be reconstructed to obtain multiple initial Gaussian facets, wherein the sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool.

[0110] The optimization module 520 is used to optimize each initial Gaussian surface based on the inverse depth map, the rendered scene image and the normal vector map obtained by rendering the initial Gaussian surface, so as to obtain a plurality of first target Gaussian surfaces.

[0111] The first determination module 530 is used to determine the second target Gaussian surface element from multiple first target Gaussian surface elements based on the importance of the first target Gaussian surface element, wherein the importance of the first target Gaussian surface element is determined based on multiple sub-models and a target scene image to be reconstructed that matches each sub-model, the multiple sub-models are obtained by gridding the multiple first target Gaussian surface elements, each sub-model is composed of the first target Gaussian surface elements in each grid, and the target scene image to be reconstructed that matches each sub-model is determined based on the pose of the scene image to be reconstructed and the rendering result of the sub-model.

[0112] The second determination module 540 is used to determine, based on the spatial range of the grid, a third target Gaussian surface element from the second target Gaussian surface element for rendering and reconstructing the scene image.

[0113] According to an embodiment of the present invention, the image processing apparatus 500 for scene reconstruction and rendering further includes a depth estimation module, a third determination module and a fourth determination module.

[0114] The depth estimation module is used to perform depth estimation on each scene image to be reconstructed using a monocular estimation base model to obtain an initial inverse depth map corresponding to each scene image to be reconstructed. The third determination module is used to determine a target inverse depth map that is consistent with the size of each scene image to be reconstructed based on the initial inverse depth map. The fourth determination module is used to determine a target normal vector map based on the target inverse depth map and the target scene image.

[0115] According to an embodiment of the present invention, the image processing device 500 for scene reconstruction and rendering further includes a mesh division module, a fifth determination module, a fine-tuning module and a sixth determination module.

[0116] The meshing module is used to mesh multiple first target Gaussian face elements based on spatial relationships to obtain multiple sub-models. The fifth determination module is used to determine the target scene image to be reconstructed that matches each sub-model from multiple scene images to be reconstructed based on the pose of the scene image to be reconstructed and the rendering results of the sub-model. The fine-tuning module is used to fine-tune the first target Gaussian face element in each sub-model based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed, and the normal vector map of the target scene image to be reconstructed. The sixth determination module is used to determine the importance of the first target Gaussian face element in each sub-model when it is determined that the number of fine-tuning meets the preset number of iterations.

[0117] According to an embodiment of the present invention, the image processing apparatus 500 for scene reconstruction and rendering further includes a vectorization module, a configuration module and a compression module.

[0118] The vectorization module is used to vectorize the spherical harmonic coefficients of the least important third target Gaussian surface element. The configuration module is used to configure the preset precision of the spherical harmonic coefficients of other third target Gaussian surface elements except the least important third target Gaussian surface element among the plurality of third target Gaussian surface elements. The compression module is used to compress the plurality of third target Gaussian surface elements based on the compression scheme of the octree.

[0119] According to an embodiment of the present invention, any multiple modules of the initialization module 510, the optimization module 520, the first determination module 530, and the second determination module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the initialization module 510, the optimization module 520, the first determination module 530, and the second determination module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), an on-chip method, a method on a substrate, a method on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the initialization module 510, the optimization module 520, the first determination module 530, and the second determination module 540 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0120] It should be noted that the image processing device part for scene reconstruction and rendering in the embodiment of the present invention corresponds to the image processing method part for scene reconstruction and rendering in the embodiment of the present invention. The description of the image processing device part for scene reconstruction and rendering specifically refers to the image processing method part for scene reconstruction and rendering, which will not be repeated here.

[0121] Figure 6 A block diagram of an electronic device suitable for implementing an image processing method for scene reconstruction and rendering according to an embodiment of the present invention is schematically shown.

[0122] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0123] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0124] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0125] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0126] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0127] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0128] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0129] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0130] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0131] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or couplings fall within the scope of the present invention.

[0134] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. An image processing method for scene reconstruction and rendering, characterized in that: The method comprises: Initializing a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed to obtain a plurality of initial Gaussian facets, wherein the sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool; Based on the inverse depth map, the rendered scene image, and the normal vector map obtained by rendering the initial Gaussian surface element, each of the initial Gaussian surface elements is optimized to obtain a plurality of first target Gaussian surface elements; Based on the importance of the first target Gaussian surface element, determining a second target Gaussian surface element from a plurality of the first target Gaussian surface elements, wherein the importance of the first target Gaussian surface element is determined based on a plurality of sub-models and a target scene image to be reconstructed matching each of the sub-models, the plurality of sub-models are obtained by meshing a plurality of the first target Gaussian surface elements, each of the sub-models is composed of the first target Gaussian surface elements in each mesh, and the target scene image to be reconstructed matching each of the sub-models is determined based on the pose of the scene image to be reconstructed and a rendering result of the sub-model; and Based on the spatial range of the grid, a third target Gaussian surface element for rendering and reconstructing the scene image is determined from the second target Gaussian surface elements.

2. The method according to claim 1, characterized in that The method further comprises: Using a monocular estimation base model to perform depth estimation on each of the scene images to be reconstructed, to obtain an initial inverse depth map corresponding to each of the scene images to be reconstructed; Based on the initial inverse depth map, determining a target inverse depth map having the same size as each of the scene images to be reconstructed; and A target normal vector map is determined according to the target inverse depth map and the target scene image.

3. The method according to claim 2, characterized in that The inverse depth map, the rendered scene image and the normal vector map obtained by rendering the initial Gaussian surface element, optimizing each of the initial Gaussian surface elements to obtain a plurality of first target Gaussian surface elements, comprises: For each of the initial Gaussian bins: Determining a depth estimation loss value based on the target inverse depth map and the inverse depth map; Determining an image loss value based on the target scene image and the rendered scene image; Determining a normal vector loss value based on the target normal vector map and the normal vector map; Obtaining a total loss value according to the sum of the depth estimation loss value, the image loss value and the normal vector loss value; According to the total loss value, iteratively optimize each of the initial Gaussian bins; When it is determined that the number of iterative optimizations meets the preset number of iterative optimizations, determining the gradient of the projection position of the imaging plane; and When it is determined that the average value of the gradient of the projection position of the imaging plane is greater than an average threshold and the major-minor axis ratio of the optimized initial Gaussian surface element is less than a preset major-minor axis ratio, the optimized initial Gaussian surface element is cloned or split to obtain a plurality of the first target Gaussian surface elements.

4. The method according to claim 1, characterized in that: The method further comprises: Based on the spatial relationship, meshing the plurality of the first target Gaussian surface elements to obtain a plurality of the sub-models; Based on the pose of the scene image to be reconstructed and the rendering result of the sub-model, determining the target scene image to be reconstructed that matches each of the sub-models from a plurality of the scene images to be reconstructed; Based on the inverse depth map of the target scene image to be reconstructed, the target scene image to be reconstructed, and the normal map of the target scene image to be reconstructed, fine-tuning the first target Gaussian surface element in each of the sub-models; and When it is determined that the number of fine-tuning times meets the preset number of iterations, the importance of the first target Gaussian surface element in each of the sub-models is determined.

5. The method according to claim 4, characterized in that The rendering result of the sub-model includes a first rendering result obtained when the opacity of the first target Gaussian surface element included in the sub-model is set to zero and a second rendering result obtained when the opacity is kept at the original value; The step of determining the target scene image to be reconstructed that matches each of the sub-models from a plurality of the scene images to be reconstructed based on the pose of the scene image to be reconstructed and the rendering result of the sub-model comprises: When it is determined that the pose of the scene image to be reconstructed is within the grid space of the sub-model and the loss values ​​of the first rendering result and the second rendering result exceed a preset loss value, the scene image to be reconstructed is determined as the target scene image to be reconstructed that matches each of the sub-models.

6. The method according to claim 4, characterized in that When it is determined that the number of fine-tuning times satisfies the preset number of iterations, determining the importance of the first target Gaussian surface element in each of the sub-models includes: In a case where it is determined that the number of fine-tuning times satisfies the preset number of iterations, determining pixels covered by the projection of each of the first target Gaussian surface elements on the target scene image to be reconstructed; determining the opacity of each of the first target Gaussian bins; and The importance of the first target Gaussian surface element in each of the sub-models is determined based on the pixel, the opacity, and the first target Gaussian surface element in each of the sub-models.

7. The method according to claim 1, characterized in that The step of determining, based on the spatial range of the grid, a third target Gaussian surface element for rendering and reconstructing the scene image from the second target Gaussian surface element comprises: According to the spatial range of the grid, the second target Gaussian surface element located at the center of the spatial range of the grid is determined as the third target Gaussian surface element.

8. The method according to claim 1, characterized in that The third target Gaussian surface element includes a plurality of; The method further comprises: Vector quantizing the spherical harmonic coefficients of the third target Gaussian bin with the lowest importance; configuring preset precisions for the spherical harmonic coefficients of the other third target Gaussian bins except the least important third target Gaussian bin among the plurality of third target Gaussian bins; and The plurality of third target Gaussian bins are compressed based on an octree compression scheme.

9. An image processing device for scene reconstruction and rendering, characterized in that: The device comprises: An initialization module, used to initialize a sparse point cloud corresponding to each of a plurality of scene images to be reconstructed, to obtain a plurality of initial Gaussian facets, wherein the sparse point cloud is obtained by processing the scene images to be reconstructed using a preset reconstruction tool; An optimization module, configured to optimize each of the initial Gaussian surfaces based on the inverse depth map, the rendered scene image, and the normal vector map obtained by rendering the initial Gaussian surfaces, to obtain a plurality of first target Gaussian surfaces; a first determination module, configured to determine a second target Gaussian surface element from a plurality of the first target Gaussian surface elements based on the importance of the first target Gaussian surface element, wherein the importance of the first target Gaussian surface element is determined based on a plurality of sub-models and a target scene image to be reconstructed matching each of the sub-models, the plurality of sub-models are obtained by meshing a plurality of the first target Gaussian surface elements, each of the sub-models is composed of the first target Gaussian surface elements in each mesh, and the target scene image to be reconstructed matching each of the sub-models is determined based on the pose of the scene image to be reconstructed and a rendering result of the sub-model; and The second determination module is used to determine, based on the spatial range of the grid, a third target Gaussian surface element from the second target Gaussian surface element for rendering and reconstructing the scene image.

10. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images