Three-dimensional scene reconstruction method, device, equipment, medium and program product

Abstract

The disclosed embodiments provide a method, device, equipment, medium and program product for reconstructing a three-dimensional scene. The method includes: obtaining a scene image captured of a three-dimensional scene; determining sparse point cloud data and camera parameter information corresponding to the scene image based on the scene image; initializing a model based on the sparse point cloud data to obtain a current three-dimensional data model; rendering based on the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map and a current normal map from the perspective of the scene image, and determining a pseudo-normal map from the perspective of the scene image based on the current depth map; training the current three-dimensional data model based on the current color map, the current normal map, the pseudo-normal map and the actual color map to obtain a target three-dimensional data model after training. Through the technical solution of the disclosed embodiments, higher-quality automated reconstruction of three-dimensional scenes can be achieved, which reduces reconstruction costs and improves the fineness and rendering quality of scene models.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to computer technology, and more particularly to a three-dimensional scene reconstruction method, apparatus, device, medium and program product. Background Art

[0002] With the rapid development of computer technology, it is usually necessary to use the collected 3D scene data to reconstruct the scene and obtain a 3D scene model. The 3D scene model reconstructed by the existing 3D reconstruction method is usually a 3D mesh model. However, the quality of this 3D mesh model is relatively rough, and professionals are required to further repair the 3D network model to meet the 3D reconstruction requirements. Summary of the invention

[0003] The present disclosure provides a 3D scene reconstruction method, apparatus, device, medium and program product to achieve higher quality automated reconstruction of 3D scenes, reduce reconstruction costs, and improve the precision and rendering quality of scene models.

[0004] In a first aspect, an embodiment of the present disclosure provides a three-dimensional scene reconstruction method, comprising:

[0005] Acquire three-dimensional scene data collected for the three-dimensional scene to be reconstructed, wherein the three-dimensional scene data includes a scene image of the three-dimensional scene;

[0006] Determine, according to the scene image, sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image;

[0007] Initialize the model according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene;

[0008] Rendering is performed according to the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image, and a pseudo-normal map under the perspective of the scene image is determined according to the current depth map;

[0009] The current three-dimensional data model is trained according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain a target three-dimensional data model after the training is completed.

[0010] In a second aspect, the present disclosure also provides a three-dimensional scene reconstruction device, including:

[0011] A three-dimensional scene data acquisition module, used to acquire three-dimensional scene data collected for the three-dimensional scene to be reconstructed, wherein the three-dimensional scene data includes a scene image of the three-dimensional scene;

[0012] A sparse point cloud data determination module, used to determine the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image according to the scene image;

[0013] A model initialization module, used to initialize the model according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene;

[0014] A model rendering module, used to render according to the camera parameter information and the current three-dimensional data model, obtain a current color map, a current depth map and a current normal map under the perspective of the scene image, and determine a pseudo-normal map under the perspective of the scene image according to the current depth map;

[0015] The model training module is used to train the current three-dimensional data model according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain the target three-dimensional data model after the training is completed.

[0016] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] a storage device for storing one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional scene reconstruction method as described in any of the embodiments of the present disclosure.

[0020] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the three-dimensional scene reconstruction method as described in any one of the embodiments of the present disclosure.

[0021] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements the three-dimensional scene reconstruction method as described in any one of the embodiments of the present disclosure.

[0022] The disclosed embodiment determines the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image according to the scene image collected for the three-dimensional scene to be reconstructed; performs model initialization according to the sparse point cloud data to obtain the initialized current three-dimensional data model; performs rendering according to the camera parameter information and the current three-dimensional data model to obtain the current color map, current depth map and current normal map under the perspective of the scene image, and determines the pseudo-normal map under the perspective of the scene image according to the current depth map; performs self-supervisory training on the current three-dimensional data model according to the current color map, current normal map, pseudo-normal map and the actual color map of the scene image, so that a higher quality target three-dimensional data model can be fully automatically obtained, reducing the reconstruction cost. Moreover, the target three-dimensional data model has a stronger scene expression capability than the traditional three-dimensional mesh model, thereby improving the fineness and rendering quality of the scene model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0024] Figure 1 is a flowchart of a three-dimensional scene reconstruction method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart of another three-dimensional scene reconstruction method provided by an embodiment of the present disclosure;

[0026] Figure 3 is a flowchart of another three-dimensional scene reconstruction method provided by an embodiment of the present disclosure;

[0027] Figure 4 is a structural schematic diagram of a three-dimensional scene reconstruction device provided by an embodiment of the present disclosure;

[0028] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0036] Figure 1 The present invention provides a flow chart of a 3D scene reconstruction method provided by an embodiment of the present invention. The present invention is applicable to the reconstruction of a real 3D scene, and the reconstructed model can be directly applied to virtual application scenes such as a virtual live studio. The method can be executed by a 3D scene reconstruction device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC, or a server.

[0037] like Figure 1 As shown, the three-dimensional scene reconstruction method specifically includes the following steps:

[0038] S110 , acquiring three-dimensional scene data collected for a three-dimensional scene to be reconstructed, where the three-dimensional scene data includes a scene image of the three-dimensional scene.

[0039] Among them, the three-dimensional scene can be any three-dimensional physical scene in reality that needs to be reconstructed. The three-dimensional scene can be a closed scene or a non-closed scene. For example, the three-dimensional scene can be a tourist attraction, etc. The three-dimensional scene data refers to the scene data collected by the acquisition device for the three-dimensional scene. The three-dimensional scene data may include but is not limited to the scene image of the three-dimensional scene. The scene image may be a real image of the three-dimensional scene collected by the image acquisition device. The image acquisition device may include but is not limited to: at least one of a SLR, a mobile phone, a drone, a panoramic camera, a laser device and a depth camera. The number of scene images is multiple, and all scene images contain all scene contents of the three-dimensional scene, so that a complete three-dimensional scene model can be reconstructed based on the scene image.

[0040] Specifically, at least one acquisition device is used to acquire the three-dimensional scene to be reconstructed according to the scene acquisition requirements corresponding to each acquisition device, and the acquired three-dimensional scene data is obtained. For example, at least one image acquisition device can be used to acquire images of the three-dimensional scene to be reconstructed according to the scene acquisition requirements corresponding to each image acquisition device, and all acquired scene images are obtained. Among them, the scene acquisition requirements corresponding to each acquisition device may include information such as the placement position and acquisition path of the acquisition device, so that the scene images that meet the scene reconstruction requirements can be quickly acquired by acquiring according to the scene acquisition requirements. By pre-configuring the simple and standard scene acquisition requirements corresponding to each acquisition device, non-professionals can also quickly acquire high-quality three-dimensional scene data, thereby reducing the acquisition technology threshold and improving the three-dimensional reconstruction effect.

[0041] S120. Determine, according to the scene image, sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image.

[0042] Among them, sparse point cloud data may refer to data of some discrete feature points in a three-dimensional scene. Feature points refer to three-dimensional space points with obvious features in a three-dimensional scene, such as corners and edge points of buildings. Sparse point cloud data can be used to characterize the sparse geometric structure of a three-dimensional scene. Camera parameter information may refer to the current addition information when capturing a scene image. For example, camera parameter information may include camera pose information and phase intrinsic parameter information. Different scene images may correspond to different camera parameter information.

[0043] Specifically, by utilizing the Structure from Motion (SfM) algorithm and the Bundle Adjustment (BA) algorithm, sparse point cloud reconstruction and camera pose estimation can be performed on all collected scene images to obtain sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to each scene image.

[0044] S130 . Initialize the model according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene.

[0045] The current three-dimensional data model may refer to a three-dimensional data model at the current moment. The current three-dimensional data model may change with the dynamic change of the model parameter value. Different model parameter values ​​correspond to different three-dimensional data models. The three-dimensional data model may be a collection of spatial data for describing a three-dimensional scene. The three-dimensional data model may be a model for reconstructing a three-dimensional scene into a light field. The light field is a function for describing the amount of light passing through each point and each direction in the three-dimensional space.

[0046] Exemplarily, the current three-dimensional data model may include a neural radiance field (NeRF) or a three-dimensional Gaussian. Among them, the neural radiation field uses a neural network to store radiation information from any spatial position to any direction. Radiation information can also be called energy distribution, which can refer to information such as color, brightness and shadow. The neural radiation field is to reconstruct the three-dimensional scene into a radiation field represented by a neural network. The three-dimensional Gaussian is to reconstruct the three-dimensional scene into a Gaussian represented by Gaussian distribution. The neural radiation field and the three-dimensional Gaussian have stronger scene expression capabilities and view rendering capabilities than the traditional three-dimensional mesh model, thereby improving the fineness and rendering quality of the scene model. Among them, the model parameters in the neural radiation field may include: volume density values, color values, depth values ​​and normal values ​​corresponding to the voxels in the three-dimensional scene. The model parameters in the three-dimensional Gaussian may include: position information, spherical harmonic function coefficients, transparency, rotation values, scale values ​​and normal values ​​corresponding to the voxels in the three-dimensional scene.

[0047] Specifically, the initial value of each model parameter in the three-dimensional data model can be determined based on the sparse point cloud data, and the model parameters can be assigned to achieve model initialization, thereby obtaining the initialized current three-dimensional data model. For example, the volume density value, color value, depth value and normal value corresponding to each voxel in the neural radiation field are initialized based on the sparse point cloud data to obtain the initialized neural radiation field. Alternatively, the position information, spherical harmonic function coefficients, transparency, rotation value, scale value and normal value corresponding to each voxel in the three-dimensional Gaussian body are initialized based on the sparse point cloud data to obtain the initialized three-dimensional Gaussian body.

[0048] S140, rendering is performed according to the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image, and a pseudo-normal map under the perspective of the scene image is determined according to the current depth map.

[0049] Among them, the scene image perspective refers to the camera perspective when shooting the scene image. The current color map may include the current color value corresponding to each pixel. The current depth map may include the current depth value corresponding to each pixel. The current normal map may include the current normal value corresponding to each pixel. The current color map, the current depth map, and the current normal map are all the same size as the scene image, and their pixels are also one-to-one corresponding. The current color map, the current depth value, and the current normal value corresponding to each pixel are all obtained by rendering the current three-dimensional data model. The pseudo-normal map may include the pseudo-normal value corresponding to each pixel. The pseudo-normal value can be regarded as the actual normal value corresponding to the pixel, that is, the true value of the normal.

[0050] Specifically, for each scene image, the current three-dimensional data model is rendered based on the camera parameter information corresponding to the scene image, so that the current three-dimensional data model is rendered into a two-dimensional current color map, current depth map and current normal map with the same perspective as the scene image, and local gradient calculation is performed based on the current depth map to obtain a pseudo-normal map under the perspective of the scene image.

[0051] Exemplarily, "rendering according to camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map, and a current normal map from the perspective of the scene image" in step S140 may include: in response to the current three-dimensional data model being a neural radiation field, rendering is performed according to the camera parameter information and the current three-dimensional data model through a neural rendering method to obtain the current color map, current depth map, and current normal map from the perspective of the scene image; in response to the current three-dimensional data model being a three-dimensional Gaussian body, rendering is performed according to the camera parameter information and the current three-dimensional data model through a Gaussian rasterization method to obtain the current color map, current depth map, and current normal map from the perspective of the scene image.

[0052] Specifically, for each scene image, when the current three-dimensional data model is a neural radiation field, it can be rendered according to the camera parameter information corresponding to the scene image and the current three-dimensional data model through a neural rendering method to obtain the current color map, current depth map and current normal map from the perspective of the scene image. Among them, the neural rendering method uses camera rays to sample the current neural radiation field, and estimates the rendering information for each sampling point, that is, the color value, depth value and normal value, so as to realize the volume rendering of the neural radiation field, and obtain the rendered image from the perspective of the scene image, that is, the current color map, current depth map and current normal map. When the current three-dimensional data model is a three-dimensional Gaussian volume, it is rendered according to the camera parameter information corresponding to the scene image and the current three-dimensional data model through a Gaussian rasterization method to obtain the current color map, current depth map and current normal map from the perspective of the scene image. Among them, the Gaussian rasterization method is to project the ellipse onto the camera plane, and then rasterize it to the pixel plane, and finally estimate the rendering information for each pixel, that is, the color value, depth value and normal value, so as to rasterize the three-dimensional Gaussian volume into a rendered image and realize the three-dimensional Gaussian splashing process.

[0053] S150, training the current three-dimensional data model according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain a target three-dimensional data model after the training is completed.

[0054] Specifically, since the actual color value of the scene image can be directly obtained, and the pseudo-normal map is obtained by calculating the current rendering image, the actual color value and the pseudo-normal map are used as training labels to perform self-supervisory training on the current three-dimensional data model without manual annotation, and constraints on depth and normal are added during the model training process, thereby improving the training speed and quality of the three-dimensional data model. By dynamically adjusting the model parameters in the current three-dimensional data model, the current three-dimensional data model is automatically iteratively trained, and the current three-dimensional data model after the training is determined as the target three-dimensional data model corresponding to the three-dimensional scene. The target three-dimensional data model refers to the three-dimensional data model finally reconstructed by the three-dimensional scene, such as a neural radiation field or a three-dimensional Gaussian body. Since the target three-dimensional data model contains richer model information than the traditional three-dimensional mesh model, the target three-dimensional data model has a stronger model expression ability, such as the ability to express geometric attributes such as scene surface normals and depth, so that the target three-dimensional data model can achieve ultra-fine and realistic reconstruction effects, improving the quality of geometric rendering.

[0055] The technical solution of the disclosed embodiment is to determine the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image according to the scene image collected for the three-dimensional scene to be reconstructed; initialize the model according to the sparse point cloud data to obtain the initialized current three-dimensional data model; render according to the camera parameter information and the current three-dimensional data model to obtain the current color map, current depth map and current normal map under the perspective of the scene image, and determine the pseudo-normal map under the perspective of the scene image according to the current depth map; perform self-supervised training on the current three-dimensional data model according to the current color map, current normal map, pseudo-normal map and the actual color map of the scene image, so that a higher quality target three-dimensional data model can be obtained fully automatically, reducing the reconstruction cost. Moreover, the target three-dimensional data model has a stronger scene expression capability than the traditional three-dimensional mesh model, thereby improving the precision and rendering quality of the scene model.

[0056] Based on the above technical solution, step S120 may include: performing data processing on the scene image to obtain scene image data in a preset image format; preprocessing the scene image data in the preset image format to obtain preprocessed scene image data; sparsely reconstructing the preprocessed scene image data based on a motion recovery structure algorithm and a bundle adjustment algorithm to obtain sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image.

[0057] Among them, the preset image format can be a pre-set universal image format. Specifically, since the scene images collected by different image acquisition devices have different image formats, it is necessary to unify all image formats to obtain scene image data under a universal preset image format. For example, different imaging models are used for parameterization based on different image formats, such as fisheye imaging model, panoramic imaging model, pinhole imaging model or multi-camera group model, etc., so that scene images of different image formats are processed into a unified camera model data structure, and then the scene image data under the preset image format is obtained. Since the collected scene images are processed in a unified format, different types of acquisition devices can be used for image acquisition, which reduces the acquisition cost and improves the adaptability of three-dimensional reconstruction. The scene image data under the preset image format can be pre-processed by denoising, super-resolution, feature extraction and matching, etc. through deep learning, so that the success rate and accuracy of three-dimensional reconstruction can be improved, and feature extraction and matching between different data formats can also be supported. By using the Structure from Motion (SfM) algorithm and the Bundling Adjustment (BA) algorithm to reconstruct sparse point clouds and estimate camera poses for all preprocessed scene images, the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to each scene image can be obtained more accurately, thereby further improving the three-dimensional reconstruction effect.

[0058] Exemplarily, the three-dimensional scene data also includes: scene acquisition data in non-image format. The scene acquisition data in non-image format may refer to scene data acquired by non-image acquisition equipment, such as positioning data acquired by the Global Positioning System (GPS).

[0059] Specifically, for the reconstructed 3D scene, in addition to collecting scene images, non-image format scene acquisition data can also be collected. In the case where non-image format scene acquisition data exists in the collected 3D scene data, the non-image format scene acquisition data can be used as additional acquisition data and added to the unified camera model data structure in the form of point clouds, depth maps, control points, etc., so as to perform additional processing during sparse reconstruction.

[0060] Exemplarily, based on the motion recovery structure algorithm and the bundle adjustment algorithm, sparse reconstruction is performed on the preprocessed scene image data to obtain sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image, which can include: adding the scene acquisition data in a non-image format as an algorithm constraint condition to the motion recovery structure algorithm and the bundle adjustment algorithm, sparse reconstruction is performed on the preprocessed scene image data, and sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image is obtained.

[0061] Specifically, when there is scene acquisition data in non-image format in the collected three-dimensional scene data, the scene acquisition data in non-image format is introduced as an additional algorithm constraint condition into the motion recovery structure algorithm and the bundle adjustment algorithm for optimization, so that the sparse point cloud reconstruction and camera pose estimation can be performed more accurately, further improving the scene reconstruction accuracy.

[0062] On the basis of the above technical solutions, when the three-dimensional scene is a non-closed scene, it is necessary to convert the non-closed scene into a closed scene so as to be suitable for the reconstruction of the three-dimensional data model. For example, for the neural radiation field, since it is necessary to emit a ray from the camera and sample on the ray during the rendering process, it is necessary to define the starting point and end point of the sampling. For large-scale outdoor scenes, there is no predefined boundary (such as the sky), so the end point cannot be defined. Therefore, for non-closed scenes (i.e., unbounded scenes), a mapping can be defined to convert the unbounded non-closed scene into a bounded scene to be suitable for the reconstruction of a larger scene range. For example, a mapping is defined to convert the unbounded space into a sphere with a radius of a preset value (such as 2). The samples within the unit sphere are not modified, while the samples outside the unit sphere are shrunk to fit the sphere with a radius of a preset value, so that the calculation of all sampling points is to map a point in space to the inside of a sphere with a radius of a preset value, and then the neural radiation field can be used to reconstruct the three-dimensional data model in the non-closed scene. For a three-dimensional Gaussian body, a spherical point cloud can be added to the outside of the reconstructed sparse point cloud to perform three-dimensional reconstruction on the inside of the sphere, so that a three-dimensional data model in a non-closed scene can also be reconstructed using the three-dimensional Gaussian body.

[0063] Figure 2 A flow chart of another three-dimensional scene reconstruction method provided by the embodiment of the present disclosure. Based on the above disclosed embodiments, the embodiment of the present disclosure optimizes the step of "determining the pseudo-normal map under the perspective of the scene image according to the current depth map" and the step of "training the current three-dimensional data model according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain the target three-dimensional data model after the training". The optimization of these two steps can exist separately or simultaneously, and the embodiment of the present disclosure does not limit this. The explanations of the terms that are the same or corresponding to the above disclosed embodiments are not repeated here.

[0064] like Figure 2 As shown, the three-dimensional scene reconstruction method specifically includes the following steps:

[0065] S210 , acquiring three-dimensional scene data collected for a three-dimensional scene to be reconstructed, where the three-dimensional scene data includes a scene image of the three-dimensional scene.

[0066] S220 . Determine sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image according to the scene image.

[0067] S230 , initializing a model according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene.

[0068] S240, rendering is performed according to the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image.

[0069] S250: Determine spatial position information corresponding to each scene pixel in the scene image according to the current depth map.

[0070] The spatial position information corresponding to each scene pixel can be represented by the three-dimensional coordinates xyz in the world coordinate system. Specifically, according to the current depth value corresponding to each pixel in the current depth map, the two-dimensional coordinates of each scene pixel in the view can be converted into the three-dimensional world space to obtain the spatial position information xyz corresponding to each scene pixel.

[0071] S260: Determine horizontal gradient information and vertical gradient information corresponding to each scene pixel according to the spatial position information.

[0072] Among them, the horizontal gradient information corresponding to the scene pixel refers to the gradient information in the horizontal direction of the scene pixel. The vertical gradient information refers to the gradient information in the vertical direction of the scene pixel. Specifically, for each scene pixel, the horizontal gradient information and vertical gradient information corresponding to the scene pixel can be determined based on the spatial position information corresponding to the surrounding scene pixels of the scene pixel. Among them, the surrounding scene pixels can refer to the scene pixels in a neighborhood range with a preset size (such as 3×3) centered on the scene pixel. The horizontal gradient information corresponding to the scene pixel is determined using the spatial position information corresponding to the surrounding scene pixels in the horizontal direction. The vertical gradient information corresponding to the scene pixel is determined using the spatial position information corresponding to the surrounding scene pixels in the vertical direction. For example, there are 8 surrounding scene pixels in the 3×3 neighborhood range, and the surrounding scene pixels in the horizontal direction include: the surrounding scene pixel xyz located in the 1st row and the 1st column 00 , the surrounding scene pixel xyz located in row 1 and column 3 02 , the surrounding scene pixel xyz located in row 2 and column 1 10 , the surrounding scene pixel xyz located in the 2nd row and 3rd column 12 , the surrounding scene pixel xyz located in row 3, column 1 20 and the surrounding scene pixel xyz at row 3, column 3 22 The surrounding scene pixels in the vertical direction include: the surrounding scene pixel xyz located in the 1st row and the 1st column 00 , the surrounding scene pixel xyz located in row 1 and column 2 01 , the surrounding scene pixel xyz located in row 1 and column 3 02 , the surrounding scene pixel xyz located in row 3, column 1 20, the surrounding scene pixel xyz located in the 3rd row and 2nd column 21 and the surrounding scene pixel xyz at row 3, column 3 22 For example, the scene pixel xyz 11 The corresponding horizontal gradient information is: -0.125×xyz 00 +0.125×xyz 02 —0.25×xyz 10 +0.25×xyz 12 —0.125×xyz 20 +0.125×xyz 22 The scene pixel xyz 11 The corresponding vertical gradient information is: -0.125×xyz 002 —0.25×xyz 01 —0.125×xyz 02 +0.125×xyz 20 +0.25×xyz 21 +0.125×xyz 22 It should be noted that the horizontal gradient information and the vertical gradient information corresponding to each scene pixel can be represented by a three-dimensional column vector.

[0073] S270 , determining a normal corresponding to each scene pixel according to the horizontal gradient information and the vertical gradient information, and normalizing the normal to determine a pseudo normal map under the viewing angle of the scene image.

[0074] Specifically, for each scene pixel, the horizontal gradient information and the vertical gradient information corresponding to the scene pixel can be vector cross-multiplied to obtain the normal corresponding to the scene pixel. The normal can also be represented by a three-dimensional column vector. Based on the maximum modulus of all normals, the normal corresponding to each scene pixel is normalized, and the normalized normal is used as a pseudo-normal value, thereby generating a pseudo-normal map under the perspective of the scene image based on the pseudo-normal values ​​corresponding to all scene pixels. By using the local gradient method, the pseudo-normal map under the perspective of each scene image can be accurately obtained.

[0075] S280, determining a current color loss value according to the current color map and the actual color map of the scene image, and determining a current normal loss value according to the current normal map and the pseudo-normal map.

[0076] The current color loss value is used to characterize the difference between the current color map and the actual color map, and the current normal loss value is used to characterize the difference between the current normal map and the pseudo normal map.

[0077] Specifically, based on the current color map and the actual color map corresponding to each scene image, the absolute difference and structural similarity at the pixel level between the current color map and the actual color map can be determined, and the absolute difference and structural pixel degree corresponding to each scene image are weighted summed, and the summed result is used as the current color loss value. Based on the current normal map and pseudo-normal map corresponding to each scene image, the mean square error between the current normal map and the pseudo-normal map can be determined, and the mean square error is used as the current normal loss value.

[0078] S290. Determine the current model loss value based on the current color loss value and the current normal loss value, and adjust the model parameters in the current three-dimensional data model based on the current model loss value until the training ends when the preset convergence condition is met, and obtain the target three-dimensional data model after the training ends.

[0079] Specifically, the current color loss value and the current normal loss value corresponding to each scene image can be added, and the obtained addition result is the current model loss value. The current model loss value is back-propagated to the current three-dimensional data model, the model parameters in the current three-dimensional data model are adjusted, and the rendering operation and parameter adjustment operation are performed based on the adjusted current three-dimensional data model reconstruction, that is, the operation of steps S240-S290, until the preset convergence condition is met, such as the number of iterations is equal to the preset number of times, or the model loss value changes tend to be stable, the model training is determined to be completed, and the current three-dimensional data model after the training is completed is determined as the target three-dimensional data model finally reconstructed, thereby achieving fully automated high-quality reconstruction of the three-dimensional scene. For example, when the current three-dimensional data model is a neural radiation field, iterative training is performed by adjusting the volume density value, color value, depth value and normal value corresponding to each voxel in the neural radiation field. When the current three-dimensional data model is a three-dimensional Gaussian body, iterative training is performed by adjusting the position information, spherical harmonic function coefficients, transparency, rotation value, scale value and normal value corresponding to each voxel in the three-dimensional Gaussian body.

[0080] Exemplarily, "determining the current model loss value based on the current color loss value and the current normal loss value" in step S290 may include: determining the current smoothness loss value based on the current normal map, the current depth map and the actual color map of the scene image; determining the current model loss value based on the current color loss value, the current normal loss value and the current smoothness loss value.

[0081] Specifically, based on the preset smoothing function, the normal smoothing loss value is determined according to the current normal map and the actual color map of the scene image, the depth smoothing loss value is determined according to the current depth map and the actual color map of the scene image, and the normal smoothing loss value and the depth smoothing loss value are weighted summed to obtain the current smoothing loss value. The current color loss value, the current normal loss value and the current smoothing loss value corresponding to each scene image are added, and the addition result is the current model loss value. By considering the smoothing loss, the model training effect can be further improved, thereby further improving the quality of 3D reconstruction.

[0082] The technical solution of the disclosed embodiment determines the spatial position information corresponding to each scene pixel in the scene image according to the current depth map, and determines the horizontal gradient information and vertical gradient information corresponding to each scene pixel according to the spatial position information, determines the normal corresponding to each scene pixel according to the horizontal gradient information and the vertical gradient information, and normalizes the normal, so as to accurately determine the pseudo-normal map under the perspective of the scene image. The current color loss value is determined according to the current color map and the actual color map of the scene image, and the current normal loss value is determined according to the current normal map and the pseudo-normal map; the current model loss value is determined based on the current color loss value and the current normal loss value, and the model is trained based on the current model loss value, so that the trained target three-dimensional data model has a stronger scene expression ability, and the refinement and rendering quality of the scene model are further improved.

[0083] Figure 3 This is a flow chart of another 3D scene reconstruction method provided by the embodiment of the present disclosure. Based on the above disclosed embodiments, the present disclosed embodiment directly uses the target 3D data model to perform lighting and shadowing. The explanations of the terms that are the same or corresponding to the above disclosed embodiments are not repeated here.

[0084] like Figure 3 As shown, the three-dimensional scene reconstruction method specifically includes the following steps:

[0085] S310: Acquire three-dimensional scene data collected for a three-dimensional scene to be reconstructed, where the three-dimensional scene data includes a scene image of the three-dimensional scene.

[0086] S320. Determine, according to the scene image, sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image.

[0087] S330 , initializing a model according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene.

[0088] S340, rendering is performed according to the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image, and a pseudo-normal map under the perspective of the scene image is determined according to the current depth map.

[0089] S350, training the current three-dimensional data model according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain a target three-dimensional data model after the training is completed.

[0090] S360: Perform lighting and shadow adding processing according to the target three-dimensional data model, the current position information and light source type information of the virtual light source, and the current parameter information of the virtual camera to obtain a virtual scene image with lighting and shadow effects.

[0091] Among them, the virtual light source may be a virtual light-emitting device placed in a three-dimensional scene. The current position information of the virtual light source may be pre-set based on business needs. The light source type information may include a parallel light source or a point light source. The virtual camera may be a virtual shooting device placed in a three-dimensional scene. The current parameter information of the virtual camera may include the current position information and current internal parameter information of the virtual camera. The virtual scene image may refer to a two-dimensional virtual image at a certain viewing angle in a three-dimensional scene with lighting and shadows.

[0092] Specifically, the user can set the current position information of the virtual light source and the current parameter information of the virtual camera based on the lighting requirements and the perspective imaging requirements. Since the reconstructed target 3D data model has a stronger scene expression capability, the target 3D data model can be directly rendered by adding lighting and shadows based on the current position information and light source type information of the virtual light source and the current parameter information of the virtual camera, and high-quality virtual scene images with lighting and shadow effects can be quickly obtained without human intervention, thereby improving the rendering quality and rendering speed of the virtual scene image.

[0093] For example, in a virtual live broadcast room, the reconstructed target three-dimensional data model can be subjected to rendering processing such as lighting and projection to obtain a virtual scene image with lighting and shadow effects, and the virtual scene image can be directly displayed as a live broadcast screen in the virtual live broadcast room, or the virtual scene image can be used as a live broadcast background image and synthesized with the anchor screen to obtain a live broadcast screen of the anchor in the virtual three-dimensional scene, thereby achieving a virtual live broadcast effect.

[0094] Exemplarily, step S360 may include: rendering according to current position information and light source type information of the virtual light source and the target three-dimensional data model to obtain a first depth map from the perspective of the virtual light source; rendering according to current parameter information of the virtual camera and the target three-dimensional data model to obtain a target color map, a second depth map, a target normal map and a target opacity map from the perspective of the virtual camera; determining a target shadow map based on the first depth map and the second depth map; determining a virtual scene image with lighting and shadow effects based on the target color map, the target normal map, the target opacity map and the target shadow map.

[0095] Specifically, the target projection mode can be determined according to the light source type information of the virtual light source. For example, if the light source type information is a parallel light source, the target projection mode is determined to be an orthogonal projection; if the light source type information is a point light source, the target projection mode is determined to be a perspective projection. According to the current position information of the virtual light source, the target three-dimensional data model is rendered to obtain a first depth map with the target projection mode from the perspective of the virtual light source, such as a first depth map under orthogonal projection or a first depth map under perspective projection. In the case where the current three-dimensional data model is a neural radiation field, a neural rendering method can be used to render according to the current position information of the virtual light source and the target three-dimensional data model to obtain a first depth map with the target projection mode from the perspective of the virtual light source, and a neural rendering method can be used to render according to the current parameter information of the virtual camera and the target three-dimensional data model to obtain a target color map, a second depth map, a target normal map, and a target opacity map from the perspective of the virtual camera. Alternatively, when the current three-dimensional data model is a three-dimensional Gaussian body, a first depth map with a target projection mode under the perspective of the virtual light source is rendered according to the current position information of the virtual light source and the target three-dimensional data model through Gaussian rasterization, and a target color map, a second depth map, a target normal map and a target opacity map under the perspective of the virtual camera are rendered according to the current parameter information of the virtual camera and the target three-dimensional data model through Gaussian rasterization. The second depth map under the perspective of the virtual camera is mapped to the light source position space, and compared with the depth value in the first depth map under the perspective of the virtual light source to generate a target shadow map. According to the target color map, target normal map, target opacity map and target shadow map under the perspective of the virtual camera, calculation is performed to determine the final color map after lighting and adding shadows under the perspective of the virtual camera, that is, a virtual scene image with lighting and shadow effects, thereby improving the view rendering quality and rendering speed.

[0096] The technical solution of the disclosed embodiment can quickly obtain a virtual scene image with lighting and shadow effects by performing lighting and shadow adding processing according to the target three-dimensional data model, the current position information and light source type information of the virtual light source, and the current parameter information of the virtual camera, without the need for human intervention, thereby improving the rendering quality and rendering speed of the virtual scene image.

[0097] Figure 4 A schematic diagram of the structure of a three-dimensional scene reconstruction device provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the device specifically includes: a three-dimensional scene data acquisition module 410, a sparse point cloud data determination module 420, a model initialization module 430, a model rendering module 440 and a model training module 450.

[0098] Among them, the three-dimensional scene data acquisition module 410 is used to obtain the three-dimensional scene data collected for the three-dimensional scene to be reconstructed, and the three-dimensional scene data includes the scene image of the three-dimensional scene; the sparse point cloud data determination module 420 is used to determine the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image according to the scene image; the model initialization module 430 is used to initialize the model according to the sparse point cloud data to obtain the current three-dimensional data model corresponding to the three-dimensional scene; the model rendering module 440 is used to render according to the camera parameter information and the current three-dimensional data model to obtain the current color map, current depth map and current normal map under the perspective of the scene image, and determine the pseudo-normal map under the perspective of the scene image according to the current depth map; the model training module 450 is used to train the current three-dimensional data model according to the current color map, current normal map, pseudo-normal map and the actual color map of the scene image to obtain the target three-dimensional data model after the training is completed.

[0099] The technical solution provided by the disclosed embodiment determines the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image according to the scene image collected for the three-dimensional scene to be reconstructed; performs model initialization according to the sparse point cloud data to obtain the initialized current three-dimensional data model; performs rendering according to the camera parameter information and the current three-dimensional data model to obtain the current color map, current depth map and current normal map under the perspective of the scene image, and determines the pseudo-normal map under the perspective of the scene image according to the current depth map; performs self-supervisory training on the current three-dimensional data model according to the current color map, current normal map, pseudo-normal map and the actual color map of the scene image, so that a higher quality target three-dimensional data model can be obtained fully automatically, reducing the reconstruction cost. Moreover, the target three-dimensional data model has a stronger scene expression capability than the traditional three-dimensional mesh model, thereby improving the fineness and rendering quality of the scene model.

[0100] Based on the above technical solution, the sparse point cloud data determination module 420 includes:

[0101] A data processing unit, used to perform data processing on the scene image to obtain scene image data in a preset image format;

[0102] A preprocessing unit, used to preprocess the scene image data in a preset image format to obtain preprocessed scene image data;

[0103] The sparse reconstruction unit is used to perform sparse reconstruction on the preprocessed scene image data based on the motion recovery structure algorithm and the bundle adjustment algorithm to obtain the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image.

[0104] On the basis of the above technical solutions, the three-dimensional scene data further includes: scene acquisition data in non-image format;

[0105] The sparse reconstruction unit is specifically used to: add the scene acquisition data in non-image format as algorithm constraints to the motion recovery structure algorithm and the bundle adjustment algorithm, perform sparse reconstruction on the preprocessed scene image data, and obtain the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image.

[0106] Based on the above technical solutions, the current three-dimensional data model includes a neural radiation field or a three-dimensional Gaussian volume;

[0107] The model parameters in the neural radiation field include: volume density value, color value, depth value and normal value corresponding to the voxels in the three-dimensional scene;

[0108] The model parameters in the three-dimensional Gaussian volume include: position information corresponding to voxels in the three-dimensional scene, spherical harmonic function coefficients, transparency, rotation value, scale value and normal value.

[0109] Based on the above technical solutions, the model rendering module 440 is specifically used for:

[0110] In response to the current three-dimensional data model being a neural radiation field, rendering is performed according to the camera parameter information and the current three-dimensional data model by neural rendering to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image;

[0111] In response to the current three-dimensional data model being a three-dimensional Gaussian body, rendering is performed according to the camera parameter information and the current three-dimensional data model through Gaussian rasterization to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image.

[0112] Based on the above technical solutions, the model rendering module 440 includes:

[0113] A spatial position information determining unit, configured to determine spatial position information corresponding to each scene pixel in the scene image according to the current depth map;

[0114] a gradient information determining unit, configured to determine horizontal gradient information and vertical gradient information corresponding to each scene pixel according to the spatial position information;

[0115] The pseudo normal map determining unit is used to determine the normal corresponding to each scene pixel according to the horizontal gradient information and the vertical gradient information, and normalize the normal to determine the pseudo normal map under the viewing angle of the scene image.

[0116] Based on the above technical solutions, the model training module 450 includes:

[0117] a current normal loss value determining unit, configured to determine a current color loss value according to a current color map and an actual color map of the scene image, and to determine a current normal loss value according to a current normal map and a pseudo-normal map;

[0118] The model training unit is used to determine the current model loss value based on the current color loss value and the current normal loss value, and adjust the model parameters in the current three-dimensional data model based on the current model loss value until the training is completed when the preset convergence conditions are met, and the target three-dimensional data model after the training is obtained.

[0119] Based on the above technical solutions, the model training unit is specifically used to: determine the current smoothing loss value based on the current normal map, the current depth map and the actual color map of the scene image; determine the current model loss value based on the current color loss value, the current normal loss value and the current smoothing loss value.

[0120] On the basis of the above technical solutions, the device also includes:

[0121] The virtual scene image determination module is used to perform lighting and shadow adding processing according to the target three-dimensional data model, the current position information and light source type information of the virtual light source and the current parameter information of the virtual camera to obtain a virtual scene image with lighting and shadow effects.

[0122] On the basis of the above technical solutions, the virtual scene image determination module is specifically used to:

[0123] Rendering is performed according to the current position information and light source type information of the virtual light source and the target three-dimensional data model to obtain a first depth map from the perspective of the virtual light source; rendering is performed according to the current parameter information of the virtual camera and the target three-dimensional data model to obtain a target color map, a second depth map, a target normal map and a target opacity map from the perspective of the virtual camera; a target shadow map is determined according to the first depth map and the second depth map; a virtual scene image with lighting and shadow effects is determined according to the target color map, the target normal map, the target opacity map and the target shadow map.

[0124] The three-dimensional scene reconstruction device provided in the embodiments of the present disclosure can execute the three-dimensional scene reconstruction method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0126] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 5 , which shows an electronic device (eg, Figure 5 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0127] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0128] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0129] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0130] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0131] The electronic device provided by the embodiment of the present disclosure and the three-dimensional scene reconstruction method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0132] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the three-dimensional scene reconstruction method provided by the above embodiments is implemented.

[0133] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0134] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0135] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0136] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains three-dimensional scene data collected for the three-dimensional scene to be reconstructed, the three-dimensional scene data including a scene image of the three-dimensional scene; determines sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image according to the scene image; performs model initialization according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene; performs rendering according to the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map and a current normal map from the perspective of the scene image, and determines a pseudo-normal map from the perspective of the scene image according to the current depth map; trains the current three-dimensional data model according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain a target three-dimensional data model after the training is completed.

[0137] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0138] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0140] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0141] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] The embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements the three-dimensional scene reconstruction method provided in the above embodiments.

[0143] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0144] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0145] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0146] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A three-dimensional scene reconstruction method, characterized in that: include: Acquire three-dimensional scene data collected for the three-dimensional scene to be reconstructed, wherein the three-dimensional scene data includes a scene image of the three-dimensional scene; Determine, according to the scene image, sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image; Initialize the model according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene; Rendering is performed according to the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map and a current normal map under the perspective of the scene image, and a pseudo-normal map under the perspective of the scene image is determined according to the current depth map, wherein the current color map includes a current color value corresponding to each pixel, the current depth map includes a current depth value corresponding to each pixel, and the current normal map includes a current normal value corresponding to each pixel; The current three-dimensional data model is trained according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain a target three-dimensional data model after the training is completed.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The determining, according to the scene image, the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image includes: Performing data processing on the scene image to obtain scene image data in a preset image format; Preprocessing scene image data in a preset image format to obtain preprocessed scene image data; Based on the motion recovery structure algorithm and the bundle adjustment algorithm, the preprocessed scene image data is sparsely reconstructed to obtain sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image.

3. The three-dimensional scene reconstruction method according to claim 2, characterized in that: The three-dimensional scene data also includes: scene acquisition data in non-image format; The method of performing sparse reconstruction on the preprocessed scene image data based on the motion recovery structure algorithm and the bundle adjustment algorithm to obtain sparse point cloud data corresponding to the three-dimensional scene and camera parameter information corresponding to the scene image includes: The scene acquisition data in non-image format is added as an algorithm constraint to the motion recovery structure algorithm and the bundle adjustment algorithm, and the pre-processed scene image data is sparsely reconstructed to obtain the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image.

4. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The current three-dimensional data model includes a neural radiation field or a three-dimensional Gaussian volume; The model parameters in the neural radiation field include: volume density value, color value, depth value and normal value corresponding to the voxels in the three-dimensional scene; The model parameters in the three-dimensional Gaussian volume include: position information corresponding to voxels in the three-dimensional scene, spherical harmonic function coefficients, transparency, rotation value, scale value and normal value.

5. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The rendering is performed according to the camera parameter information and the current three-dimensional data model to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image, including: In response to the current three-dimensional data model being a neural radiation field, rendering is performed according to the camera parameter information and the current three-dimensional data model by neural rendering to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image; In response to the current three-dimensional data model being a three-dimensional Gaussian body, rendering is performed according to the camera parameter information and the current three-dimensional data model through Gaussian rasterization to obtain a current color map, a current depth map, and a current normal map under the perspective of the scene image.

6. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The determining the pseudo normal map under the viewing angle of the scene image according to the current depth map includes: Determine spatial position information corresponding to each scene pixel in the scene image according to the current depth map; Determining horizontal gradient information and vertical gradient information corresponding to each scene pixel according to the spatial position information; A normal corresponding to each scene pixel is determined according to the horizontal gradient information and the vertical gradient information, and the normal is normalized to determine a pseudo normal map under the viewing angle of the scene image.

7. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The training of the current three-dimensional data model according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain the target three-dimensional data model after the training includes: Determine a current color loss value according to the current color map and the actual color map of the scene image, and determine a current normal loss value according to the current normal map and the pseudo-normal map; The current model loss value is determined based on the current color loss value and the current normal loss value, and based on the current model loss value, the model parameters in the current three-dimensional data model are adjusted until the training ends when the preset convergence conditions are met, and the target three-dimensional data model after the training is obtained.

8. The three-dimensional scene reconstruction method according to claim 7, characterized in that: The determining the current model loss value based on the current color loss value and the current normal loss value includes: Determining a current smoothing loss value based on a current normal map, a current depth map, and an actual color map of the scene image; The current model loss value is determined based on the current color loss value, the current normal loss value, and the current smoothness loss value.

9. The three-dimensional scene reconstruction method according to any one of claims 1 to 8, characterized in that: The method further comprises: Lighting and shadow adding processing is performed according to the target three-dimensional data model, current position information and light source type information of the virtual light source, and current parameter information of the virtual camera to obtain a virtual scene image with lighting and shadow effects.

10. The three-dimensional scene reconstruction method according to claim 9, characterized in that: The step of performing lighting and shadow adding processing according to the target three-dimensional data model, the current position information and light source type information of the virtual light source, and the current parameter information of the virtual camera to obtain a virtual scene image with lighting and shadow effects includes: Rendering is performed according to the current position information and light source type information of the virtual light source and the target three-dimensional data model to obtain a first depth map under the perspective of the virtual light source; Rendering is performed according to the current parameter information of the virtual camera and the target three-dimensional data model to obtain a target color map, a second depth map, a target normal map, and a target opacity map under the perspective of the virtual camera; Determining a target shadow map according to the first depth map and the second depth map; A virtual scene image with lighting and shadow effects is determined according to the target color map, the target normal map, the target opacity map and the target shadow map.

11. A three-dimensional scene reconstruction device, characterized in that: include: A three-dimensional scene data acquisition module, used to acquire three-dimensional scene data collected for the three-dimensional scene to be reconstructed, wherein the three-dimensional scene data includes a scene image of the three-dimensional scene; A sparse point cloud data determination module, used to determine the sparse point cloud data corresponding to the three-dimensional scene and the camera parameter information corresponding to the scene image according to the scene image; A model initialization module, used to initialize the model according to the sparse point cloud data to obtain a current three-dimensional data model corresponding to the three-dimensional scene; A model rendering module, used to render according to the camera parameter information and the current three-dimensional data model, obtain a current color map, a current depth map and a current normal map under the perspective of the scene image, and determine a pseudo-normal map under the perspective of the scene image according to the current depth map, wherein the current color map includes a current color value corresponding to each pixel, the current depth map includes a current depth value corresponding to each pixel, and the current normal map includes a current normal value corresponding to each pixel; The model training module is used to train the current three-dimensional data model according to the current color map, the current normal map, the pseudo-normal map and the actual color map of the scene image to obtain the target three-dimensional data model after the training is completed.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional scene reconstruction method as described in any one of claims 1-10.

13. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the three-dimensional scene reconstruction method as described in any one of claims 1-10 when executed by a computer processor.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the three-dimensional scene reconstruction method as described in any one of claims 1 to 10.

Images