Sparse view angle complex outdoor scene three-dimensional reconstruction method, electronic equipment and storage medium

Abstract

The invention discloses a sparse view angle complex outdoor scene three-dimensional reconstruction method, electronic equipment and a storage medium. The method comprises the steps of obtaining camera pose information and initial three-dimensional point cloud information with rich geometric priori through an initialization module; according to the camera pose information and the initial three-dimensional point cloud information, in combination with a self-attention mechanism, a first target image is generated through a new view angle enhancement module, and the first target image is used as a pseudo-supervision signal to supervise an original rendering image; according to the camera pose information and the initial three-dimensional point cloud information, a high-fidelity second target image with transient shielding removed is acquired through a shielding processing module; and optimizing the three-dimensional Gaussian radiation field through a course learning strategy according to the first target image and the second target image, and reconstructing to obtain a three-dimensional image of the complex outdoor scene. The embodiment of the invention can reproduce the landscape scene vividly, provides immersive roaming experience, can improve the reconstruction effect and visual consistency, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a three-dimensional reconstruction method for a sparse-viewing angle complex outdoor scene, an electronic device and a storage medium. Background Art

[0002] Common methods for sparse view scene reconstruction include methods based on regularization strategies and methods based on depth priors. The core idea of ​​the regularization strategy-based method is to improve modeling accuracy by introducing different regularization strategies to guide model training. For example, the RegNeRF model proposed in the related art designs a regularization strategy based on image blocks, reduces artifacts and improves the geometric modeling ability of the model. There is also a related art that proposes a CoR-GS model, introduces a three-dimensional Gaussian splashing technology as a scene representation method, and improves the reconstruction quality through pruning and pseudo-view joint regularization. Methods based on depth priors usually introduce additional depth information to improve the model effect. There is also a related art that proposes a DS-NeRF model based on depth information supervision. There is also a related art that proposes a DNGaussian model based on three-dimensional Gaussian splashing technology, which introduces a pre-trained depth estimation model to obtain more accurate depth information, thereby achieving more accurate modeling results. However, these methods face challenges when applied to complex outdoor scenes because they assume that the scene is stationary and unchanged, and do not consider transient occlusions (such as pedestrians, vehicles, etc.) and dynamic appearance colors caused by different lighting conditions.

[0003] In order to cope with complex outdoor scenes, the related technology proposed the first NeRF-W model that applies implicit radiation fields to complex outdoor scenes. This model optimizes the appearance vector of each image to characterize different lighting conditions, and trains a multi-layer perceptron to simulate transient radiation fields to remove transient occlusions. The CR-NeRF model proposed in the related technology handles transient occlusions by optimizing the two-dimensional visibility map, and introduces a CNN-based appearance encoder to model dynamic appearance. With the emergence and development of three-dimensional Gaussian splashing technology, the related technology proposed a GS-W model based on three-dimensional Gaussian splashing technology, which introduces intrinsic and dynamic appearance features for each Gaussian point to better simulate different lighting conditions. The related technology also proposed a WildGaussians model, which uses a learnable multi-layer perceptron to extract a global appearance vector from a reference image, and introduces a DINO feature prior to handle occlusions. However, these methods essentially rely on neural networks to classify static objects based on multi-view images. Figure 1 It is difficult to achieve good reconstruction results under sparse view input conditions. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to propose a sparse perspective complex outdoor scene three-dimensional reconstruction method, electronic device and storage medium, which can realistically reproduce the landscape scene, provide an immersive roaming experience, and improve the reconstruction effect and visual consistency.

[0005] To achieve the above object, an embodiment of the present invention provides a method for 3D reconstruction of a complex outdoor scene with sparse viewing angles, comprising the following steps:

[0006] Obtain camera pose information and initial 3D point cloud information with rich geometric priors through the initialization module;

[0007] According to the camera pose information and the initial three-dimensional point cloud information, in combination with a self-attention mechanism, a first target image is generated through a new perspective enhancement module, where the first target image is used as a pseudo supervision signal to supervise the original rendered image;

[0008] According to the camera pose information and the initial three-dimensional point cloud information, a high-fidelity second target image with transient occlusion removed is acquired through an occlusion processing module;

[0009] According to the first target image and the second target image, the three-dimensional Gaussian radiation field is optimized through a curriculum learning strategy to reconstruct a three-dimensional image of a complex outdoor scene.

[0010] In some embodiments, the step of obtaining the camera pose information and the initial three-dimensional point cloud information with rich geometric priors through the initialization module includes the following steps:

[0011] Acquire multiple complex outdoor scene images;

[0012] The multi-view stereo vision model DUSt3R is used to obtain camera poses and 3D point clouds with rich geometric priors from complex outdoor scene images.

[0013] Use a pre-trained segmentation model EVF-SAM to generate corresponding occlusion masks based on the predetermined text guidance;

[0014] After replacing the color of the occluded area with Gaussian noise, the camera pose information and the initial three-dimensional point cloud information are obtained through a new formula for obtaining the point cloud and camera pose.

[0015] In some embodiments, the process of replacing the color of the occluded area with Gaussian noise is expressed as:

[0016]

[0017] in, represents a collection of complex outdoor scene images with occluded areas replaced; M i Represents complex outdoor scene images The occlusion mask of ; ⊙ represents the element-by-element product; Represents the mean μ and variance σ 2 Gaussian noise; Represents a complex outdoor scene image; 1 represents the same as M i A matrix of the same shape, but with all elements set to 1;

[0018] The new formula for obtaining point cloud and camera pose is:

[0019]

[0020] in, represents a 3D point cloud with rich geometric priors; C represents the camera pose; Represents the multi-view stereo vision model DUSt3R.

[0021] In some embodiments, the step of generating a first target image through a new perspective enhancement module based on the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism comprises the following steps:

[0022] First, we sample new viewpoints from the training camera pose C; for each new viewpoint, we render an image I n , apply DDIM inversion to transform it into a standard Gaussian distribution x T ;

[0023] x T Input into two branches of the counter-diffusion process;

[0024] Reconstruction branch gradually to x T Denoising to reconstruct the original rendered image; using a fine-tuned diffusion model via the enhancement branch Generate high-quality images

[0025] In some embodiments, the generating of the first target image by a new perspective enhancement module based on the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism further includes the following steps:

[0026] The structural features of the original rendered image are injected into the enhancement branch, specifically:

[0027] In the self-attention mechanism, the query and key in the reconstruction branch are represented by Q r and K r Indicates; in the query of the enhanced branch, the key and value are represented by Q e ,K e and V e express;

[0028] By adding Q e and Ke Replace with Q r and K r Implement self-attention injection operation;

[0029] Among them, the expression of the self-attention injection operation is:

[0030] Among them, the output feature F e quilt Used to predict noise, d represents the vector dimension;

[0031] After T steps of denoising, a high-quality image is obtained

[0032] Use AdaIn as post-processing to manipulate the image based on a user-provided reference image The global appearance of To supervise the original rendered image I n .

[0033] In some embodiments, the step of obtaining a high-fidelity second target image with transient occlusion removed by an occlusion processing module according to the camera pose information and the initial three-dimensional point cloud information comprises the following steps:

[0034] Using a new strategy, we introduce the pre-trained segmentation network EVF-SAM to obtain the occlusion mask

[0035] Based on the similarity between the original information in the occluded area and the information in the adjacent area, this similarity is used to fill the occlusion, specifically:

[0036] Through the diffusion model Intrinsic self-attention mechanism and constrained priors to enhance occluded regions;

[0037] Given a training viewpoint, render image I t Its supervisory signal Use DDIM inversion to obtain the corresponding Gaussian distribution x′ T and

[0038] Use the corresponding mask M i For x′ T and Perform weighted fusion to obtain The expression of this process is:

[0039] right Perform the reverse diffusion process of the two branches and use the corresponding mask M iPerform weighted self-attention injection, the expression of this process is:

[0040] Q OH =M i ⊙Q′ e +(1-M i )⊙Q r ′,

[0041] K OH =M i ⊙K e ′+(1-M i )⊙K r ′,

[0042]

[0043] Among them, the output feature F e 'quilt Used to predict noise; output feature F e 'quilt Used to predict noise and output the feature F e ′ is used as a supervisory signal to optimize the Gaussian radiation field; Q OH represents the query of the enhanced branch after fusion; K OH represents the bond in the enhanced branch after fusion; Q′ e represents the query of the enhancement branch in the occlusion processing module; K e ′ represents the key of the enhancement branch in the occlusion processing module; Q r ′ represents the query of the reconstruction branch in the occlusion processing module; K r ′ represents the key of the reconstruction branch in the occlusion processing module; V e ′ represents the value of the enhancement branch in the occlusion processing module.

[0044] In some embodiments, the optimizing the three-dimensional Gaussian radiation field by the curriculum learning strategy comprises the following steps:

[0045] First train the Gaussian radiation field with simple samples, then gradually move to more complex samples;

[0046] Randomly select training perspectives to train the Gaussian radiation field until it fits, and then gradually incorporate the sampled new perspectives into the training process in three stages according to the complexity of the new perspectives;

[0047] These new perspectives are divided into three levels based on their Euclidean distance from the training perspective: easy, medium, and difficult;

[0048] New perspectives are selected for training with probability β. Through this curriculum learning strategy, high-quality and multi-perspective consistent Gaussian radiation fields can eventually be obtained.

[0049] In some embodiments, the method further comprises the following steps:

[0050] Rendering image I for the new perspective n , using its pseudo-supervisory signal To conduct supervision: Among them, L1 represents the L1 loss function, L SSIM represents the SSIM loss function; L c represents the loss of luminosity;

[0051] For the training view rendered image I t , when the iteration number is less than τ, use the corresponding mask M i Mask transient occlusion; when the iteration round is greater than or equal to τ, use the pseudo supervision signal Supervision of the entire image:

[0052]

[0053] in, L o represents occlusion loss; I t Represents the training viewpoint rendered image;

[0054] The total loss function is expressed as: L = L o +λ3L c ;

[0055] Among them, λ1, λ2, λ3, τ are hyperparameters.

[0056] Another aspect of the embodiments of the present invention further provides a sparse viewpoint complex outdoor scene 3D reconstruction system, comprising:

[0057] The first module is used to obtain the camera pose information and the initial 3D point cloud information with rich geometric priors through the initialization module;

[0058] A second module is used to generate a first target image through a new perspective enhancement module according to the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism, wherein the first target image is used as a pseudo supervision signal to supervise the original rendered image;

[0059] A third module is used to obtain a high-fidelity second target image with transient occlusion removed through an occlusion processing module according to the camera posture information and the initial three-dimensional point cloud information;

[0060] The fourth module is used to optimize the three-dimensional Gaussian radiation field according to the first target image and the second target image through a curriculum learning strategy to reconstruct a three-dimensional image of a complex outdoor scene.

[0061] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0062] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0063] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0064] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a sparse viewpoint complex outdoor scene three-dimensional reconstruction method, electronic device and storage medium, the scheme obtains camera pose information and initial three-dimensional point cloud information with rich geometric priors through an initialization module; according to the camera pose information and the initial three-dimensional point cloud information, combined with the self-attention mechanism, a first target image is generated through a new viewpoint enhancement module, and the first target image is used as a pseudo-supervisory signal to supervise the original rendered image; according to the camera pose information and the initial three-dimensional point cloud information, a high-fidelity second target image with transient occlusion removed is obtained through an occlusion processing module; according to the first target image and the second target image, a three-dimensional Gaussian radiation field is optimized through a curriculum learning strategy to reconstruct a three-dimensional image of a complex outdoor scene. The embodiments of the present invention can realistically reproduce landscape scenes, provide an immersive roaming experience, and improve reconstruction effects and visual consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0066] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0067] Figure 3 It is a schematic diagram of the overall structure of SparseGS-W provided in an embodiment of the present invention;

[0068] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0070] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0071] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0073] The sparse perspective complex outdoor scene three-dimensional reconstruction method, electronic device and storage medium provided by the embodiment of the present invention relate to the field of computer technology. The sparse perspective complex outdoor scene three-dimensional reconstruction method provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a sparse perspective complex outdoor scene three-dimensional reconstruction method, etc., but is not limited to the above forms.

[0074] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0076] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0077] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0078] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0079] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a method for three-dimensional reconstruction of complex outdoor scenes with sparse perspectives. The following is explained using the example of the method for three-dimensional reconstruction of complex outdoor scenes with sparse perspectives being applied to the server 101. It can be understood that the method can also be applied to the terminal 102.

[0080] Reference Figure 2 , Figure 2 The flowchart of the method for 3D reconstruction of complex outdoor scenes with sparse viewing angles applied to a server provided in an embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0081] Obtain camera pose information and initial 3D point cloud information with rich geometric priors through the initialization module;

[0082] According to the camera pose information and the initial three-dimensional point cloud information, in combination with a self-attention mechanism, a first target image is generated through a new perspective enhancement module, where the first target image is used as a pseudo supervision signal to supervise the original rendered image;

[0083] According to the camera pose information and the initial three-dimensional point cloud information, a high-fidelity second target image with transient occlusion removed is acquired through an occlusion processing module;

[0084] According to the first target image and the second target image, the three-dimensional Gaussian radiation field is optimized through a curriculum learning strategy to reconstruct a three-dimensional image of a complex outdoor scene.

[0085] In some embodiments, the step of obtaining the camera pose information and the initial three-dimensional point cloud information with rich geometric priors through the initialization module includes the following steps:

[0086] Acquire multiple complex outdoor scene images;

[0087] The multi-view stereo vision model DUSt3R is used to obtain camera poses and 3D point clouds with rich geometric priors from complex outdoor scene images.

[0088] Use a pre-trained segmentation model EVF-SAM to generate corresponding occlusion masks based on the predetermined text guidance;

[0089] After replacing the color of the occluded area with Gaussian noise, the camera pose information and the initial three-dimensional point cloud information are obtained through a new formula for obtaining the point cloud and camera pose.

[0090] In some embodiments, the process of replacing the color of the occluded area with Gaussian noise is expressed as:

[0091]

[0092] in, represents a collection of complex outdoor scene images with occluded areas replaced; M i Represents complex outdoor scene images The occlusion mask of ; ⊙ represents the element-by-element product; Represents the mean μ and variance σ 2 Gaussian noise; Represents a complex outdoor scene image; 1 represents the same as M i A matrix of the same shape, but with all elements set to 1;

[0093] The new formula for obtaining point cloud and camera pose is:

[0094]

[0095] in, represents a 3D point cloud with rich geometric priors; C represents the camera pose; Represents the multi-view stereo vision model DUSt3R.

[0096] In some embodiments, the step of generating a first target image through a new perspective enhancement module based on the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism comprises the following steps:

[0097] First, we sample new viewpoints from the training camera pose C; for each new viewpoint, we render an image I n , apply DDIM inversion to transform it into a standard Gaussian distribution x T ;

[0098] x T Input into two branches of the counter-diffusion process;

[0099] Reconstruction branch gradually to x T Denoising to reconstruct the original rendered image; using a fine-tuned diffusion model via the enhancement branch Generate high-quality images

[0100] In some embodiments, the generating of the first target image by a new perspective enhancement module based on the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism further includes the following steps:

[0101] The structural features of the original rendered image are injected into the enhancement branch, specifically:

[0102] In the self-attention mechanism, the query and key in the reconstruction branch are represented by Q r and K r Indicates; in the query of the enhanced branch, the key and value are represented by Q e ,K e and V e express;

[0103] By adding Q e and K e Replace with Q r and K r Implement self-attention injection operation;

[0104] Among them, the expression of the self-attention injection operation is:

[0105] Among them, the output feature F e quilt Used to predict noise, d represents the vector dimension;

[0106] After T steps of denoising, a high-quality image is obtained

[0107] Use AdaIn as post-processing to manipulate the image based on a user-provided reference image The global appearance of To supervise the original rendered image I n .

[0108] In some embodiments, the step of obtaining a high-fidelity second target image with transient occlusion removed by an occlusion processing module according to the camera pose information and the initial three-dimensional point cloud information comprises the following steps:

[0109] Using a new strategy, the pre-trained segmentation network EVF-SAM is introduced to obtain the occlusion mask M;

[0110] Based on the similarity between the original information in the occluded area and the information in the adjacent area, this similarity is used to fill the occlusion, specifically:

[0111] Through the diffusion model Intrinsic self-attention mechanism and constrained priors to enhance occluded regions;

[0112] Given a training viewpoint, render image I t Its supervisory signal Use DDIM inversion to obtain the corresponding Gaussian distribution x′ T and

[0113] Use the corresponding mask M i For x′ T and Perform weighted fusion to obtain The expression of this process is:

[0114] right Perform the reverse diffusion process of the two branches and use the corresponding mask M i Perform weighted self-attention injection, the expression of this process is:

[0115] Q OH =M i ⊙Q′ e +(1-M i )⊙Q r ′,

[0116] K OH =M i ⊙K e ′+(1-M i )⊙K r ′,

[0117]

[0118] Among them, the output feature F e 'quilt Used to predict noise; output feature F e 'quilt Used to predict noise and output the feature F e′ is used as a supervisory signal to optimize the Gaussian radiation field; Q OH represents the query of the enhanced branch after fusion; K OH represents the bond in the enhanced branch after fusion; Q′ e represents the query of the enhancement branch in the occlusion processing module; K e ′ represents the key of the enhancement branch in the occlusion processing module; Q r ′ represents the query of the reconstruction branch in the occlusion processing module; K r ′ represents the key of the reconstruction branch in the occlusion processing module; V′ e Represents the value of the enhancement branch in the occlusion processing module.

[0119] In some embodiments, the optimizing the three-dimensional Gaussian radiation field by the curriculum learning strategy comprises the following steps:

[0120] First train the Gaussian radiation field with simple samples, then gradually move to more complex samples;

[0121] Randomly select training perspectives to train the Gaussian radiation field until it fits, and then gradually incorporate the sampled new perspectives into the training process in three stages according to the complexity of the new perspectives;

[0122] These new perspectives are divided into three levels based on their Euclidean distance from the training perspective: easy, medium, and difficult;

[0123] New perspectives are selected for training with probability β. Through this curriculum learning strategy, high-quality and multi-perspective consistent Gaussian radiation fields can eventually be obtained.

[0124] In some embodiments, the method further comprises the following steps:

[0125] Rendering image I for the new perspective n , using its pseudo-supervisory signal To conduct supervision: Among them, L1 represents the L1 loss function, L SSIM represents the SSIM loss function; L c represents the loss of luminosity;

[0126] For the training view rendered image I t , when the iteration number is less than τ, use the corresponding mask M i Mask transient occlusion; when the iteration round is greater than or equal to τ, use the pseudo supervision signal Supervision of the entire image:

[0127]

[0128] in, L o represents occlusion loss; It Represents the training viewpoint rendered image;

[0129] The total loss function is expressed as: L = L o +λ3L c ;

[0130] Among them, λ1, λ2, λ3, τ are hyperparameters.

[0131] The implementation process of the embodiment of the present invention in a specific application scenario is described in detail below with reference to the accompanying drawings of the specification:

[0132] Specifically, the overall structure of SparseGS-W is as follows: Figure 3 As shown, the method consists of three modules, namely, an initialization module, a new perspective enhancement module and an occlusion processing module. Given a sparse perspective of a complex outdoor scene, the initialization module is responsible for estimating the camera pose and generating an initial point cloud. The new perspective enhancement module uses a diffusion model prior to enhance low-quality new perspectives to ensure that the generated image has high quality and three-dimensional consistency. The occlusion processing module is responsible for removing transient occlusions during training. In addition, this embodiment introduces a curriculum learning strategy to accelerate model convergence and improve reconstruction accuracy.

[0133] The following is a detailed description of the specific contents of each module:

[0134] 1. Initialization module:

[0135] The 3D Gaussian splashing technique is an explicit representation method for static scenes. The ability to model the scene depends largely on the quality of the initial point cloud and the supervision signal. The traditional motion restoration structure method cannot provide sufficient geometric priors, which will affect the training quality of the three-dimensional Gaussian radiation field. Therefore, the embodiment of the present invention adopts the multi-view stereo vision model DUSt3R, using To express, in order to obtain the camera pose C and the 3D point cloud P with rich geometric priors:

[0136]

[0137] However, the present invention observed that the DUSt3R model assumes that there is no transient occlusion in the image, which causes it to mistake transient occlusion for the texture of the static object surface, which in turn affects subsequent training. To solve this problem, the present invention uses a pre-trained segmentation model EVF-SAM to generate the corresponding occlusion mask based on the text guidance given by the user. Then, the embodiment of the present invention replaces the color of the occluded area with Gaussian noise:

[0138]

[0139] Among them, 1 means that i A matrix of the same shape, but with all elements set to 1. represents the mean μ and variance σ 2 The Gaussian noise of , the mean and variance come from the non-occluded area. The embodiment of the present invention modifies the formula for obtaining the point cloud and camera pose:

[0140]

[0141] 2. New perspective enhancement module:

[0142] The goal of the embodiment of the present invention is to enhance the low-quality images rendered by the three-dimensional Gaussian radiation field during the training process. In order to avoid identity shift during the enhancement process, the embodiment of the present invention uses sparse training views as anchor points to fine-tune the diffusion model ε θ (x t ,t), thereby restricting the generated space to a clean subspace. In order to distinguish it from the original diffusion model, the fine-tuned diffusion model is expressed as

[0143] The embodiment of the present invention first samples a new perspective from the training camera pose C. For each new perspective, the rendered image I n , the embodiment of the present invention applies DDIM inversion to convert it into a standard Gaussian distribution x T . Then, x T Input to two branches of the reverse diffusion process. The reconstruction branch gradually T The denoising branch is used to reconstruct the original rendered image, while the enhancement branch focuses on using Generate high-quality images

[0144] The present invention observes that although the fine-tuning diffusion model can effectively restrict the generation space to produce high-quality images with the same identity, it is difficult to accurately preserve the structure of the image, resulting in poor three-dimensional consistency. To this end, the present invention injects the structural features of the original rendered image into the enhancement branch. Specifically, in the self-attention mechanism, the query and key in the reconstruction branch are represented by Q r and K r Indicates that the query in the enhanced branch, the key and value (Value) are respectively represented by Q e ,K e and V e By e and K e Replace with Q r and K r Implemented self-attention injection operation:

[0145]

[0146] Among them, the output feature F e quilt is used to predict noise, and d represents the vector dimension. After T steps of denoising, the embodiment of the present invention obtains a high-quality image The image removes artifacts while retaining identity information. The embodiment of the present invention does not train an additional appearance extraction network, but uses AdaIn as a post-processing to control the image according to the reference image provided by the user. The global appearance of . Finally, this image is used as a pseudo supervision signal To supervise the original rendered image I n .

[0147] 3. Occlusion processing module:

[0148] Previous methods all use dense viewpoints to train neural networks to predict transient occlusions, which is difficult to achieve under sparse viewpoint conditions. In addition, their unsupervised training paradigm also leads to poor flexibility and inability to remove specific occlusions according to user needs. Therefore, the embodiment of the present invention uses a new strategy to introduce a pre-trained segmentation network EVF-SAM to flexibly obtain occlusion masks. The embodiments of the present invention observe that the original information in the occluded area is usually similar to the information in the adjacent area, or similar to some areas in other training views. Based on this discovery, the embodiments of the present invention can use this similarity to fill the occlusion. Specifically, the embodiments of the present invention use the diffusion model to Intrinsic self-attention mechanism and constrained priors are used to enhance occluded regions. Given a training view, rendered image I t Its supervisory signal The embodiment of the present invention uses DDIM inversion to obtain the corresponding x′ T and Then, use the corresponding mask M i For x′ T and Perform weighted fusion to obtain

[0149]

[0150] Similar to the previous section, the present invention embodiment Perform the reverse diffusion process of the two branches. In order to distinguish the symbols, the embodiment of the present invention uses superscripts to distinguish Q' e , K e ′,V e ′,Q r ′,K r ′ and Q e , K e , V e , Qr and K r The difference is that the embodiment of the present invention uses the corresponding mask M i Perform weighted self-attention injection:

[0151] Q OH =M i ⊙Q′ e +(1-M i )⊙Q r ′,

[0152] K OH =M i ⊙K e ′+(1-M i )⊙K r ′,

[0153]

[0154] Output feature F e 'quilt Used to predict noise. After T-step denoising and post-processing, the embodiment of the present invention obtains a high-fidelity image with transient occlusion removed. It is used as a supervisory signal to optimize the Gaussian radiation field. The weighted fusion and injection operations help minimize the information loss inherent in the diffusion process, making the constrained Focus on enhancing occluded areas.

[0155] 4. Course learning strategies:

[0156] Inspired by the fact that human learning usually proceeds from simple to complex, the embodiment of the present invention introduces a curriculum learning strategy to gradually optimize the three-dimensional Gaussian radiation field. Specifically, the embodiment of the present invention first trains the Gaussian radiation field with simple samples, and then gradually turns to more complex samples. The embodiment of the present invention randomly selects training perspectives to train the Gaussian radiation field until it is fitted, and then gradually incorporates the sampled new perspectives into the training process in three stages according to the complexity of the new perspectives. These new perspectives are divided into three levels according to the Euclidean distance from the training perspective: simple, medium, and difficult. In order to reduce the error introduced by the diffusion model, the embodiment of the present invention selects new perspectives for training with a probability β. Through this curriculum learning strategy, the embodiment of the present invention can eventually obtain a high-quality and multi-perspective consistent Gaussian radiation field, thereby improving the reconstruction effect and visual consistency.

[0157] 5. Model loss function:

[0158] Rendering image I for the new perspective n , the embodiment of the present invention uses its pseudo supervisory signal To conduct supervision:

[0159]

[0160] Where L1 represents the L1 loss function, L SSIM Represents the SSIM loss function.

[0161] For the training view rendered image I t When the iteration number is less than τ, the embodiment of the present invention uses the corresponding mask M i When the iteration number is greater than or equal to τ, a pseudo supervision signal is used. Supervision of the entire image:

[0162]

[0163] in Combining all loss terms, the total loss function can be expressed as:

[0164] L=L o +λ3L c ,

[0165] Among them, λ1, λ2, λ3, τ are hyperparameters, which are set to 0.8, 0.2, 1.0 and 6500 respectively in this method. This method is trained on the dataset for 7500 rounds. This method sets β to 0.3.

[0166] In summary, the present invention proposes for the first time a 3D reconstruction method SparseGS-W for complex outdoor scenes with sparse viewpoints. The method uses the prior information of the pre-trained diffusion model to simplify the problem of synthesizing high-quality new viewpoints into an image enhancement problem with 3D consistency. The method uses the given sparse viewpoint to quickly fine-tune the pre-trained diffusion model, and introduces a self-attention injection strategy to retain the unique identity information and local geometric structure of the original low-quality image during the enhancement process. Based on the observation that the original texture of the occluded area is similar to that of the adjacent area or similar areas in other training viewpoints, the method uses the fine-tuned diffusion model to gradually remove transient occlusions in the generated space. In order to be able to model multiple appearances, the method adopts a simple but effective post-processing operation, which can render high-quality images of similar appearance based on any reference image given by the user without adding additional training costs.

[0167] The present invention enables users to visit places of interest around the world, such as the Brandenburg Gate in Germany, in an immersive way at home. Users can achieve this goal by simply selecting a few photos taken during previous travels from their personal albums, or by directly searching and downloading tourist photos uploaded by others from the Internet. The present invention can realistically reproduce landscape scenes, and according to user preferences, remove any transient occlusions in virtual scenes, simulate different seasons and light and shadow effects, and provide an immersive roaming experience. After experimental comparison, the method of the embodiment of the present invention is 5.74dB higher in PSNR index, 0.24 higher in LPIPS index, and 0.20 higher in SSIM index than the current advanced methods on the commonly used complex outdoor scene dataset PhotoTourism. In addition, the method of the embodiment of the present invention also surpasses the existing advanced methods on the commonly used unsupervised indicators FID, MUSIQ and ClipIQA, which fully proves the effectiveness and superiority of the method of the embodiment of the present invention.

[0168] The method based on the present invention can be widely applied to the following related fields:

[0169] 1. Virtual roaming outdoor scenes in the fields of culture, tourism, education, electronic entertainment, and VR. SparseGS-W can be used to virtually roam around places of interest around the world. Users only need to provide a few sparse perspective photos to generate realistic 3D scenes, remove transient occlusions according to user preferences, and simulate different seasons and light and shadow effects, making virtual tourism possible.

[0170] 2. The plug-and-play new perspective enhancement module can be applied to any other new perspective synthesis tasks.

[0171] 3. The curriculum learning strategy can be applied to other 3D reconstruction tasks based on 3D Gaussian splattering technology.

[0172] 4. Cultural heritage protection and digitization: The SparseGS-W method can achieve high-quality reconstruction under limited data conditions, providing support for the digital protection of cultural heritage.

[0173] Another aspect of the embodiments of the present invention further provides a sparse viewpoint complex outdoor scene 3D reconstruction system, comprising:

[0174] The first module is used to obtain the camera pose information and the initial 3D point cloud information with rich geometric priors through the initialization module;

[0175] A second module is used to generate a first target image through a new perspective enhancement module according to the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism, wherein the first target image is used as a pseudo supervision signal to supervise the original rendered image;

[0176] A third module is used to obtain a high-fidelity second target image with transient occlusion removed through an occlusion processing module according to the camera posture information and the initial three-dimensional point cloud information;

[0177] The fourth module is used to optimize the three-dimensional Gaussian radiation field according to the first target image and the second target image through a curriculum learning strategy to reconstruct a three-dimensional image of a complex outdoor scene.

[0178] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0179] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned sparse perspective complex outdoor scene three-dimensional reconstruction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0180] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0181] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0182] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0183] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the sparse viewpoint complex outdoor scene three-dimensional reconstruction method of the embodiment of the present invention;

[0184] Input / output interface 403, used to implement information input and output;

[0185] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0186] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0187] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0188] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned sparse perspective complex outdoor scene three-dimensional reconstruction method is implemented.

[0189] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0190] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0191] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0192] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0193] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0194] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0196] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0197] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0198] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0201] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0202] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A 3D reconstruction method for complex outdoor scenes with sparse viewing angles, characterized in that: The following steps are involved: Obtain camera pose information and initial 3D point cloud information with rich geometric priors through the initialization module; According to the camera pose information and the initial three-dimensional point cloud information, in combination with a self-attention mechanism, a first target image is generated through a new perspective enhancement module, where the first target image is used as a pseudo supervision signal to supervise the original rendered image; According to the camera pose information and the initial three-dimensional point cloud information, a high-fidelity second target image with transient occlusion removed is acquired through an occlusion processing module; According to the first target image and the second target image, the three-dimensional Gaussian radiation field is optimized through a curriculum learning strategy to reconstruct a three-dimensional image of a complex outdoor scene.

2. The method for 3D reconstruction of complex outdoor scenes with sparse viewing angles according to claim 1, characterized in that: The method of obtaining the camera pose information and the initial three-dimensional point cloud information with rich geometric priors through the initialization module includes the following steps: Acquire multiple complex outdoor scene images; The multi-view stereo vision model DUSt3R is used to obtain camera poses and 3D point clouds with rich geometric priors from complex outdoor scene images. Use a pre-trained segmentation model EVF-SAM to generate corresponding occlusion masks based on the predetermined text guidance; After replacing the color of the occluded area with Gaussian noise, the camera pose information and the initial three-dimensional point cloud information are obtained through a new formula for obtaining the point cloud and camera pose.

3. The method for 3D reconstruction of complex outdoor scenes with sparse viewing angles according to claim 2, characterized in that: The expression of the process of replacing the color of the occluded area with Gaussian noise is: in, represents a collection of complex outdoor scene images with occluded areas replaced; M i Represents complex outdoor scene images The occlusion mask of ; ⊙ represents the element-by-element product; Represents the mean μ and variance σ 2 Gaussian noise; Represents a complex outdoor scene image; 1 represents the same as M i A matrix of the same shape, but with all elements set to 1; The new formula for obtaining point cloud and camera pose is: in, represents a 3D point cloud with rich geometric priors; C represents the camera pose; Represents the multi-view stereo vision model DUSt3R.

4. The method for 3D reconstruction of complex outdoor scenes with sparse viewing angles according to claim 1, characterized in that: The method of generating a first target image through a new perspective enhancement module according to the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism comprises the following steps: First, we sample new viewpoints from the training camera pose C; for each new viewpoint, we render an image I n , apply DDIM inversion to transform it into a standard Gaussian distribution x T ; x T Input into two branches of the counter-diffusion process; Reconstruction branch gradually to x T Denoising to reconstruct the original rendered image; using a fine-tuned diffusion model via the enhancement branch Generate high-quality images 5. The method for 3D reconstruction of complex outdoor scenes with sparse viewing angles according to claim 4, characterized in that: The method further comprises the following steps: generating a first target image through a new perspective enhancement module according to the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism: The structural features of the original rendered image are injected into the enhancement branch, specifically: In the self-attention mechanism, the query and key in the reconstruction branch are represented by Q r and K r Indicates; in the query of the enhanced branch, the key and value are represented by Q e ,K e and V e express; By adding Q e and K e Replace with Q r and K r Implement self-attention injection operation; Among them, the expression of the self-attention injection operation is: Among them, the output feature F e quilt Used to predict noise, d represents the vector dimension; After T steps of denoising, a high-quality image is obtained Use AdaIn as post-processing to manipulate the image based on a user-provided reference image The global appearance of To supervise the original rendered image I n .

6. The method for 3D reconstruction of complex outdoor scenes with sparse viewing angles according to claim 1, characterized in that: The method of obtaining a high-fidelity second target image with transient occlusion removed by an occlusion processing module according to the camera pose information and the initial three-dimensional point cloud information comprises the following steps: Using a new strategy, we introduce the pre-trained segmentation network EVF-SAM to obtain the occlusion mask Based on the similarity between the original information in the occluded area and the information in the adjacent area, this similarity is used to fill the occlusion, specifically: Through the diffusion model Intrinsic self-attention mechanism and constrained priors to enhance occluded regions; Given a training viewpoint, render image I t Its supervisory signal Use DDIM inversion to obtain the corresponding Gaussian distribution x ′ T and Use the corresponding mask M i x ′ T and Perform weighted fusion to obtain The expression of this process is: right Perform the reverse diffusion process of the two branches and use the corresponding mask M i Perform weighted self-attention injection, the expression of this process is: Q OH =M i ⊙Q ′ e +(1-M i )⊙Q r ′ , K OH =M i ⊙K e ′ +(1-M i )⊙K r ′ , Among them, the output feature F e ′ quilt Used to predict noise; output feature F e ′ quilt Used to predict noise and output the feature F e ′ As a supervisory signal, optimize the Gaussian radiation field; Q OH represents the query of the enhanced branch after fusion; K OH represents the bond in the enhanced branch after fusion; Q ′ e represents the query of the enhancement branch in the occlusion processing module; K e ′ Represents the key of the enhancement branch in the occlusion processing module; Q r ′ Represents the query of the reconstruction branch in the occlusion processing module; K r ′ Represents the key of the reconstruction branch in the occlusion processing module; V e ′ Represents the value of the enhancement branch in the occlusion processing module.

7. The method for 3D reconstruction of complex outdoor scenes with sparse viewing angles according to claim 1, characterized in that: The method of optimizing the three-dimensional Gaussian radiation field by using a curriculum learning strategy comprises the following steps: First train the Gaussian radiation field with simple samples, then gradually move to more complex samples; Randomly select training perspectives to train the Gaussian radiation field until it fits, and then gradually incorporate the sampled new perspectives into the training process in three stages according to the complexity of the new perspectives; These new perspectives are divided into three levels based on their Euclidean distance from the training perspective: easy, medium, and difficult; New perspectives are selected for training with probability β. Through this curriculum learning strategy, high-quality and multi-perspective consistent Gaussian radiation fields can eventually be obtained.

8. A method for 3D reconstruction of complex outdoor scenes with sparse viewing angles according to any one of claims 1 to 7, characterized in that: The method further comprises the following steps: Rendering image I for the new perspective n , using its pseudo-supervisory signal To conduct supervision: Among them, L1 represents the L1 loss function, L SSIM represents the SSIM loss function; L c represents the loss of luminosity; For the training view rendered image I t , when the iteration number is less than τ, use the corresponding mask M i Mask out transient occlusion; When the iteration number is greater than or equal to τ, the pseudo supervision signal is used Supervision of the entire image: in, L o represents occlusion loss; I t Represents the training viewpoint rendered image; The total loss function is expressed as: L = L o +λ3L c ; Among them, λ1, λ2, λ3, τ are hyperparameters.

9. A 3D reconstruction system for complex outdoor scenes with sparse viewing angles, characterized in that: include: The first module is used to obtain the camera pose information and the initial 3D point cloud information with rich geometric priors through the initialization module; A second module is used to generate a first target image through a new perspective enhancement module according to the camera pose information and the initial three-dimensional point cloud information in combination with a self-attention mechanism, wherein the first target image is used as a pseudo supervision signal to supervise the original rendered image; A third module is used to obtain a high-fidelity second target image with transient occlusion removed through an occlusion processing module according to the camera posture information and the initial three-dimensional point cloud information; The fourth module is used to optimize the three-dimensional Gaussian radiation field according to the first target image and the second target image through a curriculum learning strategy to reconstruct a three-dimensional image of a complex outdoor scene.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images