Three-dimensional reconstruction method and device for resisting transient change interference

Abstract

The invention discloses a transient change interference resistant three-dimensional reconstruction method and device. The method comprises the steps of obtaining a camera pose and a scene sparse point cloud of an initial input image; performing scene reconstruction on the initial input image according to the camera pose and the scene sparse point cloud through a three-dimensional Gaussian splashing strategy of delayed Gaussian growth to obtain a target rendering image; according to the target rendering image and the initial input image, predicting a transient object region in the initial input image through a scale cascade mask guide method to obtain a target transient mask prediction result; and obtaining a three-dimensional reconstruction result according to the target rendering image and the target transient mask prediction result. The method can improve the accuracy of three-dimensional reconstruction, and can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method and device capable of resisting transient change interference. Background Art

[0002] 3D reconstruction from multi-view images is a key research problem in the field of computer vision, which is widely used in urban planning, autonomous driving, virtual reality and other fields. Among them, 3D Gaussian Splatting has become a popular method due to its real-time and realistic rendering. Most current methods assume that the scene conditions are static, but this assumption often does not hold in real scenes with transient objects. This mismatch breaks the multi-view Figure 1 The need for consistency leads to a significant degradation in reconstruction quality. Existing methods design a 2D mask to block transient objects in the image, thereby stopping the normal gradients in these transient regions and avoiding incorrectly optimizing the properties of the 3D representation. However, this meticulous design still has certain limitations, resulting in the resulting 2D mask being unable to accurately detect areas of the image affected by transient objects. Summary of the Invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a three-dimensional reconstruction method and device that is resistant to transient change interference, in order to solve at least one of the problems of the prior art. The present invention can improve the accuracy of three-dimensional reconstruction.

[0004] To achieve the above-mentioned object, an embodiment of the present invention provides a three-dimensional reconstruction method that is resistant to transient change interference, the method comprising the following steps:

[0005] Get the camera pose and scene sparse point cloud of the initial input image;

[0006] Reconstructing the scene on the initial input image according to the camera pose and the scene sparse point cloud by a three-dimensional Gaussian splatting strategy of delayed Gaussian growth to obtain a target rendered image;

[0007] According to the target rendered image and the initial input image, a transient object region in the initial input image is predicted by a scale cascade mask guided method to obtain a target transient mask prediction result;

[0008] A three-dimensional reconstruction result is obtained according to the target rendered image and the target transient mask prediction result.

[0009] In some embodiments, obtaining the camera pose of the initial input image and the scene sparse point cloud comprises the following steps:

[0010] Acquiring the initial input image;

[0011] The initial input image is extracted by using structure-from-motion technology to obtain the camera pose and the scene sparse point cloud.

[0012] In some embodiments, the three-dimensional Gaussian splatting strategy of delayed Gaussian growth is used to reconstruct the scene on the initial input image according to the camera pose and the scene sparse point cloud to obtain a target rendered image, comprising the following steps:

[0013] Initializing an initial three-dimensional Gaussian representation based on the sparse point cloud of the scene;

[0014] Projecting the initial three-dimensional Gaussian representation into a two-dimensional space for rendering to obtain an initial rendered image;

[0015] Obtaining a reconstruction loss according to the initial input image and the initial rendered image;

[0016] Optimizing the initial 3D Gaussian representation according to the reconstruction loss, and returning to the step of projecting the initial 3D Gaussian representation into a 2D space for rendering until the reconstruction loss is less than a first preset error threshold, thereby obtaining an intermediate 3D Gaussian representation;

[0017] A densification operation is performed on the intermediate three-dimensional Gaussian representation to obtain the target rendered image.

[0018] In some embodiments, the initial three-dimensional Gaussian representation is projected into a two-dimensional space for rendering to obtain an initial rendered image, and the formula used includes:

[0019]

[0020] Among them, C p represents the RGB color of pixel p; N represents the total number of initial three-dimensional Gaussian representations that the ray emitted to pixel p passes through in three-dimensional space; c i represents the color of the i-th initial three-dimensional Gaussian representation; α i represents the cumulative transmittance of the i-th initial three-dimensional Gaussian representation; α j represents the cumulative transmittance of the jth initial 3D Gaussian representation.

[0021] In some embodiments, performing a densification operation on the intermediate three-dimensional Gaussian representation to obtain the target rendered image comprises the following steps:

[0022] Determining whether a scale attribute of the intermediate three-dimensional Gaussian representation is greater than a scale threshold;

[0023] When the scale attribute is greater than a scale threshold, performing a splitting operation on the intermediate three-dimensional Gaussian representation;

[0024] When the scale attribute is less than or equal to a scale threshold, a cloning operation is performed on the intermediate three-dimensional Gaussian representation.

[0025] In some embodiments, predicting the transient object region in the initial input image by a scale cascade mask guided method based on the target rendered image and the initial input image to obtain a target transient mask prediction result includes the following steps:

[0026] Constructing an initial multi-layer perceptron network model, and predicting a probability mask of the transient object region by using the initial multi-layer perceptron network model to obtain an initial transient mask prediction result;

[0027] According to the target rendered image, the initial input image, and the initial transient mask prediction result, performing low-resolution and high-resolution optimization supervision on the initial multi-layer perceptron network model to obtain a target multi-layer perceptron network model;

[0028] The probability mask of the transient object area is predicted by the target multi-layer perceptron network model to obtain the target transient mask prediction result.

[0029] In some embodiments, performing low-resolution and high-resolution optimization supervision on the initial multi-layer perceptron network model according to the target rendered image, the initial input image, and the initial transient mask prediction result to obtain a target multi-layer perceptron network model includes the following steps:

[0030] Downsampling the target rendered image and the initial input image respectively to obtain a low-resolution rendered image and a low-resolution input image;

[0031] performing feature extraction on the low-resolution rendered image and the low-resolution input image respectively to obtain a first image feature and a second image feature;

[0032] Obtaining a first multi-layer perceptron loss according to the first image feature, the second image feature, and the initial transient mask prediction result;

[0033] performing low-resolution optimization supervision on the initial multi-layer perceptron network model according to the first multi-layer perceptron loss, and returning to the step of respectively extracting features from the low-resolution rendered image and the low-resolution input image until the first multi-layer perceptron loss is less than a second preset error threshold, thereby obtaining an intermediate multi-layer perceptron network model;

[0034] According to the target rendered image and the initial input image, the intermediate multi-layer perceptron network model is optimized and supervised at high resolution to obtain the target multi-layer perceptron network model.

[0035] In some embodiments, performing high-resolution optimization supervision on the intermediate multi-layer perceptron network model based on the target rendered image and the initial input image to obtain the target multi-layer perceptron network model includes the following steps:

[0036] Predicting a probability mask of the transient object region by using the intermediate multi-layer perceptron network model to obtain an intermediate transient mask prediction result;

[0037] performing feature extraction on the target rendered image and the initial input image respectively to obtain a third image feature and a fourth image feature;

[0038] Obtaining a second multi-layer perceptron loss according to the third image feature, the fourth image feature, and the intermediate transient mask prediction result;

[0039] Based on the second multi-layer perceptron loss, the intermediate multi-layer perceptron network model is optimized and supervised at high resolution, and feature extraction is performed on the target rendered image and the initial input image respectively until the second multi-layer perceptron loss is less than a third preset error threshold, thereby obtaining the target multi-layer perceptron network model.

[0040] To achieve the above-mentioned object, another aspect of an embodiment of the present invention provides a 3D reconstruction device that is resistant to transient change interference, the device comprising:

[0041] The first module is used to obtain the camera pose and scene sparse point cloud of the initial input image;

[0042] A second module is configured to reconstruct the scene on the initial input image according to the camera pose and the scene sparse point cloud by using a three-dimensional Gaussian splatting strategy of delayed Gaussian growth to obtain a target rendered image;

[0043] a third module, configured to predict the transient object region in the initial input image by a scale cascade mask guided method based on the target rendered image and the initial input image, to obtain a target transient mask prediction result;

[0044] The fourth module is used to obtain a three-dimensional reconstruction result according to the target rendered image and the target transient mask prediction result.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the three-dimensional reconstruction method that is resistant to transient change interference as described above.

[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned three-dimensional reconstruction method that is resistant to transient change interference.

[0047] To achieve the above objectives, another aspect of an embodiment of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method for 3D reconstruction with immunity to transient change interference.

[0048] Embodiments of the present invention include at least the following beneficial effects: The present application provides a three-dimensional reconstruction method and apparatus that is resistant to transient change interference. The solution obtains the camera pose and scene sparse point cloud of an initial input image; reconstructs the scene of the initial input image based on the camera pose and the scene sparse point cloud using a three-dimensional Gaussian splashing strategy with delayed Gaussian growth to obtain a target rendered image; predicts the transient object area in the initial input image using a scale cascade mask guidance method based on the target rendered image and the initial input image to obtain a target transient mask prediction result; and obtains a three-dimensional reconstruction result based on the target rendered image and the target transient mask prediction result. The present invention introduces a three-dimensional Gaussian splashing strategy with delayed Gaussian growth, and through the scale cascade mask guidance method, reduces overfitting of transient objects in early optimization, achieves more accurate mask prediction, and thus improves the accuracy of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a flowchart of the steps of a three-dimensional reconstruction method that is resistant to transient change interference provided by an embodiment of the present application;

[0051] Figure 2 This is a flow chart of a three-dimensional reconstruction method that is resistant to transient change interference provided by an embodiment of the present application;

[0052] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this application. They are merely examples of devices and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0054] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0057] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0058] 3D Gaussian Splatting (3DGS) is a technology used for 3D scene representation and rendering. Its core idea is to describe and render the geometric and appearance information in a 3D scene through Gaussian distribution.

[0059] Deep learning, which originated from the study of artificial neural networks, combines low-level features to form more abstract high-level representation attribute categories or features to discover distributed feature representations of data.

[0060] Structure from Motion (SfM) is a computer vision technique used to reconstruct the structure of a 3D scene from a series of 2D images. Its core idea is to infer the 3D geometry of the scene and the camera's motion trajectory by analyzing the motion of objects in the image sequence.

[0061] 3D reconstruction from multi-view images is a key research problem in computer vision, which is widely used in urban planning, autonomous driving, virtual reality and other fields. Among them, 3D Gaussian Splatting has become a popular method due to its real-time performance and realistic rendering.

[0062] However, existing methods usually assume that the scene to be reconstructed is in a completely static state without any interference factors, but this assumption often does not hold true in real scenes with transient objects. Figure 1 The consistency requirement leads to severe degradation in reconstruction quality. Existing methods also include designing two-dimensional image masks through deep learning-based models to shield the impact of transient interference on scene reconstruction. However, these learning-based methods use the photometric residual or feature similarity between the input and rendered images as supervision. In the early stages of reconstruction, the three-dimensional Gaussian representation is not optimized enough, resulting in overly smooth rendering, large photometric residual, and weak feature similarity between transient and static regions. Using these unreliable signals as supervision for mask estimation will lead to inaccurate transient masks. Small masks cannot remove transients and cause artifacts, while overly smooth early reconstructions will misclassify static areas, hinder optimization and lead to insufficient reconstruction.

[0063] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a three-dimensional reconstruction method that is resistant to transient change interference. The method may include but is not limited to steps S100 to S400:

[0064] Step S100, obtaining the camera pose and scene sparse point cloud of the initial input image;

[0065] Step S200, reconstructing the scene on the initial input image according to the camera pose and the scene sparse point cloud by using a three-dimensional Gaussian splatting strategy of delayed Gaussian growth to obtain a target rendered image;

[0066] Step S300 , predicting a transient object region in the initial input image by a scale cascade mask guided method based on the target rendered image and the initial input image, to obtain a target transient mask prediction result;

[0067] Step S400 : obtaining a three-dimensional reconstruction result according to the target rendered image and the target transient mask prediction result.

[0068] In steps S100 to S400 of some embodiments, by integrating a delayed Gaussian growth strategy into the optimization process of three-dimensional Gaussian splashing and designing a scale cascade mask guidance method, combined with the proposed deep learning-based transient mask prediction, accurate three-dimensional reconstruction that is resistant to transient interference is achieved.

[0069] In some embodiments, step S100 may include but is not limited to steps S110 to S120:

[0070] Step S110, collecting the initial input image;

[0071] Step S120 : extracting the initial input image by using structure-from-motion technology to obtain the camera pose and the scene sparse point cloud.

[0072] In some embodiments, steps S110 to S120 capture an initial input image and extract it using Structure from Motion (SfM) to obtain a camera pose and a sparse point cloud of the scene. Alternatively, a camera can capture a set of input images from multiple perspectives, where transient objects such as pedestrians and vehicles may be present, and use Structure from Motion to obtain the camera pose and sparse point cloud corresponding to the initial input image.

[0073] In step S200 of some embodiments, a delayed Gaussian growing strategy is combined with an optimized mask learning strategy to perform scene reconstruction using 3D Gaussian splatting. In scene reconstruction based on the delayed Gaussian growing 3D Gaussian splatting strategy, disabling Gaussian densification significantly improves the learning of low-frequency static components while maintaining the original number of Gaussians, helping the reconstruction model better capture static elements before adapting to transient elements.

[0074] In some embodiments, step S200 may include but is not limited to steps S210 to S250:

[0075] Step S210 , initializing an initial three-dimensional Gaussian representation according to the scene sparse point cloud;

[0076] Step S220, projecting the initial three-dimensional Gaussian representation into a two-dimensional space for rendering to obtain an initial rendered image;

[0077] Step S230, obtaining a reconstruction loss according to the initial input image and the initial rendered image;

[0078] Step S240: Optimizing the initial 3D Gaussian representation according to the reconstruction loss, and returning to the step of projecting the initial 3D Gaussian representation into a 2D space for rendering, until the reconstruction loss is less than a first preset error threshold, thereby obtaining an intermediate 3D Gaussian representation.

[0079] Step S250 : performing a densification operation on the intermediate three-dimensional Gaussian representation to obtain the target rendered image.

[0080] In step S210 of some embodiments, the three-dimensional Gaussian representation is initialized according to the input scene sparse point cloud to obtain an initialized initial three-dimensional Gaussian representation, wherein the three-dimensional Gaussian representation includes attributes such as color, position, scale, rotation, and opacity.

[0081] In step S220 of some embodiments, the initial input image is rasterized, that is, a 3D Gaussian representation in 3D space is projected into 2D space for rendering using a 3D Gaussian splatting strategy with delayed Gaussian growth. For any pixel in the initial input image, the color of the rendered image is calculated using α-blending, as shown in the following formula:

[0082]

[0083] Among them, C p represents the RGB color obtained by rendering pixel p; N represents the total number of initial three-dimensional Gaussian representations that the ray emitted from the camera center to pixel p passes through in three-dimensional space; c i represents the color of the i-th initial three-dimensional Gaussian representation; α i represents the cumulative transmittance of the i-th initial three-dimensional Gaussian representation; α j represents the cumulative transmittance of the jth initial three-dimensional Gaussian representation before the i-th initial three-dimensional Gaussian representation.

[0084] In steps S230 to S240 of some embodiments, the L1 loss and D-SSIM loss between the initial input image and the initial rendered image are obtained, and then various attributes of the initial three-dimensional Gaussian representation are optimized, which is expressed as follows:

[0085] L=(1-λ)L1+λL D-SSIM

[0086] Among them, L represents the reconstruction loss; L1 represents the L1 loss between the initial input image and the initial rendered image; L D-MSIM represents the D-SSIM loss of the initial input image and the initial rendered image, which is the inverse of the structural similarity index (SSIM); λ represents the weight hyperparameter for balancing the two losses. Optionally, the weight hyperparameter can be set to 0.2.

[0087] In some embodiments, the L1 loss between the initial input image and the initial rendered image is calculated as:

[0088]

[0089] The calculation formula of D-SSIM loss of initial input image and initial rendered image is:

[0090]

[0091] Where I represents the initial input image; Represents the initial rendered image; Represents the structural similarity index between the initial input image and the initial rendered image.

[0092] In the scene reconstruction of steps S210 to S240 of some embodiments, the number of initial three-dimensional Gaussian representations is maintained to be the same as that of the input scene sparse point cloud, and no densification operation is performed on it, that is, delayed Gaussian growth, and only the properties of the initial three-dimensional Gaussian representation are optimized, which helps to preferentially fit the static areas of the scene under the controlled number of Gaussians. After several cycles of rasterizing the initial input image and optimizing the various properties of the initial three-dimensional Gaussian representation, if the reconstruction loss L is less than a customized first preset error threshold, the intermediate three-dimensional Gaussian representation after target optimization can be obtained. Optionally, the number of cycles can be set to 10,000 times.

[0093] In step S250 of some embodiments, after the delayed Gaussian growing process is completed, in order to represent the details of the scene, the number of Gaussians needs to be increased to fit the high-frequency detail areas, that is, the intermediate three-dimensional Gaussian representation needs to be densified. This process includes two different operations: cloning and splitting. After the densification operation, the scene reconstruction has been preliminarily optimized and has good initial quality, which can more accurately reflect the overall geometric structure of the scene. At the same time, the subsequent mask prediction can more accurately identify transient objects, effectively avoiding the ambiguity of mask prediction caused by insufficient optimization of the scene in the early reconstruction stage.

[0094] In some embodiments, step S250 may include but is not limited to steps S251 to S253:

[0095] Step S251, determining whether the scale attribute of the intermediate three-dimensional Gaussian representation is greater than a scale threshold;

[0096] Step S252: when the scale attribute is greater than a scale threshold, performing a splitting operation on the intermediate three-dimensional Gaussian representation;

[0097] Step S253: When the scale attribute is less than or equal to a scale threshold, a cloning operation is performed on the intermediate three-dimensional Gaussian representation.

[0098] In step S251 of some embodiments, it is determined whether the scale attribute of the intermediate three-dimensional Gaussian representation is greater than a scale threshold. Optionally, the scale threshold is 0.01 multiplied by the scene scale, and the scene scale is modulated according to the input camera pose.

[0099] In step S252 of some embodiments, a larger intermediate three-dimensional Gaussian representation, that is, a scale attribute of the intermediate three-dimensional Gaussian greater than a scale threshold, is generally considered to be an over-reconstructed area, and therefore a splitting operation is required to divide it into two, thereby improving the ability to represent the detail area.

[0100] In step S253 of some embodiments, a smaller intermediate three-dimensional Gaussian representation, that is, the scale attribute of the intermediate three-dimensional Gaussian is less than or equal to the scale threshold, is generally considered to be an under-reconstructed area, and therefore a cloning operation is required to clone it into another identical Gaussian, thereby improving the ability to represent the detail area.

[0101] In step S300 of some embodiments, in order to identify the area where transient objects appear in the image, a deep learning-based method is used to predict the two-dimensional image mask of the transient object area. Optionally, a scale cascade mask guidance method is combined. First, the similarity of low-resolution features is used for supervision to achieve a reliable initial transient mask estimation. Low-resolution features have stronger semantic consistency and robustness to noise, which helps to obtain more stable mask prediction results in the early stages. Subsequently, a gradual transition to high-resolution supervision is made to further improve the accuracy of mask prediction and ensure that the final transient mask can more accurately reflect the dynamic object area in the scene.

[0102] In some embodiments, step S300 may include but is not limited to steps S310 to S330:

[0103] Step S310, constructing an initial multi-layer perceptron network model, and predicting the probability mask of the transient object area by using the initial multi-layer perceptron network model to obtain an initial transient mask prediction result;

[0104] Step S320, performing low-resolution and high-resolution optimization supervision on the initial multi-layer perceptron network model according to the target rendered image, the initial input image, and the initial transient mask prediction result, to obtain a target multi-layer perceptron network model;

[0105] Step S330 , predicting the probability mask of the transient object area through the target multi-layer perceptron network model to obtain the target transient mask prediction result.

[0106] In step S310 of some embodiments, an initial multi-layer perceptron (MLP) network model is constructed. The initial multi-layer perceptron network model is used to predict the probability mask of the transient object region, thereby obtaining an initial transient mask prediction result. Optionally, the MLP model used includes two linear layers, the middle layer uses a ReLU activation function, and the last layer can output a prediction value between 0 and 1 using a sigmoid function, indicating the probability that the region is a transient object. 0 indicates a completely transient object, while 1 indicates a completely static region.

[0107] In step S320 of some embodiments, the optimization of the MLP model weights requires appropriate supervision. In order to optimize the MLP model so that it can accurately identify which pixels in the image belong to the transient object area, a two-part supervision strategy is adopted. Optionally, the first part of the supervision introduces an image robust loss based on image residual information; the second part of the supervision adopts a feature robust loss to optimize the model by utilizing the feature similarity information between the rendered image and the input image. Exemplarily, based on the target rendered image, the initial input image, and the initial transient mask prediction results, the initial multi-layer perceptron network model is optimized and supervised from low resolution to high resolution, thereby obtaining the target multi-layer perceptron network model.

[0108] In some embodiments, step S320 may include but is not limited to steps S321 to S325:

[0109] Step S321 , downsampling the target rendered image and the initial input image respectively to obtain a low-resolution rendered image and a low-resolution input image;

[0110] Step S322, performing feature extraction on the low-resolution rendered image and the low-resolution input image respectively to obtain a first image feature and a second image feature;

[0111] Step S323: obtaining a first multi-layer perceptron loss according to the first image feature, the second image feature, and the initial transient mask prediction result;

[0112] Step S324: performing low-resolution optimization supervision on the initial multi-layer perceptron network model according to the first multi-layer perceptron loss, and returning to the step of performing feature extraction on the low-resolution rendered image and the low-resolution input image respectively until the first multi-layer perceptron loss is less than a second preset error threshold, thereby obtaining an intermediate multi-layer perceptron network model.

[0113] Step S325 , performing high-resolution optimization supervision on the intermediate multi-layer perceptron network model according to the target rendered image and the initial input image to obtain the target multi-layer perceptron network model.

[0114] In some embodiments, in steps S321 to S322, the input image is downsampled to an image with a lower resolution than the original resolution used for reconstruction, and a low-resolution rendered image with a lower resolution than the image used for reconstruction is rendered from the three-dimensional Gaussian splash, and image features corresponding to the low-resolution input image and the low-resolution rendered image are extracted respectively. Exemplarily, the target rendered image and the initial input image are first downsampled to a lower resolution, and the low-resolution rendered image and the low-resolution input image obtained after downsampling are used to extract image features at low resolution, thereby obtaining first image features of the low-resolution rendered image and second image features of the low-resolution input image. Optionally, the initial input image and the target rendered image are downsampled to 1 / 4 of the original resolution; the pre-trained weights of the large visual model DINOv2 are loaded, and the low-resolution rendered image and the low-resolution input image are used as input to extract image features at low resolution.

[0115] In step S323 of some embodiments, the first multi-layer perceptron loss is obtained by calculating the loss of two parts. In the first part, based on the low-resolution rendered image and the corresponding low-resolution input image, the image robust loss based on the image residual information at low resolution is calculated, which is as follows:

[0116] L residual,low =max((U 1,low -M t,low ),0)+max((M t,low -U 2,low ),0)

[0117] Among them, L residual,low represents the image robust loss based on image residual information at low resolution; M t,low represents the initial transient mask prediction result; U 1,low 、U 2,low Represents the residual information of the binarized image calculated according to different thresholds.

[0118] In the second part, based on the image features corresponding to the low-resolution input image and the low-resolution rendered image, the low-resolution feature robust loss is calculated, which has the following formula:

[0119] M cos,low =max(2cos(f t,low ,f′ t,low )-1,0)

[0120] L cos,low =||M t,low -M cos,low ||

[0121] Among them, Mcos,low represents the first feature cosine similarity supervision mask; f t,low represents the image feature corresponding to the low-resolution input image, that is, the second image feature; f′ t,low represents the image feature corresponding to the low-resolution rendered image, that is, the first image feature; L cos,low Represents a low-resolution feature robust loss. Optionally, the first feature cosine similarity supervision mask is a feature cosine similarity supervision mask calculated based on the first image feature and the second image feature, and the mask is normalized, its value range is limited to between 0 and 1, and truncated, where 0 represents the least similarity and 1 represents the most similarity.

[0122] In step S324 of some embodiments, the MLP model for supervised mask prediction is optimized by calculating two losses at low resolution (i.e., the image robust loss based on image residual information at low resolution and the low-resolution feature robust loss), so that the first multi-layer perceptron loss is less than a preset second predetermined error threshold, thereby obtaining an intermediate multi-layer perceptron network model. The overall MLP model optimization calculation formula is as follows:

[0123] L MLP,low =λ residual,low L residual,low +λ cos,low L cos,low

[0124] Among them, L MLP,loe represents the first multi-layer perceptron loss; λ residual,low ,λ cos,low represents a fixed weight.

[0125] In step S325 of some embodiments, after a period of low-resolution optimization supervision, the supervision of the MLP model is switched from the low-resolution image and its features to the original resolution image and its features, gradually transitioning from low-resolution to high-resolution optimization supervision, thereby ensuring fine-grained distinction between transient and static areas.

[0126] In some embodiments, step S325 may include but is not limited to steps S3251 to S3254:

[0127] Step S3251, predicting the probability mask of the transient object area through the intermediate multi-layer perceptron network model to obtain an intermediate transient mask prediction result;

[0128] Step S3252: performing feature extraction on the target rendered image and the initial input image respectively to obtain a third image feature and a fourth image feature;

[0129] Step S3253: Obtain a second multi-layer perceptron loss based on the third image feature, the fourth image feature, and the intermediate transient mask prediction result;

[0130] Step S3254: Based on the second multi-layer perceptron loss, the intermediate multi-layer perceptron network model is optimized and supervised at high resolution, and feature extraction is performed on the target rendered image and the initial input image respectively, until the second multi-layer perceptron loss is less than a third preset error threshold, thereby obtaining the target multi-layer perceptron network model.

[0131] In some embodiments, in steps S3251 and S3252, the intermediate multilayer perceptron network model is used to predict the probability mask of the transient object region, thereby obtaining an intermediate transient mask prediction result. Image features corresponding to the target rendered image and the initial input image are then extracted, respectively, to obtain a third image feature of the target rendered image and a fourth image feature of the initial input image. Alternatively, the large visual model DINOv2 is used, loaded with its pre-trained weights, and the target rendered image and the initial input image are used as input to extract image features corresponding to each image.

[0132] In step S3253 of some embodiments, a second multi-layer perceptron loss is obtained through two losses. In the first part, based on the original resolution rendered image (i.e., the target rendered image) and the corresponding original resolution input image (i.e., the initial input image), an image robust loss and a feature robust loss based on image residual information are calculated to further optimize the MLP model for mask prediction, which is as follows:

[0133] L residual,high =max((U 1,high -M t,high ),0)+max((M t,high -U 2,high ),0)

[0134] Among them, L residual,high represents the image robust loss based on image residual information at high resolution; M t,high Indicates the intermediate transient mask prediction result; U 1,high 、U 2,high Represents the residual information of the binarized image calculated according to different thresholds.

[0135] In the second part, the high-resolution feature robust loss is calculated based on the image features corresponding to the original resolution rendered image and the corresponding original resolution input image. The formula is as follows:

[0136] M cos,high =max(2cos(f t,high ,f′ t,high )-1,0)

[0137] L cos,high =||M t,high -M cos,high ||

[0138] Among them, M cos,high represents the second feature cosine similarity supervision mask; f t,high represents the image feature corresponding to the target rendered image, i.e., the third image feature; f′ t,high represents the image feature corresponding to the initial input image, i.e., the fourth image feature; L cos,high Represents a high-resolution feature robust loss. Optionally, the second feature cosine similarity supervision mask is a feature cosine similarity supervision mask calculated based on the third image feature and the fourth image feature, and the mask is normalized, its value range is limited to between 0 and 1, and truncated, where 0 represents the least similarity and 1 represents the most similarity.

[0139] In step S3254 of some embodiments, the MLP model for supervised mask prediction is optimized by calculating two losses at the original resolution (i.e., the image robust loss based on image residual information at high resolution and the high-resolution feature robust loss), so that the second multi-layer perceptron loss is less than a preset third predetermined error threshold, thereby obtaining a target multi-layer perceptron network model. The overall MLP model optimization calculation formula is as follows:

[0140] L MLP,high =λ residual,high L residual,high +λ cos,high L cos,high

[0141] Among them, L MLP,low represents the second multi-layer perceptron loss; λ residual,high ,λ cos,high represents a fixed weight.

[0142] In step S330 of some embodiments, after low-resolution and high-resolution optimization supervision, the probability mask of the transient object area is predicted by the target multi-layer perceptron network model, and the target transient mask prediction result can be obtained to guide further optimization of scene reconstruction.

[0143] In step S400 of some embodiments, a three-dimensional reconstruction result may be obtained according to the target rendered image and the target transient mask prediction result.

[0144] like Figure 2 As shown, taking the 3D reconstruction of a scene randomly photographed in a real outdoor environment as an example, the 3D reconstruction method of the embodiment of the present invention may include the following steps:

[0145] Step 1: A set of multi-view input images is randomly collected through the camera. The input images may contain transient objects such as pedestrians and vehicles.

[0146] Step 2: Use structure-from-motion technology to obtain the camera pose and scene sparse point cloud corresponding to each input image.

[0147] Step 3: Through 3D Gaussian splatting with delayed Gaussian growth strategy, the input image is reconstructed according to the camera pose and the scene sparse point cloud to obtain the rendered image.

[0148] Step 4: Using the scale cascade mask guidance method, the deep learning model used for transient mask estimation (such as the MLP model) is optimized from low resolution to high resolution for supervision, and the transient object area in the image is predicted by the optimized supervised deep learning model to obtain the transient mask prediction result, thereby realizing three-dimensional reconstruction that is resistant to transient change interference.

[0149] An embodiment of the present invention further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the aforementioned method for 3D reconstruction that resists transient interference. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0150] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] refer to Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0152] The processor 501 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0153] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called by the processor 501 to execute the three-dimensional reconstruction method for resisting transient change interference in the embodiments of this application.

[0154] Input / output interface 503, used to implement information input and output;

[0155] Communication interface 504, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0156] Bus 505 , which transmits information between various components of the device (e.g., processor 501 , memory 502 , input / output interface 503 , and communication interface 504 );

[0157] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via a bus 505 .

[0158] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned three-dimensional reconstruction method that is resistant to transient change interference.

[0159] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0160] An embodiment of the present invention further provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method for 3D reconstruction with immunity to transient change interference.

[0161] In summary, the present invention prioritizes static scene optimization by integrating a delayed Gaussian growth strategy into three-dimensional Gaussian splattering, and designs a scale cascade mask guidance method to reliably suppress transient objects. It can completely and accurately predict transient object masks, thereby reducing the impact of transient objects on scene reconstruction and achieving three-dimensional reconstruction without transient interference.

[0162] A 3D reconstruction method and apparatus resistant to transient change interference according to an embodiment of the present invention has the following advantages:

[0163] 1. The embodiment of the present invention introduces a delayed Gaussian growing strategy, which prioritizes optimizing static scene structures before allowing the number of Gaussians to grow, thereby alleviating overfitting of transient objects in early optimization.

[0164] 2. The embodiment of the present invention designs a scale-cascade mask guidance method, which first uses low-resolution feature similarity supervision to perform reliable initial transient mask estimation, takes advantage of its stronger semantic consistency and robustness to noise, and then develops to high-resolution supervision to achieve more accurate mask prediction.

[0165] 3. The embodiment of the present invention does not require any additional pre-processing and post-processing modules. The proposed method and the models and features used are lightweight and achieve highly efficient three-dimensional reconstruction.

[0166] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0167] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0168] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0169] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0170] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0171] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0172] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0173] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0174] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A three-dimensional reconstruction method that is resistant to transient change interference, characterized in that: The following steps are involved: Get the camera pose and scene sparse point cloud of the initial input image; Reconstructing the scene on the initial input image according to the camera pose and the scene sparse point cloud by a three-dimensional Gaussian splatting strategy of delayed Gaussian growth to obtain a target rendered image; According to the target rendered image and the initial input image, a transient object region in the initial input image is predicted by a scale cascade mask guided method to obtain a target transient mask prediction result; A three-dimensional reconstruction result is obtained according to the target rendered image and the target transient mask prediction result.

2. The method for 3D reconstruction resistant to transient change interference according to claim 1, characterized in that: The step of obtaining the camera pose of the initial input image and the scene sparse point cloud comprises the following steps: Acquiring the initial input image; The initial input image is extracted by using structure-from-motion technology to obtain the camera pose and the scene sparse point cloud.

3. The three-dimensional reconstruction method for resisting transient change interference according to claim 1, characterized in that: The method of reconstructing the scene of the initial input image by using a three-dimensional Gaussian splattering strategy with delayed Gaussian growth according to the camera pose and the sparse point cloud of the scene to obtain a target rendered image includes the following steps: Initializing an initial three-dimensional Gaussian representation based on the sparse point cloud of the scene; Projecting the initial three-dimensional Gaussian representation into a two-dimensional space for rendering to obtain an initial rendered image; Obtaining a reconstruction loss according to the initial input image and the initial rendered image; Optimizing the initial 3D Gaussian representation according to the reconstruction loss, and returning to the step of projecting the initial 3D Gaussian representation into a 2D space for rendering until the reconstruction loss is less than a first preset error threshold, thereby obtaining an intermediate 3D Gaussian representation; A densification operation is performed on the intermediate three-dimensional Gaussian representation to obtain the target rendered image.

4. The method for 3D reconstruction resistant to transient change interference according to claim 3, characterized in that: The initial three-dimensional Gaussian representation is projected into a two-dimensional space for rendering to obtain an initial rendered image, and the formula used includes: Among them, C p represents the RGB color of pixel p; N represents the total number of initial three-dimensional Gaussian representations that the ray emitted to pixel p passes through in three-dimensional space; c i represents the color of the i-th initial three-dimensional Gaussian representation; α i represents the cumulative transmittance of the i-th initial three-dimensional Gaussian representation; α j represents the cumulative transmittance of the jth initial 3D Gaussian representation.

5. The three-dimensional reconstruction method for resisting transient change interference according to claim 3, characterized in that: The densification operation is performed on the intermediate three-dimensional Gaussian representation to obtain the target rendered image, comprising the following steps: Determining whether a scale attribute of the intermediate three-dimensional Gaussian representation is greater than a scale threshold; When the scale attribute is greater than a scale threshold, performing a splitting operation on the intermediate three-dimensional Gaussian representation; When the scale attribute is less than or equal to a scale threshold, a cloning operation is performed on the intermediate three-dimensional Gaussian representation.

6. The three-dimensional reconstruction method for resisting transient change interference according to claim 1, characterized in that: The method of predicting the transient object region in the initial input image by using a scale cascade mask guidance method based on the target rendered image and the initial input image to obtain a target transient mask prediction result comprises the following steps: Constructing an initial multi-layer perceptron network model, and predicting a probability mask of the transient object region by using the initial multi-layer perceptron network model to obtain an initial transient mask prediction result; According to the target rendered image, the initial input image, and the initial transient mask prediction result, performing low-resolution and high-resolution optimization supervision on the initial multi-layer perceptron network model to obtain a target multi-layer perceptron network model; The probability mask of the transient object area is predicted by the target multi-layer perceptron network model to obtain the target transient mask prediction result.

7. The method for 3D reconstruction resistant to transient change interference according to claim 6, characterized in that: The step of performing low-resolution and high-resolution optimization supervision on the initial multi-layer perceptron network model according to the target rendered image, the initial input image, and the initial transient mask prediction result to obtain a target multi-layer perceptron network model comprises the following steps: Downsampling the target rendered image and the initial input image respectively to obtain a low-resolution rendered image and a low-resolution input image; performing feature extraction on the low-resolution rendered image and the low-resolution input image respectively to obtain a first image feature and a second image feature; Obtaining a first multi-layer perceptron loss according to the first image feature, the second image feature, and the initial transient mask prediction result; performing low-resolution optimization supervision on the initial multi-layer perceptron network model according to the first multi-layer perceptron loss, and returning to the step of respectively extracting features from the low-resolution rendered image and the low-resolution input image until the first multi-layer perceptron loss is less than a second preset error threshold, thereby obtaining an intermediate multi-layer perceptron network model; According to the target rendered image and the initial input image, the intermediate multi-layer perceptron network model is optimized and supervised at high resolution to obtain the target multi-layer perceptron network model.

8. The method for 3D reconstruction resistant to transient change interference according to claim 7, characterized in that: The step of performing high-resolution optimization supervision on the intermediate multi-layer perceptron network model according to the target rendered image and the initial input image to obtain the target multi-layer perceptron network model comprises the following steps: Predicting a probability mask of the transient object region by using the intermediate multi-layer perceptron network model to obtain an intermediate transient mask prediction result; performing feature extraction on the target rendered image and the initial input image respectively to obtain a third image feature and a fourth image feature; Obtaining a second multi-layer perceptron loss according to the third image feature, the fourth image feature, and the intermediate transient mask prediction result; Based on the second multi-layer perceptron loss, the intermediate multi-layer perceptron network model is optimized and supervised at high resolution, and feature extraction is performed on the target rendered image and the initial input image respectively until the second multi-layer perceptron loss is less than a third preset error threshold, thereby obtaining the target multi-layer perceptron network model.

9. A three-dimensional reconstruction device resistant to transient change interference, characterized in that: include: The first module is used to obtain the camera pose and scene sparse point cloud of the initial input image; A second module is configured to reconstruct the scene on the initial input image according to the camera pose and the scene sparse point cloud by using a three-dimensional Gaussian splatting strategy of delayed Gaussian growth to obtain a target rendered image; a third module, configured to predict the transient object region in the initial input image by a scale cascade mask guided method based on the target rendered image and the initial input image, to obtain a target transient mask prediction result; The fourth module is used to obtain a three-dimensional reconstruction result according to the target rendered image and the target transient mask prediction result.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images