Image-driven three-dimensional scene reconstruction method and apparatus without sfm initialization

Abstract

The application discloses an image-driven three-dimensional scene reconstruction method and device without SfM initialization. The method comprises the following steps: acquiring multi-view images; performing sparse matching on the multi-view images, and distinguishing between successfully matched point pairs and unmatched points; performing lightweight triangulation on the matched area to obtain the three-dimensional spatial position coordinates of each feature point; performing monocular depth estimation on the unmatched area to obtain the three-dimensional spatial position coordinates of each feature point; constructing a Gaussian point set based on the three-dimensional spatial position coordinates; performing differentiable rendering based on the Gaussian point set to generate simulated images under each view angle; constructing a total loss function based on the simulated images and real images, and jointly optimizing the Gaussian point parameters of the Gaussian point set based on the total loss function to obtain a final Gaussian model. The application can realize realistic rendering of images of any view angle, improve the robustness and adaptability of three-dimensional modeling in complex scenes such as weak features and no texture, and does not require SfM initialization.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D reconstruction technology, and in particular to an image-driven 3D scene reconstruction method, apparatus, storage medium and electronic device that does not require SfM initialization. Background Technology

[0002] In 3D reconstruction tasks, the traditional Structure-from-Motion (SfM) method based on sparse point clouds recovers camera pose and 3D structure through feature matching and geometric estimation of multi-view images. However, in practical applications, due to problems such as low texture, repetitive patterns, uneven lighting, and motion blur in images, the SfM method often cannot reliably obtain high-quality initial point clouds. It heavily relies on the accuracy of feature extraction and matching, limiting its versatility and robustness in real-world complex scenes. For example, in typical reconstruction systems such as COLMAP, if the image set contains weak feature regions such as reflective glass or solid-color walls, the SfM point cloud will be severely degraded, resulting in voids in the subsequent Gaussian modeling point cloud structure.

[0003] In recent years, 3D Gaussian Splatting has emerged as a new 3D reconstruction technique. It utilizes a Gaussian distribution to model scenes and achieves the mapping from point clouds to images through differentiable rendering. This method boasts advantages such as high rendering efficiency, strong optimizability, and excellent reconstruction results. It has been widely applied in fields such as virtual reality, cultural heritage digitization, robot vision, and the film industry.

[0004] However, current mainstream Gaussian sputtering modeling methods still suffer from strong dependence on SfM results and poor performance in weak feature regions. Summary of the Invention

[0005] This application provides an image-driven 3D scene reconstruction method, apparatus, storage medium, and electronic device that does not require SfM initialization. It can achieve realistic rendering of images from any viewpoint, improve the robustness and adaptability of 3D modeling in complex scenes with weak features and no texture, and does not require SfM initialization.

[0006] This application provides an image-driven 3D scene reconstruction method that does not require SfM initialization, including:

[0007] Acquire multi-view images;

[0008] Sparse matching is performed on the multi-view image to distinguish between successfully matched point pairs and unmatched points;

[0009] Lightweight triangulation is performed on the matching regions corresponding to the successfully matched points to obtain the three-dimensional spatial coordinates of each feature point; monocular depth estimation is performed on the unmatched regions corresponding to the unmatched points to obtain the three-dimensional spatial coordinates of each feature point.

[0010] Construct a set of Gaussian points based on the three-dimensional spatial coordinates;

[0011] Differentiable rendering is performed based on the Gaussian point set to generate simulated images from various viewpoints;

[0012] A total loss function is constructed based on the simulated image and the real image. The Gaussian point parameters of the Gaussian point set are jointly optimized based on the total loss function to obtain the final Gaussian model.

[0013] Furthermore, according to the above-described image-driven 3D scene reconstruction method without SfM initialization, lightweight triangulation is performed on the matching region corresponding to the successfully matched point pairs to obtain the 3D spatial coordinates of each feature point, including:

[0014] Calculate the fundamental matrix based on the successfully matched point pairs;

[0015] The essential matrix is ​​calculated based on the aforementioned fundamental matrix and camera intrinsic parameters;

[0016] The camera extrinsic parameters are recovered based on the fundamental matrix and the essential matrix;

[0017] Triangulation based on camera extrinsic parameters yields the three-dimensional spatial coordinates of each feature point.

[0018] Furthermore, according to the above-described image-driven 3D scene reconstruction method without SfM initialization, the method involves performing monocular depth estimation on the unmatched regions corresponding to the unmatched points to obtain the 3D spatial coordinates of each feature point, including:

[0019] The salient regions are extracted from the unmatched regions using a saliency detection algorithm;

[0020] The relative depth of each pixel within the salient region is predicted using a monocular depth estimation network;

[0021] The relative depth is normalized to obtain a normalized depth value;

[0022] The center point of the salient region is combined with the corresponding normalized depth value to generate three-dimensional spatial coordinates, and then assigned covariance matrix, pixel color, and transparency attributes.

[0023] Furthermore, according to the above-described image-driven 3D scene reconstruction method without SfM initialization, constructing a Gaussian point set based on the 3D spatial location coordinates includes:

[0024] Based on the pixel color of the multi-view image, the three-dimensional spatial coordinates of each feature point, the covariance matrix, and the transparency attribute, a Gaussian point set is constructed and initialized.

[0025] Furthermore, according to the above-described image-driven 3D scene reconstruction method without SfM initialization, the process of generating simulated images from various viewpoints by performing differentiable rendering based on the Gaussian point set includes:

[0026] Projecting the points in the Gaussian point set onto a two-dimensional image yields a two-dimensional Gaussian ellipse;

[0027] Collect two-dimensional Gaussian ellipses within the coverage area of ​​each pixel and perform weighted calculations based on the transparency attribute and the covariance matrix;

[0028] Use volume rendering formulas to process multiple Gaussian ellipses. - Blend to obtain the color value of the pixel;

[0029] The color values ​​of the pixels are combined to obtain a rendered image sequence, and the rendered image sequence is determined as the simulated image.

[0030] Furthermore, according to the above-described image-driven 3D scene reconstruction method without SfM initialization, the total loss function includes reconstruction error and a regularization term, and the total loss function is:

[0031]

[0032] in, For the total loss function, Indicates differentiable rendering. Let be the set of Gaussian points. For image Viewpoint parameters, For real images, These are the weighting coefficients. is the regularization term, and N is the number of training images.

[0033] Furthermore, according to the above-described image-driven 3D scene reconstruction method without SfM initialization, the regularization term is used to constrain the geometric consistency and control the covariance between Gaussian points. Expanding the regularization term yields:

[0034]

[0035] in, This represents the connection weight between Gaussian point pairs. It is the Frobenius norm. To adjust the parameters, Represents the three-dimensional spatial coordinates of different points.

[0036] Furthermore, according to the above image-driven 3D scene reconstruction method without SfM initialization, the regularization term is also used to constrain the spatial consistency and control sparsity between Gaussian points. The total loss function after expanding the regularization term is:

[0037]

[0038] in, To simulate an image, For real images, , These are the weighting coefficients. Let i be the connection weight between Gaussian points i and k. Let Gaussian point be its three-dimensional spatial coordinates. This is the transparency attribute.

[0039] This application also provides an image-driven 3D scene reconstruction device that does not require SfM initialization, including:

[0040] The acquisition module is used to acquire images from multiple perspectives;

[0041] The sparse matching module is used to perform sparse matching on the multi-view image and distinguish between successfully matched point pairs and unmatched points.

[0042] The three-dimensional spatial coordinate calculation module is used to perform lightweight triangulation on the matching region corresponding to the successfully matched point pair to obtain the three-dimensional spatial position coordinates of each feature point; and to perform monocular depth estimation on the unmatched region corresponding to the unmatched point to obtain the three-dimensional spatial position coordinates of each feature point.

[0043] A Gaussian point set construction module is used to construct a Gaussian point set based on the three-dimensional spatial position coordinates;

[0044] Differentiable rendering module, used to perform differentiable rendering based on the Gaussian point set to generate simulated images from various viewpoints;

[0045] The optimization module is used to construct a total loss function based on the simulated image and the real image, and to jointly optimize the Gaussian point parameters of the Gaussian point set based on the total loss function to obtain the final Gaussian model.

[0046] This application also provides a computer-readable storage medium storing a plurality of instructions adapted to be loaded by a processor to execute any of the above-described image-driven 3D scene reconstruction methods that do not require SfM initialization.

[0047] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the image-driven 3D scene reconstruction method without SfM initialization described above.

[0048] This application provides an image-driven 3D scene reconstruction method, apparatus, storage medium, and electronic device that does not require SfM initialization. The invention extracts salient information from multi-view images and initializes Gaussian point sets through monocular depth estimation or sparse matching, avoiding the strong dependence on initial point clouds and camera parameters in traditional SfM methods, thus improving adaptability to unstructured, low-texture scenes. The invention also introduces Gaussian point covariance initialization and regularization control mechanisms. Without relying on existing point cloud structures, it guides the structure to converge naturally using point-to-point geometric consistency constraints and sparsity penalty terms, ultimately outputting a clear and detailed 3D model. Furthermore, the invention uses an end-to-end differentiable optimization framework to directly feed rendering errors back to Gaussian point attribute updates, making the entire reconstruction process efficient and controllable, avoiding error accumulation caused by multi-stage dependencies, and improving modeling stability and rendering quality. Attached Figure Description

[0049] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0050] Figure 1 A flowchart of an image-driven 3D scene reconstruction method without SfM initialization provided in this application embodiment.

[0051] Figure 2 A schematic diagram of the structure of the image-driven 3D scene reconstruction device that does not require SfM initialization, provided in the embodiments of this application.

[0052] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Current mainstream Gaussian sputtering modeling methods still have the following shortcomings:

[0055] (1) Strong dependence on SfM results: The initialization of Gaussian points often depends on the sparse point cloud and camera extrinsic parameters provided by SfM or MVS (multi-view stereo matching). If SfM fails, the entire modeling process will be difficult to carry out.

[0056] (2) Poor performance in weak feature regions: In textureless, reflective or unstructured scenes, traditional feature matching algorithms are difficult to output stable matching results, resulting in missing Gaussian points or uneven density distribution;

[0057] (3) The modeling process is cumbersome and the modules are highly coupled: the current method often relies on multiple toolchains to complete image preprocessing, SfM point cloud reconstruction, parameter parsing and Gaussian initialization, resulting in a complex overall modeling process with poor scalability;

[0058] (4) Lack of image content-driven modeling mechanism: Existing methods mostly adopt a geometry-dominated reconstruction approach, which fails to fully explore image semantics, saliency or depth prior information, making it difficult to adapt to end-to-end learning paradigms or real-time application scenarios.

[0059] To address the aforementioned issues, embodiments of this application provide an image-driven 3D scene reconstruction method, apparatus, storage medium, and electronic device that does not require SfM initialization. The image-driven 3D scene reconstruction apparatus that does not require SfM initialization provided in this application can be integrated into an electronic device, such as a terminal, server, or other similar device. The terminal may include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0060] Please see Figure 1 , Figure 1 The flowchart illustrates an image-driven 3D scene reconstruction method without SfM initialization, provided in an embodiment of this application. This method, applied in an electronic device, includes the following steps:

[0061] S1, acquire multi-view images.

[0062] First, the target scene is captured using an image acquisition device (such as a camera) to obtain a multi-view image sequence with overlapping areas. The acquired images do not require known camera parameters; they only need to cover the target area and have some viewpoint variation to facilitate depth estimation and spatial reconstruction.

[0063] S2 performs sparse matching on multi-view images, distinguishing between successfully matched point pairs and unmatched points.

[0064] In one embodiment, step S2 includes:

[0065] S21, perform feature detection on multi-view images to obtain the positions of multiple sets of feature points.

[0066] Specifically, the SIFT (Scale-Invariant Feature Transform) algorithm or the ORB (Oriented FAST and Rotated BRIEF) algorithm can be used to detect features in multi-view images and obtain the position coordinates of multiple sets of feature points.

[0067] S22, Calculate the descriptor of the feature point.

[0068] The description vectors of the feature points are calculated as descriptors.

[0069] S23. Compare the descriptors of feature points in images from different viewpoints and find the nearest neighbor descriptor for matching.

[0070] For example, for a descriptor in image A, find the closest descriptor in image B by Euclidean distance (or Hamming distance, for binary descriptors such as ORB) and match it.

[0071] S24. Perform set verification on the matched point pairs, filter out the mismatched point pairs, and obtain the final successfully matched point pairs.

[0072] Randomly select a minimum subset from the matching point pairs, use this subset to calculate a hypothetical geometric model (fundamental matrix F or homography matrix H), use this model to test all other matching point pairs, count the number of "interior points" that conform to the model, repeat the above steps multiple times, and finally select the model with the most interior points, retain all interior point matching pairs, and remove outside points.

[0073] S3: Perform lightweight triangulation on the matching region corresponding to the successfully matched point pair to obtain the three-dimensional spatial coordinates of each feature point; perform monocular depth estimation on the unmatched region corresponding to the unmatched point to obtain the three-dimensional spatial coordinates of each feature point.

[0074] In one embodiment, step S3 involves performing lightweight triangulation on the matching region corresponding to the successfully matched point pairs to obtain the three-dimensional spatial coordinates of each feature point, including:

[0075] S311, calculate the fundamental matrix based on successfully matched point pairs.

[0076] S312 calculates the essential matrix based on the fundamental matrix and camera intrinsic parameters.

[0077] S313, recovers camera extrinsic parameters based on fundamental and essential matrices.

[0078] S314 uses triangulation based on camera extrinsic parameters to obtain the three-dimensional spatial coordinates of each feature point.

[0079] Among them, the direct linear transformation method or the method of minimizing reprojection error can be used to perform triangulation to obtain the three-dimensional spatial coordinates of each feature point.

[0080] In one embodiment, step S3, which involves performing monocular depth estimation on the unmatched region corresponding to the unmatched point to obtain the three-dimensional spatial coordinates of each feature point, includes:

[0081] S321, extract salient regions from unmatched regions using a saliency detection algorithm.

[0082] Through such SAM, or traditional image saliency models, extract salient regions from unmatched regions.

[0083] S322 uses a monocular depth estimation network to predict the relative depth of each pixel within a salient region.

[0084] Among them, monocular depth estimation networks can include network models such as MiDaS, DPT, and BTS.

[0085] S323, normalize the relative depth to obtain the normalized depth value.

[0086] Specifically, the depth values ​​are normalized / scaled using image size or empirical scale to ensure consistent depth across different viewpoints.

[0087] S324 combines the center point of the salient region with the corresponding normalized depth value to generate three-dimensional spatial coordinates, and assigns covariance matrix, pixel color, and transparency attributes.

[0088] S4, constructs a set of Gaussian points based on three-dimensional spatial coordinates.

[0089] Based on the pixel color of the multi-view image, the three-dimensional spatial coordinates of each feature point, the covariance matrix, and the transparency attribute, a Gaussian point set is constructed and initialized.

[0090] Specifically, for each Gaussian point Initialize the following properties:

[0091]

[0092] in, For three-dimensional spatial position coordinates, Let covariance matrix be the variance matrix. It is an RGB color vector. This is the opacity factor.

[0093] The initial values ​​of the covariance matrix at the Gaussian points satisfy the following relationship:

[0094]

[0095] in, It is the identity matrix. This is a scaling factor that is adaptively adjusted based on image size and dot density.

[0096] S5 uses differentiable rendering based on a set of Gaussian points to generate simulated images from various perspectives.

[0097] In one embodiment, step S5 includes:

[0098] S51 projects the points in the Gaussian point set onto a two-dimensional image to obtain a two-dimensional Gaussian ellipse.

[0099] S52 collects two-dimensional Gaussian ellipses within the coverage area of ​​each pixel and performs weighted calculations based on the transparency attribute and the covariance matrix.

[0100] S53, using volume rendering formulas to process multiple Gaussian ellipses. - Blend to obtain the color value of the pixel.

[0101] Furthermore, depth accumulation results can be calculated simultaneously for alignment with monocular depth priors.

[0102] S54 combines the color values ​​of the pixels to obtain a rendered image sequence, and then determines the rendered image sequence as a simulated image.

[0103] S6. Construct a total loss function based on simulated and real images. Then, jointly optimize the Gaussian point parameters of the Gaussian point set based on the total loss function to obtain the final Gaussian model.

[0104] The total loss function includes reconstruction error and regularization term, and the total loss function is:

[0105]

[0106] in, For the total loss function, Indicates differentiable rendering. Let be the set of Gaussian points. For image Viewpoint parameters, For real images, These are the weighting coefficients. Here, N is the regularization term, and N is the number of training images. This indicates the reconstruction error.

[0107] In one embodiment, the regularization term can be used to constrain the geometric consistency and control the covariance between Gaussian points. Expanding the regularization term yields:

[0108]

[0109] in, This represents the connection weight between Gaussian point pairs. It is the Frobenius norm. To adjust the parameters, Represents the three-dimensional spatial coordinates of different points. Used for geometric consistency constraints Used for covariance control.

[0110] In one embodiment, the regularization term can also be used to constrain the spatial consistency and control sparsity among Gaussian points. The total loss function after expanding the regularization term is:

[0111]

[0112] in, To simulate an image, For real images, , These are the weighting coefficients. Let i be the connection weight between Gaussian points i and k. Let Gaussian point be its three-dimensional spatial coordinates. For transparency properties, To reconstruct errors, Used for spatial consistency constraints, it encourages neighboring points to maintain reasonable geometric positions. As a sparsity regularization term, it encourages a sparser distribution of the number of points and transparency, thereby avoiding redundancy.

[0113] The total loss function is used to jointly optimize all Gaussian point parameters (position, covariance, color, and opacity) through multiple iterations using backpropagation. The optimization process is divided into stages (a three-stage training strategy): the initial stage (warm-up stage) optimizes only position and color;

[0114] In the mid-stage (point density adjustment stage), point density and covariance are adjusted; in the late-stage (detail enhancement stage), Gaussian point attributes are refined to improve edge and detail modeling capabilities. Ultimately, the output Gaussian point set can be used for image synthesis from any viewpoint and 3D scene visualization, suitable for various applications such as virtual reality, artifact digitization, and robot perception.

[0115] This invention effectively overcomes the dependence of traditional 3D modeling workflows on sparse point clouds and camera parameters. Through an initialization strategy combining saliency detection and sparse matching, and a joint optimization mechanism integrating monocular depth estimation and differentiable rendering, this method can stably complete high-quality modeling in complex scenes with weak textures, repetitive textures, and unstructured environments. Compared with existing Gaussian modeling methods based on SfM or MVS, this invention has significant advantages in modeling robustness, workflow simplification, and system generalization ability. It is applicable to various 3D reconstruction applications such as cultural relic digitization, virtual reality, and robot perception, and has promising prospects for practical application and widespread adoption.

[0116] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of an image-driven 3D scene reconstruction device that does not require SfM initialization. Specifically, the image-driven 3D scene reconstruction device that does not require SfM initialization can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0117] Please see Figure 2 , Figure 2 This application provides a specific description of an image-driven 3D scene reconstruction device that does not require SfM initialization, which is applied in electronic devices. This SfM-initialization-free image-driven 3D scene reconstruction device may include:

[0118] The acquisition module is used to acquire images from multiple perspectives;

[0119] The sparse matching module is used to perform sparse matching on the multi-view image and distinguish between successfully matched point pairs and unmatched points.

[0120] The three-dimensional spatial coordinate calculation module is used to perform lightweight triangulation on the matching region corresponding to the successfully matched point pair to obtain the three-dimensional spatial position coordinates of each feature point; and to perform monocular depth estimation on the unmatched region corresponding to the unmatched point to obtain the three-dimensional spatial position coordinates of each feature point.

[0121] A Gaussian point set construction module is used to construct a Gaussian point set based on the three-dimensional spatial position coordinates;

[0122] Differentiable rendering module, used to perform differentiable rendering based on the Gaussian point set to generate simulated images from various viewpoints;

[0123] The optimization module is used to construct a total loss function based on the simulated image and the real image, and to jointly optimize the Gaussian point parameters of the Gaussian point set based on the total loss function to obtain the final Gaussian model.

[0124] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0125] In addition, this application also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the image-driven 3D scene reconstruction method without SfM initialization provided in this application. Therefore, it can achieve the beneficial effects of any image-driven 3D scene reconstruction method without SfM initialization provided in this invention, as detailed in the preceding embodiments, and will not be repeated here.

[0126] Figure 3 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the image-driven 3D scene reconstruction method without SfM initialization provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0127] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control.

[0129] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0130] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0131] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0132] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0133] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations:

[0134] Acquire multi-view images;

[0135] Sparse matching is performed on the multi-view image to distinguish between successfully matched point pairs and unmatched points;

[0136] Lightweight triangulation is performed on the matching regions corresponding to the successfully matched points to obtain the three-dimensional spatial coordinates of each feature point; monocular depth estimation is performed on the unmatched regions corresponding to the unmatched points to obtain the three-dimensional spatial coordinates of each feature point.

[0137] Construct a set of Gaussian points based on the three-dimensional spatial coordinates;

[0138] Differentiable rendering is performed based on the Gaussian point set to generate simulated images from various viewpoints;

[0139] A total loss function is constructed based on the simulated image and the real image. The Gaussian point parameters of the Gaussian point set are jointly optimized based on the total loss function to obtain the final Gaussian model.

[0140] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0141] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the image-driven 3D scene reconstruction method without SfM initialization provided by the present invention.

[0142] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0143] Since the instructions stored in the storage medium can execute the steps in any embodiment of the image-driven 3D scene reconstruction method without SfM initialization provided in the embodiments of the present invention, the beneficial effects that any image-driven 3D scene reconstruction method without SfM initialization provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0144] The foregoing has provided a detailed description of an image-driven 3D scene reconstruction method, apparatus, storage medium, and electronic device that does not require SfM initialization, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for image-driven 3D scene reconstruction without SfM initialization, characterized in that, The method includes: Acquire multi-view images; Sparse matching is performed on the multi-view image to distinguish between successfully matched point pairs and unmatched points; Lightweight triangulation is performed on the matching region corresponding to the successfully matched point pairs to obtain the three-dimensional spatial coordinates of each feature point, including: calculating the fundamental matrix based on the successfully matched point pairs; calculating the essential matrix based on the fundamental matrix and camera intrinsic parameters; recovering the camera extrinsic parameters based on the fundamental matrix and the essential matrix; and performing triangulation based on the camera extrinsic parameters to obtain the three-dimensional spatial coordinates of each feature point. Monocular depth estimation is performed on the unmatched regions corresponding to the unmatched points to obtain the three-dimensional spatial coordinates of each feature point. This includes: extracting salient regions from the unmatched regions using a saliency detection algorithm; predicting the relative depth of each pixel within the salient region using a monocular depth estimation network; normalizing the relative depth to obtain normalized depth values; and combining the center point of the salient region with the corresponding normalized depth values ​​to generate three-dimensional spatial coordinates, and assigning covariance matrix, pixel color, and transparency attributes. Construct a set of Gaussian points based on the three-dimensional spatial coordinates; Differentiable rendering is performed based on the Gaussian point set to generate simulated images from various viewpoints; A total loss function is constructed based on the simulated image and the real image. The Gaussian point parameters of the Gaussian point set are jointly optimized based on the total loss function to obtain the final Gaussian model.

2. The image-driven 3D scene reconstruction method without SfM initialization according to claim 1, characterized in that, Based on the aforementioned three-dimensional spatial coordinates, a set of Gaussian points is constructed, including: Based on the pixel color of the multi-view image, the three-dimensional spatial coordinates of each feature point, the covariance matrix, and the transparency attribute, a Gaussian point set is constructed and initialized.

3. The image-driven 3D scene reconstruction method without SfM initialization according to claim 1, characterized in that, Differentiable rendering is performed based on the Gaussian point set to generate simulated images from various viewpoints, including: Projecting the points in the Gaussian point set onto a two-dimensional image yields a two-dimensional Gaussian ellipse; Collect two-dimensional Gaussian ellipses within the coverage area of ​​each pixel and perform weighted calculations based on the transparency attribute and the covariance matrix; Use volume rendering formulas to process multiple Gaussian ellipses. - Blend to obtain the color value of the pixel; The color values ​​of the pixels are combined to obtain a rendered image sequence, and the rendered image sequence is determined as the simulated image.

4. The image-driven 3D scene reconstruction method without SfM initialization according to claim 1, characterized in that, The total loss function includes reconstruction error and regularization term, and the total loss function is: in, For the total loss function, Indicates differentiable rendering. Let be the set of Gaussian points. For image Viewpoint parameters, For real images, These are the weighting coefficients. is the regularization term, and N is the number of training images.

5. The image-driven 3D scene reconstruction method without SfM initialization according to claim 4, characterized in that, The regularization term is used to constrain the geometric consistency and control the covariance between Gaussian points. Expanding the regularization term yields: in, This represents the connection weight between Gaussian point pairs. It is the Frobenius norm. To adjust the parameters, Represents the three-dimensional spatial coordinates of different points.

6. The image-driven 3D scene reconstruction method without SfM initialization according to claim 4, characterized in that, The regularization term is also used to constrain the spatial consistency and control sparsity among Gaussian points. The total loss function after expanding the regularization term is: in, To simulate an image, For real images, , These are the weighting coefficients. Let i be the connection weight between Gaussian points i and k. Let Gaussian point be its three-dimensional spatial coordinates. This is the transparency attribute.

7. An image-driven 3D scene reconstruction device without SfM initialization, wherein the image-driven 3D scene reconstruction device without SfM initialization is used to implement the image-driven 3D scene reconstruction method without SfM initialization as described in claim 1, characterized in that, include: The acquisition module is used to acquire images from multiple perspectives; The sparse matching module is used to perform sparse matching on the multi-view image and distinguish between successfully matched point pairs and unmatched points. The three-dimensional spatial coordinate calculation module is used to perform lightweight triangulation on the matching region corresponding to the successfully matched point pair to obtain the three-dimensional spatial position coordinates of each feature point; and to perform monocular depth estimation on the unmatched region corresponding to the unmatched point to obtain the three-dimensional spatial position coordinates of each feature point. A Gaussian point set construction module is used to construct a Gaussian point set based on the three-dimensional spatial position coordinates; Differentiable rendering module, used to perform differentiable rendering based on the Gaussian point set to generate simulated images from various viewpoints; The optimization module is used to construct a total loss function based on the simulated image and the real image, and to jointly optimize the Gaussian point parameters of the Gaussian point set based on the total loss function to obtain the final Gaussian model.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted to be loaded by a processor to execute the image-driven 3D scene reconstruction method without SfM initialization as described in any one of claims 1 to 6.

Images