Gaussian optimization method and device under sparse view angle, equipment and storage medium

Abstract

The embodiment of the invention provides a Gaussian optimization method and device under a sparse view angle, equipment and a storage medium, and relates to the technical field of computer vision and graphics. According to the method, rough three-dimensional Gaussian of a target scene is obtained, at least one training iteration is carried out, and parameter training is carried out on Gaussian elements in the rough three-dimensional Gaussian until a target Gaussian model is obtained. In a training iteration process, if an iteration round is an optimization iteration round, a pixel importance value is calculated based on a reconstruction error, a semantic prior error and a geometric prior error, for each view angle, a corresponding fine Gaussian primitive is determined based on the pixel importance value, and the rough three-dimensional Gaussian is updated according to all the fine Gaussian primitives. For each visual angle, a corresponding fine Gaussian primitive is determined according to a pixel importance value, a scene key area is focused, insufficient key information modeling caused by undifferentiated distribution of the Gaussian primitive is avoided under a sparse data condition, and the training accuracy of three-dimensional Gaussian sputtering in a sparse scene is improved.

Description

Gaussian optimization methods, apparatus, devices, and storage media from a sparse perspective Technical Field

[0001] This application relates to the fields of computer vision and graphics technology, and in particular to Gaussian optimization methods, apparatus, devices and storage media under sparse viewpoints. Background Technology

[0002] Novel view synthesis has wide applications in virtual reality, augmented reality, robotics, and content creation. 3D Gaussian Splatting (3DGS) can achieve high-fidelity real-time rendering with dense multi-view input. However, its performance degrades significantly under extreme conditions with sparse viewpoints (e.g., only three input views).

[0003] Related technologies introduce monocular or multi-view depth estimation as an additional supervisory signal to geometrically constrain the 3DGS optimization process, thereby improving the robustness of reconstruction. However, this approach applies spatially uniform constraints, failing to distinguish the geometric recoverability of different regions in the scene and unable to intelligently guide computational resources to the regions that most require refinement and can be reliably recovered. Therefore, view synthesis based on 3D Gaussian splashing performs poorly in sparse scenes. Summary of the Invention

[0004] The main objective of this application is to propose a Gaussian optimization method, apparatus, device, and storage medium under sparse perspective, so as to improve the training accuracy of 3D Gaussian splashing in sparse scenes and the accuracy of view synthesis results.

[0005] To achieve the above objectives, a first aspect of this application proposes a Gaussian optimization method under sparse viewpoints, comprising: acquiring sparse images corresponding to multiple viewpoints in a target scene, and initializing point clouds based on the sparse images to obtain a coarse three-dimensional Gaussian model, wherein the coarse three-dimensional Gaussian model includes at least one Gaussian primitive; performing at least one training iteration to train the parameters of the Gaussian primitives in the coarse three-dimensional Gaussian model until a target Gaussian model is obtained; during the training iteration, if the iteration round is an optimization iteration round, rendering is performed based on the coarse three-dimensional Gaussian model for each viewpoint to obtain a corresponding rendered image; calculating a reconstruction error based on the rendered image and the corresponding sparse image; performing semantic prior calculation based on the sparse image to obtain a semantic prior error; acquiring a monocular depth map corresponding to the viewpoint; performing geometric prior calculation based on the monocular depth image to obtain a geometric prior error; calculating pixel importance values ​​based on the reconstruction error, the semantic prior error, and the geometric prior error; for each viewpoint, determining the corresponding fine Gaussian primitives based on the pixel importance values; and updating the coarse three-dimensional Gaussian model based on all the fine Gaussian primitives.

[0006] In some embodiments, the step of performing semantic prior calculation based on the sparse image to obtain semantic prior error includes: inputting the sparse image into a pre-trained segmentation network for semantic segmentation to obtain a corresponding semantic map; for each pixel location, obtaining the pixel value in the semantic map, calculating the sum of parameters of the boundary hyperparameter and the foreground weight hyperparameter, calculating the product of the pixel value and the sum of parameters, and obtaining the semantic prior error corresponding to the pixel location.

[0007] In some embodiments, obtaining the monocular depth map corresponding to the viewpoint and performing geometric prior calculation based on the monocular depth image to obtain the geometric prior error includes: inputting the sparse image into a pre-trained feature extraction model for data processing to obtain the corresponding monocular depth map, and obtaining the first-order gradient map and second-order derivative corresponding to the monocular depth map; for each pixel position, obtaining the gradient value in the first-order gradient map, calculating the norm corresponding to the gradient value, calculating the product of the curvature weight hyperparameter and the second-order derivative, and adding the norm to obtain the geometric prior error corresponding to the pixel position.

[0008] In some embodiments, determining the corresponding fine Gaussian unit based on the pixel importance value for each viewpoint includes: obtaining geometric constraint parameters for each pixel position in the sparse image of the viewpoint; and obtaining the position information of the fine Gaussian unit based on the pixel importance value and the geometric constraint parameters corresponding to each pixel position.

[0009] In some embodiments, obtaining the geometric constraint parameters of each pixel position in the sparse image of the viewpoint includes: for each pixel position, if the corresponding geometric prior error is greater than or equal to a threshold hyperparameter, the corresponding geometric constraint parameter is one; otherwise, it is zero.

[0010] In some embodiments, obtaining the position information of the fine Gaussian primitive based on the pixel importance value and the geometric constraint region mask corresponding to each pixel position includes: traversing all pixel positions, calculating the product of the corresponding pixel importance value and the geometric constraint parameter as a position parameter, and accumulating all position parameters to obtain an intermediate position value; for each pixel position, calculating the quotient of the position parameter and the intermediate position value to obtain the position probability indicating the fine Gaussian primitive; and performing back projection based on the position probability and the pixel position to determine the position information of the fine Gaussian primitive in the target scene.

[0011] In some embodiments, after updating the coarse 3D Gaussian based on all the fine Gaussian primitives, the method further includes: obtaining a preset protection cycle, and starting from the current iteration cycle, pruning and freezing the fine Gaussian primitives for protection during the continuous preset protection cycle.

[0012] To achieve the above objectives, a second aspect of this application proposes a Gaussian optimization device under sparse viewpoints, comprising: an initial Gaussian acquisition module: configured to acquire sparse images corresponding to multiple viewpoints in a target scene, and perform point cloud initialization based on the sparse images to obtain a coarse three-dimensional Gaussian, wherein the coarse three-dimensional Gaussian includes at least one Gaussian primitive; a training iteration module: configured to perform at least one training iteration to train the parameters of the Gaussian primitives in the coarse three-dimensional Gaussian until a target Gaussian model is obtained; and an iterative optimization module: configured to, during the training iteration process, if the iteration round is an optimization iteration round, optimize the Gaussian model based on the coarse three-dimensional Gaussian primitive. A rough 3D Gaussian is rendered at each viewpoint to obtain a corresponding rendered image. The reconstruction error is calculated based on the rendered image and the corresponding sparse image. Semantic prior is calculated based on the sparse image to obtain the semantic prior error. A monocular depth map corresponding to the viewpoint is obtained. Geometric prior is calculated based on the monocular depth image to obtain the geometric prior error. Pixel importance values ​​are calculated based on the reconstruction error, the semantic prior error, and the geometric prior error. For each viewpoint, the corresponding fine Gaussian unit is determined based on the pixel importance value. The rough 3D Gaussian is updated based on all the fine Gaussian units.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0015] The sparse-view Gaussian optimization method, apparatus, device, and storage medium proposed in this application acquire sparse images corresponding to multiple views in a target scene, and initialize point clouds based on the sparse images to obtain a coarse 3D Gaussian model, wherein the coarse 3D Gaussian model includes at least one Gaussian primitive. At least one training iteration is performed to train the parameters of the Gaussian primitives in the coarse 3D Gaussian model until the target Gaussian model is obtained. During the training iteration, if the iteration round is the optimization iteration round, rendering is performed based on the coarse 3D Gaussian model for each view to obtain the corresponding rendered image. The reconstruction error is calculated based on the rendered image and the corresponding sparse image. Semantic prior calculation is performed based on the sparse image to obtain the semantic prior error. A monocular depth map corresponding to the view is acquired, and geometric prior calculation is performed based on the monocular depth image to obtain the geometric prior error. Pixel importance values ​​are calculated based on the reconstruction error, semantic prior error, and geometric prior error. For each view, the corresponding fine Gaussian primitive is determined based on the pixel importance value, and the coarse 3D Gaussian model is updated based on all the fine Gaussian primitives. This application integrates reconstruction error, semantic prior error, and geometric prior error to conduct multimodal importance assessment. A higher pixel importance value indicates a greater impact of the scene information corresponding to that pixel on modeling accuracy, necessitating optimization of Gaussian primitives in the corresponding region. For each viewpoint, refined Gaussian primitives are determined based on pixel importance values, focusing on key scene regions such as object contours, semantic cores, and geometric boundaries. This avoids insufficient modeling of key information due to indiscriminate distribution of Gaussian primitives under sparse data conditions. Simultaneously, a coarse 3D Gaussian model is updated using all refined Gaussian primitives, achieving dynamic optimization of Gaussian primitive parameters. In each iteration, semantic and geometric priors are used to enhance the model's capture of essential scene features, thereby improving the training accuracy of 3D Gaussian sputtering in sparse scenes. The trained and optimized target Gaussian model integrates multiple precise constraints from visual, semantic, and geometric perspectives, enabling it to better match the appearance and structure of the real scene during rendering from different viewpoints, thus significantly improving the accuracy of the view synthesis results. Attached Figure Description

[0016] Figure 1 is a flowchart of the Gaussian optimization method from a sparse perspective provided in an embodiment of this application.

[0017] Figure 2 is an overall flowchart of Gaussian optimization from a sparse perspective provided in the embodiments of this application.

[0018] Figure 3 is a flowchart of semantic prior calculation based on sparse images to obtain semantic prior error, provided in an embodiment of this application.

[0019] Figure 4 is a flowchart of the monocular depth map corresponding to the acquisition viewpoint provided in the embodiment of this application, and the geometric prior calculation based on the monocular depth image to obtain the geometric prior error.

[0020] Figure 5 is a flowchart of determining the corresponding fine Gaussian elements based on pixel importance values ​​for each viewpoint, as provided in an embodiment of this application.

[0021] Figure 6 is a flowchart of obtaining the position information of fine Gaussian elements based on the pixel importance value and geometric constraint region mask corresponding to each pixel position, provided in an embodiment of this application.

[0022] Figure 7 is a schematic diagram of the comparison of rendering results provided in the embodiments of this application.

[0023] Figure 8 is a block diagram of a Gaussian optimization device under sparse perspective provided in another embodiment of this application.

[0024] Figure 9 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] First, let's clarify some terms used in this application: Artificial Intelligence (AI): is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. AI also refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0029] Novel view synthesis has wide applications in virtual reality, augmented reality, robotics, and content creation. 3D Gaussian Splatting (3DGS) can achieve high-fidelity real-time rendering with dense multi-view input. However, its performance degrades significantly under extreme conditions with sparse viewpoints (e.g., only three input views). This is mainly because: geometric supervision in space becomes sparse and non-uniform; and its default dense pruning strategy blindly scatters Gaussian points, wasting the limited expressive power of the model on well-observed surfaces, while underfitting object boundaries and textured areas crucial for reconstructing photorealistic results.

[0030] To mitigate reconstruction failures under sparse views, several approaches exist in related technologies. The first is to introduce monocular or multi-view depth estimation as additional supervision signals to impose geometric constraints on the 3DGS optimization process, thereby improving reconstruction robustness. The second is to introduce multiple regularization fields, such as semantic and geometric fields, to jointly constrain 3DGS optimization, thus improving reconstruction consistency. The third is to randomly discard some Gaussian points or views during training as a regularization method to prevent overfitting. The fourth is to utilize pre-trained models to generate dense matching priors to estimate depth and initialize a denser point cloud accordingly, providing better initial points for 3DGS.

[0031] While the above strategies can improve robustness to some extent, they all have inherent limitations: The first approach often imposes spatially uniform constraints, failing to distinguish the geometric recoverability of different regions in the scene and unable to intelligently guide computational resources to the regions that most need refinement and can be reliably recovered; the second approach struggles to effectively distinguish between real geometric edges and high-frequency appearance textures or noise, easily leading to overfitting in high-texture regions or introducing unnecessary geometric noise such as "floating objects" in smooth regions; the third approach uses a single prior (such as depth or semantics) in isolation, failing to effectively integrate complementary multimodal information such as photometric errors, semantic boundaries, and geometric complexity to jointly evaluate whether a region is worthwhile and capable of refinement and reconstruction; the fourth approach relies on pre-trained models to initialize point clouds, which can provide a better starting point, but does not change the allocation strategy of Gaussian points in the subsequent optimization process, which may still lead to wasted capacity. At the same time, newly added Gaussian points are easily removed prematurely during the pruning stage under insufficient supervision, failing to effectively retain them to represent details. In summary, the methods of related technologies have problems such as blind Gaussian point allocation strategy, insufficient utilization of multimodal priors, easy overfitting of noise or texture, and limited ability to recover details when performing 3D Gaussian splash reconstruction in sparse scenes, resulting in poor view synthesis effect based on 3D Gaussian splash in sparse scenes.

[0032] Based on this, this application provides a Gaussian optimization method, apparatus, device, and storage medium under sparse viewpoints. It conducts multimodal importance assessment by fusing reconstruction error, semantic prior error, and geometric prior error. A higher pixel importance value indicates a greater impact of the scene information corresponding to that pixel on modeling accuracy, necessitating optimization of Gaussian primitives in the corresponding region. For each viewpoint, a corresponding refined Gaussian primitive is determined based on the pixel importance value, focusing on key scene regions such as object contours, semantic cores, and geometric boundaries. This avoids insufficient modeling of key information due to indiscriminate distribution of Gaussian primitives under sparse data conditions. Simultaneously, a coarse 3D Gaussian model is updated using all refined Gaussian primitives, achieving dynamic optimization of Gaussian primitive parameters. In each iteration, semantic and geometric priors are used to enhance the model's capture of essential scene features, thereby improving the training accuracy of 3D Gaussian sputtering in sparse scenes. The trained and optimized target Gaussian model integrates multiple precise constraints from vision, semantics, and geometry, enabling it to better match the appearance and structure of the real scene during rendering from different viewpoints, thus significantly improving the accuracy of the view synthesis results.

[0033] This application provides a Gaussian optimization method, apparatus, device, and storage medium from a sparse perspective, which are specifically described through the following embodiments. First, the Gaussian optimization method from a sparse perspective in this application embodiment is described.

[0034] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0035] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] The Gaussian optimization method under a sparse perspective provided in this application relates to the fields of computer vision and graphics technology. This method can be applied to a terminal, a server, or a computer program running on either a terminal or a server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting Gaussian optimization under a sparse perspective, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded into any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The Gaussian optimization method under a sparse perspective can be executed by the terminal or the server, or by the terminal and the server working together.

[0037] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device applies the Gaussian optimization method under sparse perspective of this embodiment to provide related services and improve the driving experience. The server can be an independent server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and the server can be connected via Bluetooth, Universal Serial Bus (USB), or network communication methods; this embodiment does not impose any limitations.

[0038] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0039] The following describes the Gaussian optimization method from a sparse perspective in the embodiments of this application.

[0040] Figure 1 is an optional flowchart of the Gaussian optimization method from a sparse perspective provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps 110 to 130. It is also understood that this embodiment does not specifically limit the order of steps 110 to 130 in Figure 1, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0041] Step 110: Obtain sparse images corresponding to multiple viewpoints in the target scene, and initialize point clouds based on the sparse images to obtain a coarse 3D Gaussian cloud.

[0042] In one embodiment, for the target scene, image acquisition devices (such as cameras) can be deployed in a 360-degree panoramic surround layout to achieve synchronous acquisition. This surround layout can cover the target scene from all directions, ensuring no angular information is missed, thus enabling the acquisition of data from all angles. Each image acquisition device corresponds to a specific viewpoint, similar to observing the same object from different positions, where the images seen from different positions differ. At each viewpoint, an image corresponding to the same acquisition moment is acquired using the image acquisition devices. Since these image acquisition devices are sparsely arranged, the corresponding acquired images are called sparse images. At each acquisition moment, sparse point cloud data of the target scene is generated based on the sparse images from all viewpoints.

[0043] Next, COLMAP is used to calibrate the camera system corresponding to the image acquisition device, thereby obtaining the camera's intrinsic and extrinsic parameters. The intrinsic parameters mainly include the camera's focal length, principal point position, and lens distortion parameters, which reflect the camera's internal imaging characteristics. The extrinsic parameters relate to the camera's position and attitude information in space, such as the camera's rotation angle and translation vector, aiming to determine the camera's spatial relationship with the target scene. After completing the COLMAP calibration, the camera's intrinsic and extrinsic parameters are used to initialize the sparse point cloud data, performing calibration to obtain a 3D Gaussian model corresponding to each acquisition moment. This 3D Gaussian model includes multiple Gaussian elements. Since this is a 3D Gaussian model obtained from sparse acquisition, it is... After several iterations of 3D Gaussian splash training, a coarse 3D Gaussian is obtained, which includes at least one Gaussian element.

[0044] Step 120: Perform at least one training iteration to train the parameters of the Gaussian primitives in the coarse 3D Gaussian until the target Gaussian model is obtained.

[0045] In one embodiment, in a 3D Gaussian splash, the properties of Gaussian primitives in a coarse 3D Gaussian are trained and adjusted. The properties may be: center position coordinates, color properties, opacity properties, rotation properties, scaling properties, normal vectors, and visibility values.

[0046] During training, for each viewpoint, a coarse 3D Gaussian rendering is performed to obtain the corresponding rendering result. The loss function calculated based on the rendering result and the sparse image of the viewpoint is expressed as follows:

[0047]

[0048]

[0049]

[0050] in, This represents the preset weighting coefficient. Represents a sparse image. This indicates the rendering result. This represents an error metric for calculating the norm of pixel values ​​in a pixel-by-pixel sparse image and its rendering result. Represents the structural similarity index, where, , It is the mean x and y values ​​of the image patch within the local window. , That is the corresponding variance. It is the covariance, a constant. , This loss function is used to stabilize the denominator and prevent instability when the denominator approaches zero. It is understandable that the above loss function is the commonly used loss function in 3D Gaussian splashing in related techniques.

[0051] In one embodiment, during the training iteration process, each iteration is recorded as an iteration round. When the iteration round is completed, the resulting three-dimensional Gaussian model is the target Gaussian model.

[0052] Step 130: During the training iteration, if the iteration round is the optimization iteration round, then render the corresponding rendered image based on the coarse 3D Gaussian. Calculate the reconstruction error based on the rendered image and the corresponding sparse image. Perform semantic prior calculation based on the sparse image to obtain the semantic prior error. Obtain the monocular depth map corresponding to the viewpoint. Perform geometric prior calculation based on the monocular depth image to obtain the geometric prior error. Calculate the pixel importance value based on the reconstruction error, semantic prior error, and geometric prior error. For each viewpoint, determine the corresponding fine Gaussian unit based on the pixel importance value. Update the coarse 3D Gaussian based on all the fine Gaussian units.

[0053] In one embodiment, it may be necessary to add or remove some Gaussian elements in an iteration. Therefore, in the training iteration process, in some iterations of this application, some Gaussian elements are added in a targeted manner to supplement the Gaussian elements in the rough 3D Gaussian to address the difficulty of sparse reconstruction. Such iterations that require targeted optimization are called optimization iterations.

[0054] For example, setting an initial sampling interval Based on this initial sampling interval, an optimized iterative sampling interval is generated. , is represented as:

[0055] in, Indicates the current iteration round. This represents a hyperparameter greater than 0.

[0056] It is evident that the optimization iteration sampling interval is not a fixed value, but rather a function related to the number of iteration rounds. As training progresses, the number of iteration rounds increases, and the corresponding... This will also increase linearly. As the optimization iteration sampling interval lengthens, the sampling frequency decreases. In other words, in the early stages of training, t is small, the optimization iteration sampling interval is small, and the sampling frequency is high. At the beginning of training, the coarse 3D Gaussian is very sparse, requiring rapid addition of Gaussian in key regions to construct the basic geometry and texture of the scene. Therefore, it is necessary to select optimization iteration rounds relatively frequently, adding fine Gaussian units in each iteration to quickly increase the density of 3D Gaussian in detailed regions, accelerating the reconstruction of scene details. In the later stages of training, t is large, the optimization iteration sampling interval becomes larger, and the sampling frequency decreases. At this point, it is no longer necessary to frequently select optimization iteration rounds to add new Gaussian units. Adding new Gaussian units at a high frequency at this point may damage already optimized regions, introducing instability and increasing unnecessary computational overhead.

[0057] The following describes in detail how the rough 3D Gaussian is optimized and supplemented in each optimization iteration round.

[0058] In one embodiment, referring to Figure 2, which is an overall flowchart of Gaussian optimization under sparse viewpoints provided in this application embodiment, the process involves first rendering each viewpoint based on a coarse 3D Gaussian to obtain the corresponding rendered image. Then, the reconstruction error at each pixel position is calculated based on the rendered image and the corresponding sparse image. The calculation process for the reconstruction error is as follows:

[0059] in, Indicates pixel position The corresponding reconstruction error, Represents a sparse image corresponding to a specific viewpoint. Indicates the rendered image. This indicates the computational norm.

[0060] Next, semantic prior evaluation is performed based on the sparse image. In one embodiment, referring to Figure 3, which is a flowchart of semantic prior calculation based on sparse image to obtain semantic prior error provided by the embodiment of this application, the process includes the following steps: Step 310: Input the sparse image into a pre-trained segmentation network for semantic segmentation to obtain the corresponding semantic map.

[0061] In one embodiment, the segmentation network This can be a ResNet-18-based segmentation network pre-trained on multiple preset semantic categories. Inputting a sparse image into this pre-trained segmentation network for semantic segmentation yields the corresponding semantic map. While the pixel distribution of a sparse image is discontinuous, the convolutional kernels and pooling operations of the segmentation network can capture local feature correlations. Combined with pre-trained semantic priors, the target and background in the sparse region can be accurately distinguished in the semantic map, avoiding semantic confusion caused by pixel sparsity.

[0062] Step 320: For each pixel location, obtain the pixel value in the semantic map, calculate the sum of the boundary hyperparameters and the foreground weight hyperparameters, calculate the product of the pixel value and the sum of the parameters, and obtain the semantic prior error corresponding to the pixel location.

[0063] In one embodiment, for each pixel location We obtain the pixel value of the pixel in the semantic map, then calculate the sum of the boundary hyperparameters and foreground weight hyperparameters corresponding to the pixel position, and finally calculate the product of the pixel value and the sum of the parameters to obtain the semantic prior error corresponding to the pixel position.

[0064] Therefore, pixel position Corresponding semantic prior error Represented as:

[0065] in, A semantic graph representing a sparse image. Indicates pixel position Pixel values ​​in a semantic map Indicates the position of this pixel The corresponding boundary hyperparameters, Indicates the position of this pixel The corresponding foreground weight hyperparameters. It's understandable that the boundary hyperparameters and forward weight hyperparameters here are learnable parameters, and their values ​​may differ at different pixel locations.

[0066] Next, referring to Figure 2, geometric prior is performed based on the monocular depth map estimated by monocular imaging. Referring to Figure 4, Figure 4 is a flowchart of obtaining the monocular depth map corresponding to the viewpoint provided in the embodiment of this application, calculating the geometric prior based on the monocular depth image, and obtaining the geometric prior error. Specifically, it includes the following steps: Step 410: Input the sparse image into the pre-trained feature extraction model for data processing to obtain the corresponding monocular depth map, and obtain the first-order gradient map and second-order derivative corresponding to the monocular depth map.

[0067] In one embodiment, the pre-trained feature extraction model can be a DPT model. The DPT model has the global feature capture capability of the Transformer, which can effectively compensate for the pixel loss defect of sparse images, accurately restore the depth relationship in three-dimensional space, and transform sparse visual information in sparse images into dense and reliable depth representations to obtain the corresponding monocular depth map. .

[0068] Next, obtain the first-order gradient map corresponding to the monocular depth map. and second derivative The first-order gradient map is used to capture the edges of the depth map and describes the rate of change of depth values ​​in the X and Y directions of the monocular depth map. It calculates the depth difference between adjacent pixels using a difference operator. The second-order derivative is used to describe the rate of change of the first-order gradient, capture the curvature characteristics of the monocular depth map, and reflect the concavity and convexity of the depth map and its surface smoothness.

[0069] Step 420: For each pixel position, obtain the gradient value in the first-order gradient map, calculate the norm corresponding to the gradient value, calculate the product of the curvature weight hyperparameter and the second derivative, and add the norm to obtain the geometric prior error corresponding to the pixel position.

[0070] In one embodiment, for pixel position Obtain the first-order gradient map gradient values ​​in Calculate the norm corresponding to the gradient value, and calculate the curvature weight hyperparameter. With the second derivative After multiplying by the product, the norm is added to obtain the geometric prior error corresponding to the pixel position.

[0071] Therefore, pixel position Corresponding geometric prior error Represented as:

[0072] Understandably, the curvature weight hyperparameter is a learnable parameter.

[0073] In one embodiment, after obtaining the reconstruction error, semantic prior error, and geometric prior error, the pixel importance value corresponding to each pixel position under that viewpoint can be calculated. Therefore, the pixel position... Corresponding pixel importance value Represented as:

[0074]

[0075]

[0076] in, This represents the weighting coefficients corresponding to the reconstruction error, semantic prior error, and geometric prior error, respectively. It can be understood that the pixel importance matrix can be obtained by arranging the pixel importance values ​​of all pixel locations into a matrix. .

[0077] In one embodiment, with pixel importance values, the evaluation process of Gaussian elements at that viewpoint can be performed accordingly. Referring to Figure 5, Figure 5 is a flowchart of determining the corresponding fine Gaussian elements based on pixel importance values ​​for each viewpoint, provided by an embodiment of this application. Specifically, it includes the following steps: Step 510: Obtain the geometric constraint parameters of each pixel position in the sparse image of the viewpoint.

[0078] In one embodiment, for each pixel location If the corresponding geometric prior error Greater than or equal to a threshold hyperparameter obtained from prior knowledge or that can be learned. Then set the corresponding geometric constraint parameters. The value is 1 if it is true and 0 otherwise, so the geometric constraint parameters are expressed as:

[0079] in, This indicates an indicator function whose value is 1 when the condition within the parentheses is met, and 0 otherwise.

[0080] Understandably, for each pixel position under this viewpoint, the corresponding geometric constraint parameters are calculated. Since the values ​​of the geometric constraint parameters are 1 or 0, a mask image can be generated accordingly. This is to visually represent the geometrically constrained region.

[0081] Step 520: Obtain the position information of fine Gaussian elements based on the pixel importance value and geometric constraint parameters corresponding to each pixel position.

[0082] In one embodiment, with the pixel importance value and geometric constraint parameters corresponding to each pixel position, the position information of fine Gaussian elements can be calculated. Referring to Figure 6, Figure 6 is a flowchart of obtaining the position information of fine Gaussian elements based on the pixel importance value and geometric constraint region mask corresponding to each pixel position according to an embodiment of this application. Specifically, it includes the following steps: Step 610: Traverse all pixel positions, calculate the product of the corresponding pixel importance value and geometric constraint parameters as the position parameters, and accumulate all position parameters to obtain the intermediate position value.

[0083] In one embodiment, for pixel position Calculate the corresponding pixel importance value and geometric constraint parameters The product of these values ​​is used as the position parameter. Then, all position parameters are summed to obtain the intermediate position value, which is represented as: .

[0084] Step 620: For each pixel location, calculate the quotient of the location parameter and the median value of the location to obtain the location probability of the fine-grained Gaussian element. Perform back projection based on the location probability and the pixel location to determine the location information of the fine-grained Gaussian element in the target scene.

[0085] In one embodiment, for each pixel location, the quotient of its corresponding position parameter and the median value of the location is calculated to obtain the location probability indicating whether a fine Gaussian unit exists at that location. (Based on pixel location...) For example, its position parameters are Therefore, there exists a position probability of a fine-grained Gaussian element at this location. Represented as:

[0086] Understandably, the magnitude of the position probability determines whether a fine-grained Gaussian unit should be added to a pixel. Then, based on the position probability, the pixel positions that need additional fine-grained Gaussian units are determined as sampled pixels. By back-projecting the pixel positions of these sampled pixels, the position information of each fine-grained Gaussian unit in the target scene can be determined. Therefore, the position information is represented as:

[0087] in, These are the camera intrinsic parameter matrix, rendering matrix, and translation vector corresponding to the viewpoint, respectively. d is the depth value obtained from the rendering of the pixel position, representing the actual Z-coordinate of that point in the camera coordinate system. The rendering depth map corresponding to this viewpoint can be obtained in advance, and the corresponding pixel value can be determined from the rendering depth map as the depth value. u and v are two-dimensional pixel coordinate indices. These are three-dimensional world coordinates. Meanwhile, Gaussian covariance... Initialize it as an isotropic Gaussian using a predefined scale.

[0088] In one embodiment, referring to Figure 2, multiple refined 3D Gaussians are obtained from this viewpoint following the above process. Other viewpoints can be processed in the same way, ultimately resulting in multiple refined 3D Gaussians being merged into a coarse 3D Gaussian. Furthermore, to prevent immature clipping under sparse supervision, a preset protection round can be obtained, for example... In each iteration round, starting from the current iteration round, the fine Gaussian units are clipped and frozen for protection during the continuous preset protection rounds. In other words, these fine 3D Gaussians cannot be clipped during the continuous preset protection rounds.

[0089] Therefore, referring to Figure 2, the Gaussian optimization method under the sparse perspective of this application embodiment can be divided into two core stages. The first stage is the hierarchical 3D Gaussian construction stage: during the training iteration, if it is in a specified optimization iteration round, the coarse 3D Gaussian needs to be optimized first, and then a fine 3D Gaussian is added to it. During the optimization process, firstly, the reconstruction error, semantic prior error and geometric prior error are fused to carry out multimodal importance assessment, and the pixel importance value corresponding to each pixel position is obtained, so as to accurately identify and prioritize the reconstruction of regions with reliable detail recovery potential; then, the geometrically constrained regions are intuitively presented through the mask image, and the refinement computing resources are concentrated on the geometrically key regions. The second stage is the geometrically perceptual sampling stage: the pixel position that needs to be supplemented with fine Gaussian primitives is determined as the sampling pixel according to the position probability. The position of the sampling pixel is back-projected to clarify the spatial position information of each fine Gaussian primitive in the target scene. At the same time, a protection mechanism is used to avoid the premature pruning of newly added Gaussian primitives in the under-constrained region, and a sampling rhythm that increases with time is adopted to ensure stable convergence of the model and avoid getting trapped in local optima. This method accurately allocates limited model capacity to recoverable regions with consistent multimodal evidence, rather than blindly relying on a single rendering error. It effectively suppresses artifacts caused by insufficient modeling and achieves high-quality, high-fidelity novel view synthesis under extremely sparse input conditions.

[0090] Therefore, this embodiment focuses modeling resources on recoverable geometric details under sparse view conditions, balancing global shape stability with high-fidelity reconstruction of local textures / boundaries. By combining rendering residuals, semantic priors, and geometric complexity, and fusing them with linear weights, the injection location of fine Gaussians is directly driven, suppressing errors caused by texture noise and pose / appearance inconsistencies. During Gaussian optimization, two layers of Gaussian representation are introduced: a coarse layer encodes the global shape and stable scene geometry, initialized based on point clouds and remaining relatively stable during training; a fine layer is injected on demand in regions indicated by multimodal evidence, encoding recoverable local details and adaptively updated through dynamic encryption and pruning. Simultaneously, new Gaussians are adaptively placed in geometrically reliable regions, and a protection mechanism is employed to retain newly added Gaussians, preventing premature pruning under sparse supervision and thus improving the model's robustness.

[0091] In one embodiment, the Gaussian optimization method under sparse view conditions of this application is compared with the results of CoR-GS and NexusGS in related technologies, and the rendered results are compared with real images. Referring to Figure 7, Figure 7 is a schematic diagram of the rendering result comparison provided by the embodiment of this application. As can be seen from Figure 7, the embodiment of this application obtains reliable estimates of local recoverable details by using multimodal prior-guided importance sampling, and integrates rendering residuals, semantic priors, and geometric priors in a hierarchical 3D Gaussian framework, thereby driving the selective addition of fine Gaussians. This achieves synergistic optimization of global geometric stability and local details under sparse view conditions, and its new view synthesis has the best fidelity compared to other algorithms. Furthermore, it shows significant improvements in objective indicators such as PSNR, SSIM, and LPIPS, as well as subjective visual quality.

[0092] The technical solution provided in this application involves acquiring sparse images corresponding to multiple viewpoints in a target scene, and initializing point clouds based on the sparse images to obtain a coarse 3D Gaussian model, wherein the coarse 3D Gaussian model includes at least one Gaussian primitive. At least one training iteration is performed to train the parameters of the Gaussian primitives in the coarse 3D Gaussian model until the target Gaussian model is obtained. During the training iteration, if the iteration round is an optimization iteration round, rendering is performed based on the coarse 3D Gaussian model for each viewpoint to obtain the corresponding rendered image. The reconstruction error is calculated based on the rendered image and the corresponding sparse image. Semantic prior calculation is performed based on the sparse image to obtain the semantic prior error. A monocular depth map corresponding to the viewpoint is acquired, and geometric prior calculation is performed based on the monocular depth image to obtain the geometric prior error. Pixel importance values ​​are calculated based on the reconstruction error, semantic prior error, and geometric prior error. For each viewpoint, the corresponding fine Gaussian primitive is determined based on the pixel importance value. The coarse 3D Gaussian model is updated based on all the fine Gaussian primitives. This application integrates reconstruction error, semantic prior error, and geometric prior error to conduct multimodal importance assessment. A higher pixel importance value indicates a greater impact of the scene information corresponding to that pixel on modeling accuracy, necessitating optimization of Gaussian primitives in the corresponding region. For each viewpoint, refined Gaussian primitives are determined based on pixel importance values, focusing on key scene regions such as object contours, semantic cores, and geometric boundaries. This avoids insufficient modeling of key information due to indiscriminate distribution of Gaussian primitives under sparse data conditions. Simultaneously, a coarse 3D Gaussian model is updated using all refined Gaussian primitives, achieving dynamic optimization of Gaussian primitive parameters. In each iteration, semantic and geometric priors are used to enhance the model's capture of essential scene features, thereby improving the training accuracy of 3D Gaussian sputtering in sparse scenes. The trained and optimized target Gaussian model integrates multiple precise constraints from visual, semantic, and geometric perspectives, enabling it to better match the appearance and structure of the real scene during rendering from different viewpoints, thus significantly improving the accuracy of the view synthesis results.

[0093] This application embodiment also provides a Gaussian optimization device under sparse viewpoint, which can implement the above-mentioned Gaussian optimization method under sparse viewpoint. Referring to FIG8, the device includes: an initial Gaussian acquisition module 810: used to acquire sparse images corresponding to multiple viewpoints in the target scene, and perform point cloud initialization based on the sparse images to obtain a coarse three-dimensional Gaussian, wherein the coarse three-dimensional Gaussian includes at least one Gaussian element.

[0094] Training Iteration Module 820: Used to perform at least one training iteration to train the parameters of Gaussian primitives in a coarse 3D Gaussian until the target Gaussian model is obtained.

[0095] Iterative optimization module 830: During the training iteration process, if the iteration round is the optimization iteration round, it renders the corresponding rendered image based on the coarse 3D Gaussian at each viewpoint, calculates the reconstruction error based on the rendered image and the corresponding sparse image, performs semantic prior calculation based on the sparse image to obtain the semantic prior error, obtains the monocular depth map corresponding to the viewpoint, performs geometric prior calculation based on the monocular depth image to obtain the geometric prior error, calculates the pixel importance value based on the reconstruction error, semantic prior error and geometric prior error, determines the corresponding fine Gaussian unit based on the pixel importance value for each viewpoint, and updates the coarse 3D Gaussian based on all the fine Gaussian units.

[0096] The specific implementation of the Gaussian optimization device under the sparse perspective in this embodiment is basically the same as the specific implementation of the Gaussian optimization method under the sparse perspective described above, and will not be repeated here.

[0097] This application also provides an electronic device, including: at least one memory; at least one processor; and at least one program; the program is stored in the memory, and the processor executes the at least one program to implement the Gaussian optimization method under sparse perspective described above. This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0098] Please refer to Figure 9, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 901, which can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 902, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to implement the Gaussian optimization method under sparse perspective of the embodiments of this application. The input / output interface 903 is used to realize information input and output. The communication interface 904 is used to realize communication interaction between this device and other devices. Communication can be realized through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 905 transmits information between the various components of the device (such as the processor 901, memory 902, input / output interface 903 and communication interface 904). The processor 901, memory 902, input / output interface 903 and communication interface 904 realize communication connection between each other within the device through the bus 905.

[0099] This application embodiment also provides a storage medium that stores a computer program. When the computer program is executed by a processor, it implements the above-described Gaussian optimization method from a sparse perspective.

[0100] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0101] The Gaussian optimization method, apparatus, device, and storage medium proposed in this application under a sparse perspective offer technical solutions and beneficial effects.

[0102] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0103] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0106] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0107] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0109] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A Gaussian optimization method from a sparse perspective, characterized in that, include: A sparse image corresponding to multiple viewpoints in the target scene is acquired, and point cloud initialization is performed based on the sparse image to obtain a coarse 3D Gaussian model, which includes at least one Gaussian primitive. At least one training iteration is performed to train the parameters of the Gaussian primitive in the coarse 3D Gaussian model until the target Gaussian model is obtained. During the training iteration, if the iteration round is an optimization iteration round, rendering is performed based on the coarse 3D Gaussian model for each viewpoint to obtain a corresponding rendered image. The reconstruction error is calculated based on the rendered image and the corresponding sparse image. Semantic prior calculation is performed based on the sparse image to obtain the semantic prior error. A monocular depth map corresponding to the viewpoint is acquired, and geometric prior calculation is performed based on the monocular depth image to obtain the geometric prior error. Pixel importance values ​​are calculated based on the reconstruction error, the semantic prior error, and the geometric prior error. For each viewpoint, the corresponding fine Gaussian primitive is determined based on the pixel importance value. The coarse 3D Gaussian model is updated based on all the fine Gaussian primitives.

2. The Gaussian optimization method from a sparse perspective according to claim 1, characterized in that, The step of performing semantic prior calculation based on the sparse image to obtain semantic prior error includes: inputting the sparse image into a pre-trained segmentation network for semantic segmentation to obtain a corresponding semantic map; for each pixel position, obtaining the pixel value in the semantic map, calculating the sum of the boundary hyperparameters and the foreground weight hyperparameters, calculating the product of the pixel value and the sum of the parameters, and obtaining the semantic prior error corresponding to the pixel position.

3. The Gaussian optimization method from a sparse perspective according to claim 2, characterized in that, The step of obtaining the monocular depth map corresponding to the viewpoint and performing geometric prior calculation based on the monocular depth image to obtain the geometric prior error includes: inputting the sparse image into a pre-trained feature extraction model for data processing to obtain the corresponding monocular depth map, and obtaining the first-order gradient map and second-order derivative corresponding to the monocular depth map; for each pixel position, obtaining the gradient value in the first-order gradient map, calculating the norm corresponding to the gradient value, calculating the product of the curvature weight hyperparameter and the second-order derivative, and adding the norm to obtain the geometric prior error corresponding to the pixel position.

4. The Gaussian optimization method from a sparse perspective according to claim 2, characterized in that, The step of determining the corresponding fine Gaussian unit based on the pixel importance value for each viewpoint includes: obtaining the geometric constraint parameters of each pixel position in the sparse image of the viewpoint; and obtaining the position information of the fine Gaussian unit according to the pixel importance value and the geometric constraint parameters corresponding to each pixel position.

5. The Gaussian optimization method under sparse perspective according to claim 4, characterized in that, The step of obtaining the geometric constraint parameters of each pixel position in the sparse image of the viewpoint includes: for each pixel position, if the corresponding geometric prior error is greater than or equal to the threshold hyperparameter, the corresponding geometric constraint parameter is one; otherwise, it is zero.

6. The Gaussian optimization method from a sparse perspective according to claim 4, characterized in that, The step of obtaining the position information of the fine Gaussian element based on the pixel importance value and the geometric constraint region mask corresponding to each pixel position includes: traversing all pixel positions, calculating the product of the corresponding pixel importance value and the geometric constraint parameter as a position parameter, and accumulating all position parameters to obtain an intermediate position value; for each pixel position, calculating the quotient of the position parameter and the intermediate position value to obtain the position probability indicating the fine Gaussian element, and performing back projection based on the position probability and the pixel position to determine the position information of the fine Gaussian element in the target scene.

7. The Gaussian optimization method from a sparse perspective according to claim 1, characterized in that, After updating the coarse 3D Gaussian based on all the fine Gaussian primitives, the method further includes: obtaining a preset protection cycle, and starting from the current iteration cycle, pruning and freezing the fine Gaussian primitives during the continuous preset protection cycle.

8. A Gaussian optimization device under sparse perspective, characterized in that, include: Initial Gaussian acquisition module: used to acquire sparse images corresponding to multiple viewpoints in the target scene, and perform point cloud initialization based on the sparse images to obtain a coarse three-dimensional Gaussian, wherein the coarse three-dimensional Gaussian includes at least one Gaussian element; The training iteration module is used to perform at least one training iteration to train the parameters of the Gaussian primitives in the coarse 3D Gaussian until the target Gaussian model is obtained. The iteration optimization module is used to render the corresponding rendered image based on the coarse 3D Gaussian at each viewpoint during the training iteration process if the iteration round is the optimization iteration round. The module calculates the reconstruction error based on the rendered image and the corresponding sparse image, performs semantic prior calculation based on the sparse image to obtain the semantic prior error, obtains the monocular depth map corresponding to the viewpoint, performs geometric prior calculation based on the monocular depth image to obtain the geometric prior error, calculates the pixel importance value based on the reconstruction error, the semantic prior error and the geometric prior error, determines the corresponding fine Gaussian primitive based on the pixel importance value for each viewpoint, and updates the coarse 3D Gaussian based on all the fine Gaussian primitives.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the Gaussian optimization method under sparse perspective as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the Gaussian optimization method under sparse perspective as described in any one of claims 1 to 7.

Images