Design method of generalizable plug-and-play 3D Gaussian splashing module

Abstract

The invention relates to the technical field of three-dimensional scene reconstruction, in particular to a generalizable plug-and-play 3D Gaussian splash module design method, which comprises the following steps: acquiring sparse point cloud data of a target image, and acquiring dense point cloud data of the target image by using the sparse point cloud data; inputting the dense point cloud data into a target 3D Gaussian splash to generate a target dense ellipsoid array; and training a pre-constructed deep network model by using the target dense ellipsoid array to obtain a trained model, inputting the sparse point cloud data into the trained model to generate a predicted dense ellipsoid array, and generating a target 3D Gaussian splashing module according to the predicted dense ellipsoid array. Therefore, the problems that in a 3DGS method in the related technology, a model needs to be independently trained for each scene, the cross-scene generalization ability is lacked, the rendering quality highly depends on the detail capturing ability of the SfM point cloud, and the rendering quality and performance are reduced are solved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional scene reconstruction, and in particular to a design method for a generalizable plug-and-play 3D Gaussian splash module. Background Art

[0002] 3D scene rendering has long been a challenging task in computer vision and is widely used in fields such as autonomous driving. In recent years, NeRF (Neural Radiance Field) has made significant progress in this field, enabling high-fidelity image generation without explicitly modeling 3D scenes, textures, or lighting. However, NeRF's random sampling and volume rendering are very computationally expensive, greatly affecting its performance and resulting in slow rendering speed. In order to meet the needs of real-time neural rendering, the 3DGS (3D Gaussian Splatter) method is used to represent the scene through a Gaussian ellipsoid, which combines the representation based on basic primitives and volume rendering technology to achieve real-time, high-quality rendering.

[0003] However, 3DGS in related technologies usually only processes a single scene and requires a separate model training for each scene. It lacks cross-scene generalization capabilities and the rendering quality is highly dependent on the detail capture capability of the SfM (Structure from Motion) point cloud. When the point cloud data is sparse or inaccurate, the rendering quality and performance will be significantly reduced, reducing the reliability of the rendering effect, which needs to be solved urgently. Summary of the invention

[0004] This application is based on the following problems and understandings made by the inventor:

[0005] 3D scene rendering has long been a challenging task in computer vision and is widely used in fields such as autonomous driving. In recent years, NeRF (Neural Radiance Field) has made significant progress in this field, enabling high-fidelity image generation without explicitly modeling 3D scenes, textures, or lighting. However, NeRF's random sampling and volume rendering are very computationally expensive, greatly affecting its performance and resulting in slow rendering speed. In order to address the needs of real-time neural rendering, the 3DGS (3D Gaussian Splatter) method has emerged, which represents the scene through a Gaussian ellipsoid, combining a representation based on basic primitives with volume rendering technology to achieve real-time, high-quality rendering.

[0006] However, 3DGS usually only processes a single scene and initializes the Gaussian ellipsoid parameters from a sparse but geometrically accurate point cloud through SfM (Structure from Motion), followed by iterative optimization and density control using a photometric loss function. This approach requires training a model for each scene separately and lacks cross-scene generalization capabilities. In addition, the rendering effect of 3DGS is highly dependent on the Gaussian ellipsoid heuristically initialized from the SfM point cloud. If the sparse point cloud cannot capture the details of the scene, this dependence will greatly affect the quality and performance of 3DGS rendering.

[0007] 3D scene reconstruction is one of the key requirements of intelligent driving systems and provides basic reasoning support for the visual module. Therefore, improving the quality and efficiency of scene reconstruction will provide more reliable support for other intelligent driving modules and promote the overall optimization of the system. Currently, most optimization methods for 3D Gaussian splashing are optimizations inside the 3D Gaussian. More efficient scene reconstruction is achieved by adding supervision information, changing the densification and pruning rules of the Gaussian ellipsoid, etc. However, most methods ignore the dependence of 3DGS on the quality of the initial Gaussian ellipsoid, which needs to be improved urgently.

[0008] The present application provides a design method for a generalizable plug-and-play 3D Gaussian splatter module to solve the problem that 3DGS in the related art usually only processes a single scene, requires a separate model training for each scene, lacks cross-scene generalization capability, and the rendering quality is highly dependent on the detail capture capability of the SfM point cloud. When the point cloud data is sparse or inaccurate, the rendering quality and performance will be significantly reduced, reducing the reliability of the rendering effect.

[0009] The first aspect of the present application provides a design method for a generalizable plug-and-play 3D Gaussian splash module, comprising the following steps: acquiring sparse point cloud data of a target image, and using the sparse point cloud data to acquire dense point cloud data of the target image; inputting the dense point cloud data into a target 3D Gaussian splash to generate a target dense ellipsoid array; using the target dense ellipsoid array to train a pre-built deep network model to obtain a trained model, and inputting the sparse point cloud data into the trained model to generate a predicted dense ellipsoid array, and generating a target 3D Gaussian splash module based on the predicted dense ellipsoid array.

[0010] Optionally, in one embodiment of the present application, the using the target dense ellipsoid array to train a pre-constructed deep network model includes: using the target dense ellipsoid array to determine the target true label of the pre-constructed deep network model; training the pre-constructed deep network model according to the target true label.

[0011] Optionally, in one embodiment of the present application, the sparse point cloud data is input into the trained model to generate a predicted dense ellipsoid array, including: extracting a first target dimension feature of each point based on the position information and color information of each point in the sparse point cloud data; performing feature concatenation on the first target dimension feature of each point and the target neighboring point to obtain a target dimension feature vector of each point; generating a second target dimension feature of each point based on the target dimension feature vector and the target neighborhood information; performing target processing on the second target dimension feature using a preset target fully connected layer to obtain a third target dimension feature of each point, and generating the predicted dense ellipsoid array based on the third target dimension feature.

[0012] Optionally, in one embodiment of the present application, generating the predicted dense ellipsoid array includes: regularizing the opacity in the predicted dense ellipsoid array to obtain a processed opacity; regularizing the covariance matrix in the predicted dense ellipsoid array to obtain a processed covariance matrix; inferring the positions of dense points in the predicted dense ellipsoid array using the positions in the predicted dense ellipsoid array; performing incremental processing on the colors in the predicted dense ellipsoid array to obtain incremental colors; and generating the predicted dense ellipsoid array using the processed opacity, the processed covariance matrix, the positions of the dense points and the incremental colors.

[0013] Optionally, in one embodiment of the present application, the predicted dense ellipsoid array is expressed as:

[0014] g n =(μ n ,Σ n ,α n ,C rgbn ) T*N

[0015] Among them, μ n is the mean vector, Σ n is the covariance matrix, α n is opacity, C rgbn is the color, T is a hyperparameter, and N is the number of initial sparse point clouds.

[0016] The second aspect of the present application provides a design device for a generalizable plug-and-play 3D Gaussian splash module, including: an acquisition module for acquiring sparse point cloud data of a target image, and using the sparse point cloud data to acquire dense point cloud data of the target image; a generation module for inputting the dense point cloud data into a target 3D Gaussian splash to generate a target dense ellipsoid array; a design module for training a pre-built deep network model using the target dense ellipsoid array to obtain a trained model, and inputting the sparse point cloud data into the trained model to generate a predicted dense ellipsoid array, and generating a target 3D Gaussian splash module based on the predicted dense ellipsoid array.

[0017] Optionally, in one embodiment of the present application, the design module includes: a determination unit for determining a target true label of the pre-constructed deep network model using the target dense ellipsoid array; and a training unit for training the pre-constructed deep network model according to the target true label.

[0018] Optionally, in one embodiment of the present application, the design module includes: an extraction unit, used to extract the first target dimension feature of each point according to the position information and color information of each point in the sparse point cloud data; a splicing unit, used to perform feature splicing on the first target dimension feature of each point and the target neighboring point to obtain the target dimension feature vector of each point; a first generation unit, used to generate the second target dimension feature of each point based on the target dimension feature vector and the target neighborhood information; an acquisition unit, used to perform target processing on the second target dimension feature using a preset target fully connected layer to obtain the third target dimension feature of each point, and generate the predicted dense ellipsoid array based on the third target dimension feature.

[0019] Optionally, in one embodiment of the present application, the design module includes: a first processing unit, used to regularize the opacity in the predicted dense ellipsoid array to obtain the processed opacity; a second processing unit, used to regularize the covariance matrix in the predicted dense ellipsoid array to obtain the processed covariance matrix; an inference unit, used to infer the positions of dense points in the predicted dense ellipsoid array using the positions in the predicted dense ellipsoid array; a third processing unit, used to incrementally process the colors in the predicted dense ellipsoid array to obtain the incremental colors; a second generation unit, used to generate the predicted dense ellipsoid array using the processed opacity, the processed covariance matrix, the positions of the dense points and the incremental colors.

[0020] Optionally, in one embodiment of the present application, the predicted dense ellipsoid array is expressed as:

[0021] g n = *μ n ,Σ n ,α n ,C rgbn ) T*N

[0022] Among them, μ n is the mean vector, Σ n is the covariance matrix, α n is opacity, C rgbn is the color, T is a hyperparameter, and N is the number of initial sparse point clouds.

[0023] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a method for designing a generalizable plug-and-play 3D Gaussian splatter module as described in the above embodiment.

[0024] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned generalized plug-and-play 3D Gaussian splash module design method.

[0025] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned method for designing a generalizable plug-and-play 3D Gaussian splash module.

[0026] The embodiment of the present application can use the sparse point cloud data of the target image to obtain the dense point cloud data of the target image, input the dense point cloud data into the target 3D Gaussian splash to generate a target dense ellipsoid array, use the target dense ellipsoid array to train the pre-built deep network model to obtain the trained model, and input the sparse point cloud data into the trained model to generate a predicted dense ellipsoid array, and generate a target 3D Gaussian splash module based on the predicted dense ellipsoid array, which effectively improves the cross-scene generalization ability of 3D scene reconstruction and improves the rendering quality and performance. Thus, the 3DGS method in the related art solves the problem that the model needs to be trained separately for each scene, lacks cross-scene generalization ability, and the rendering quality is highly dependent on the detail capture ability of the SfM point cloud, which reduces the rendering quality and performance.

[0027] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 A flowchart of a design method for a generalizable plug-and-play 3D Gaussian splash module provided according to an embodiment of the present application;

[0030] Figure 2 This is a functional diagram of a 3D Gaussian splash module according to a specific embodiment of the present application;

[0031] Figure 3 A schematic diagram of a network design of a 3D Gaussian splash module according to a specific embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of a design device for a generalizable plug-and-play 3D Gaussian splash module provided according to an embodiment of the present application;

[0033] Figure 5 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0035] The following describes a design method for a generalizable plug-and-play 3D Gaussian splash module of an embodiment of the present application with reference to the accompanying drawings. In view of the fact that the 3DGS method in the related art mentioned in the background technology center above needs to train a model separately for each scene, lacks cross-scene generalization capability, and the rendering quality is highly dependent on the detail capture capability of the SfM point cloud, which reduces the rendering quality and performance, the present application provides a design method for a generalizable plug-and-play 3D Gaussian splash module, in which the sparse point cloud data of the target image can be used to obtain the dense point cloud data of the target image, and the dense point cloud data is input into the target 3D Gaussian splash to generate a target dense ellipsoid array, and the pre-built deep network model is trained using the target dense ellipsoid array to obtain the trained model, and the sparse point cloud data is input into the trained model to generate a predicted dense ellipsoid array, and the target 3D Gaussian splash module is generated according to the predicted dense ellipsoid array, which effectively improves the cross-scene generalization capability of 3D scene reconstruction and improves the rendering quality and performance. This solves the problems in the related art that the 3DGS method needs to train a model separately for each scene, lacks cross-scene generalization capabilities, and the rendering quality is highly dependent on the detail capture capability of the SfM point cloud, which reduces the rendering quality and performance.

[0036] Specifically, Figure 1 A schematic flow chart of a design method for a generalizable plug-and-play 3D Gaussian splash module provided in an embodiment of the present application.

[0037] like Figure 1 As shown, the design method of the generalizable plug-and-play 3D Gaussian splash module includes the following steps:

[0038] In step S101, sparse point cloud data of a target image is acquired, and dense point cloud data of the target image is acquired using the sparse point cloud data.

[0039] It can be understood that the embodiment of the present application can obtain sparse point cloud data of the target image. For example, the embodiment of the present application can obtain sparse point cloud data based on the target image by using the SfM technology. The information in the sparse point cloud is represented as follows:

[0040] sp n =(p n ,C rgbn )N

[0041] Among them, sp n is a sparse point cloud, p n is the three-dimensional coordinates (x, y, z) of the point, C rgbn The color of the point.

[0042] Then, the dense point cloud data of the target image can be obtained based on the sparse point cloud data, thereby effectively improving the executability of the construction of the 3D Gaussian splash module.

[0043] For example, the embodiment of the present application can obtain dense point cloud data in the target image through the MVS (Multi-View Stereo) method, which can be expressed as:

[0044] dp n =(p n ,C rgbn ) D

[0045] Among them, dp n It is a dense point cloud.

[0046] In step S102 , the dense point cloud data is input into the target 3D Gaussian splash to generate a target dense ellipsoid array.

[0047] It can be understood that the embodiments of the present application can input dense point cloud data into the target 3D Gaussian splash to generate a target dense ellipsoid array, wherein, compared with sparse point cloud, inputting dense point cloud data into the target 3D Gaussian splash can generate a higher quality dense ellipsoid array, thereby effectively improving the quality of 3D scene reconstruction.

[0048] Among them, the dense ellipsoid array is expressed as:

[0049] g n =(μ n ,∑ n ,α n ,c rgbn ) D′

[0050] Among them, μ n is the mean vector, Σ n is the covariance matrix, α n is opacity, C rgbn For color.

[0051] In step S103, a pre-built deep network model is trained using a target dense ellipsoid array to obtain a trained model, and the sparse point cloud data is input into the trained model to generate a predicted dense ellipsoid array, and a target 3D Gaussian splash module is generated based on the predicted dense ellipsoid array.

[0052] In the embodiment of the present application, the target 3D Gaussian splash module is a generalizable plug-and-play 3D Gaussian splash module.

[0053] It is understandable that the embodiment of the present application can use the target dense ellipsoid array to train the pre-built deep network model to obtain a trained model, such as Figure 2 The GS-Net in the algorithm is used, and the pre-acquired sparse point cloud data is input into the GS-Net to generate a predicted dense ellipsoid array, that is, a higher quality initial Gaussian ellipsoid is generated, which effectively enhances the generalization ability across scenes, and then generates a target 3D Gaussian splash module according to the predicted dense ellipsoid array. The 3D Gaussian splash module can be used as a plug-in, which is small and lightweight, does not require high-performance hardware, and only occupies very small computing resources, which improves the 3D scene reconstruction quality of the basic 3DGS method and provides a good tool for related work on 3D scene reconstruction.

[0054] In one embodiment of the present application, the predicted dense ellipsoid array is expressed as:

[0055] g n =(μ n ,Σ n ,α n ,C rgbn ) T*N

[0056] Among them, μ n is the mean vector, Σ n is the covariance matrix, α n is opacity, C rgbn is the color, T is a hyperparameter, and N is the number of initial sparse point clouds.

[0057] Each point of the network input of the embodiment of the present application generates T Gaussian ellipsoids, which makes the representation of the local scene more dense and of higher quality. The light-colored arrows on the predicted ellipsoid array image represent this generation process.

[0058] The density of the input is increased by a factor T, where T is a hyperparameter. In the dataset of this application, T=5 can be selected to achieve optimal densification and effectively improve the performance of the model on a specific dataset.

[0059] It should be noted that if Figure 2 As shown, first of all, the purpose of this application is to provide a better quality initialization Gaussian ellipsoid for 3DGS (3D Gaussian point cloud projection, i.e. 3D Gaussian splashing). To achieve this goal, this application trains GS-Net. In order to introduce the working principle of GS-Net in detail, some basic parameter definitions are introduced below.

[0060] Among them, 3DGS uses innovative Gaussian primitives to represent the scene. A Gaussian primitive set is expressed as:

[0061]

[0062] Among them, μ n is the mean vector, Σ n is the covariance matrix, x n is the position vector of the current rendered pixel in 3D space. The covariance matrix is ​​decomposed into a rotation matrix R and a scaling matrix S, namely:

[0063] Σ n =RSS T R T

[0064] Among them, R is the rotation matrix and S is the scaling matrix.

[0065] In addition, the opacity α n and the color C calculated by the spherical harmonics SH coefficients rgbn It is defined for image rendering. The color of each pixel p is calculated by the following formula:

[0066]

[0067] Among them, α n is opacity, C rgbn For color.

[0068] Among them, in one embodiment of the present application, a pre-built deep network model is trained using a target dense ellipsoid array, including: using the target dense ellipsoid array to determine the target true label of the pre-built deep network model; training the pre-built deep network model according to the target true label.

[0069] During the actual implementation process, the embodiment of the present application can utilize the higher quality dense ellipsoid array generated in the above steps to determine the true labels for training the pre-built deep network model, thereby training the pre-built deep network model according to the true labels, effectively improving the accuracy of the model.

[0070] Optionally, in one embodiment of the present application, sparse point cloud data is input into a trained model to generate a predicted dense ellipsoid array, including: extracting the first target dimension features of each point based on the position information and color information of each point in the sparse point cloud data; performing feature concatenation on the first target dimension features of each point and the target neighboring points to obtain a target dimension feature vector for each point; generating a second target dimension feature for each point based on the target dimension feature vector and the target neighborhood information; performing target processing on the second target dimension features using a preset target fully connected layer to obtain a third target dimension feature for each point, and generating a predicted dense ellipsoid array based on the third target dimension feature.

[0071] As a possible implementation method, the trained model in the embodiment of the present application may include an encoder, a decoder, and activation functions applied to different Gaussian primitives, wherein the encoding module includes feature extraction and feature fusion, which are used to capture point-level features and local features, respectively.

[0072] Specifically, if Figure 3 As shown in the first step, the feature extraction module extracts the position of the point μ n and color C rgbn As input (6 dimensions), 16-dimensional features are output; then, the feature fusion module finds 3 nearest neighbors for each point and concatenates them with the features of the neighbors to form a 64-dimensional feature vector (16×4); finally, other feature extractions are used to further fuse the neighbor features, and finally 128-dimensional features are generated to determine the output of the encoder. By introducing neighborhood information, the model is able to more effectively represent the geometric structure of the scene, making the predicted ellipsoid more accurate. This design also enables GS-Net to adapt to various scene types.

[0073] Then, if Figure 3 As shown, the decoder in the embodiment of the present application processes the output of the encoder through three fully connected layers, gradually reducing the dimension from 128 dimensions to the final 14-dimensional output, matching the dimension required by the prediction ellipsoid array. Before obtaining the final output, each set of parameters in the prediction ellipsoid array can be regularized to improve the accuracy and convergence of the model prediction.

[0074] Optionally, in one embodiment of the present application, a predicted dense ellipsoid array is generated, including: regularizing the opacity in the predicted dense ellipsoid array to obtain a processed opacity; regularizing the covariance matrix in the predicted dense ellipsoid array to obtain a processed covariance matrix; using the positions in the predicted dense ellipsoid array to infer the positions of dense points in the predicted dense ellipsoid array; performing incremental processing on the colors in the predicted dense ellipsoid array to obtain incremental colors; and generating a predicted dense ellipsoid array using the processed opacity, the processed covariance matrix, the positions of dense points and the incremental colors.

[0075] In some embodiments, Figure 3 As shown, before obtaining the final output, the embodiment of the present application can predict the opacity α in the ellipsoid array. n Use Tanh activation function for regularization to ensure that the output value is in the range of [-1,1]; covariance matrix Σ n It consists of three-dimensional scaling and rotation (quaternion), where the scaling value is regularized by the Sigmoid activation function to ensure that it is within a reasonable range, and the rotation matrix is ​​normalized by the quaternion to ensure effectiveness; for the position μ n, without regularization, but using a relative offset-based learning method to infer the location of dense points. This method can make the location prediction more stable; Color C rgbn Use increment ΔC rgbn To more accurately represent the changes in the scene, thereby effectively improving the accuracy and convergence of the model prediction.

[0076] According to the design method of a generalizable plug-and-play 3D Gaussian splash module proposed in the embodiment of the present application, the sparse point cloud data of the target image can be used to obtain the dense point cloud data of the target image, and the dense point cloud data can be input into the target 3D Gaussian splash to generate a target dense ellipsoid array, and the pre-built deep network model is trained using the target dense ellipsoid array to obtain a trained model, and the sparse point cloud data is input into the trained model to generate a predicted dense ellipsoid array, and the target 3D Gaussian splash module is generated according to the predicted dense ellipsoid array, which effectively improves the cross-scene generalization capability of 3D scene reconstruction and improves the rendering quality and performance. Therefore, the problems that the 3DGS method in the related art needs to train a model separately for each scene, lacks cross-scene generalization capability, and the rendering quality is highly dependent on the detail capture capability of the SfM point cloud, which reduces the rendering quality and performance are solved.

[0077] Next, a design device for a generalizable plug-and-play 3D Gaussian splash module proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0078] Figure 4 It is a block diagram of a design device for a generalizable plug-and-play 3D Gaussian splash module according to an embodiment of the present application.

[0079] like Figure 4 As shown, the design device 10 of the generalizable plug-and-play 3D Gaussian splash module includes: an acquisition module 100 , a generation module 200 and a design module 300 .

[0080] Specifically, the acquisition module 100 is used to acquire sparse point cloud data of the target image, and use the sparse point cloud data to acquire dense point cloud data of the target image.

[0081] The generation module 200 is used to input the dense point cloud data into the target 3D Gaussian splash to generate a target dense ellipsoid array.

[0082] A design module 300 is used to train a pre-built deep network model using a target dense ellipsoid array to obtain a trained model, and to input sparse point cloud data into the trained model to generate a predicted dense ellipsoid array, and to generate a target 3D Gaussian splash module based on the predicted dense ellipsoid array.

[0083] Optionally, in one embodiment of the present application, the design module 300 includes: a determination unit and a training unit.

[0084] The determination unit is used to determine the target true label of the pre-built deep network model using the target dense ellipsoid array.

[0085] The training unit is used to train the pre-built deep network model according to the target true labels.

[0086] Optionally, in one embodiment of the present application, the design module 300 includes: an extraction unit, a splicing unit, a first generation unit and an acquisition unit.

[0087] Among them, the extraction unit is used to extract the first target dimension feature of each point based on the position information and color information of each point in the sparse point cloud data.

[0088] The concatenation unit is used to concatenate the first target dimension features of each point and the target neighboring points to obtain the target dimension feature vector of each point.

[0089] The first generating unit is used to generate a second target dimension feature for each point based on the target dimension feature vector and the target neighborhood information.

[0090] An acquisition unit is used to perform target processing on the second target dimension feature using a preset target fully connected layer to obtain the third target dimension feature of each point, and generate a predicted dense ellipsoid array based on the third target dimension feature.

[0091] Optionally, in one embodiment of the present application, the design module 300 includes: a first processing unit, a second processing unit, a speculation unit, a third processing unit and a second generation unit.

[0092] The first processing unit is used to perform regularization processing on the opacity in the predicted dense ellipsoid array to obtain processed opacity.

[0093] The second processing unit is used to perform regularization processing on the covariance matrix in the predicted dense ellipsoid array to obtain a processed covariance matrix.

[0094] The inference unit is used to infer the positions of dense points in the predicted dense ellipsoid array by using the positions in the predicted dense ellipsoid array.

[0095] The third processing unit is used to perform incremental processing on the colors in the predicted dense ellipsoid array to obtain incremental colors.

[0096] The second generating unit is used to generate a predicted dense ellipsoid array by using the processed opacity, the processed covariance matrix, the positions of the dense points and the incremented colors.

[0097] Optionally, in one embodiment of the present application, the predicted dense ellipsoid array is expressed as:

[0098] g n =(μ n ,Σ n ,α n ,C rgbn ) T*N

[0099] Among them, μ n is the mean vector, Σ n is the covariance matrix, α n is opacity, C rgbn is the color, T is a hyperparameter, and N is the number of initial sparse point clouds.

[0100] It should be noted that the above explanation of the embodiment of a design method for a generalizable plug-and-play 3D Gaussian splash module is also applicable to the design device for a generalizable plug-and-play 3D Gaussian splash module of this embodiment, which will not be repeated here.

[0101] According to the design device of a generalizable plug-and-play 3D Gaussian splash module proposed in the embodiment of the present application, the sparse point cloud data of the target image can be used to obtain the dense point cloud data of the target image, and the dense point cloud data can be input into the target 3D Gaussian splash to generate a target dense ellipsoid array, and the pre-built deep network model is trained using the target dense ellipsoid array to obtain a trained model, and the sparse point cloud data is input into the trained model to generate a predicted dense ellipsoid array, and the target 3D Gaussian splash module is generated according to the predicted dense ellipsoid array, which effectively improves the cross-scene generalization capability of 3D scene reconstruction and improves the rendering quality and performance. Therefore, the problems that the 3DGS method in the related art needs to train a model separately for each scene, lacks cross-scene generalization capability, and the rendering quality is highly dependent on the detail capture capability of the SfM point cloud, which reduces the rendering quality and performance are solved.

[0102] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0103] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0104] When the processor 502 executes the program, a design method for a generalizable plug-and-play 3D Gaussian splash module provided in the above embodiment is implemented.

[0105] Furthermore, the electronic device further comprises:

[0106] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0107] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0108] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0109] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0111] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0112] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for designing a generalizable plug-and-play 3D Gaussian splash module is implemented as described above.

[0113] This embodiment also provides a computer program product, including a computer program. When the computer program is executed, it is used to implement the above-mentioned design method of a generalizable plug-and-play 3D Gaussian splash module.

[0114] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0115] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0116] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0118] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0121] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A design method for a generalizable plug-and-play 3D Gaussian splash module, characterized in that: The following steps are involved: Acquire sparse point cloud data of a target image, and use the sparse point cloud data to acquire dense point cloud data of the target image; Inputting the dense point cloud data into a target 3D Gaussian splash to generate a target dense ellipsoid array; A pre-built deep network model is trained using the target dense ellipsoid array to obtain a trained model, and the sparse point cloud data is input into the trained model to generate a predicted dense ellipsoid array, and a target 3D Gaussian splash module is generated based on the predicted dense ellipsoid array.

2. The method according to claim 1, characterized in that The method of using the target dense ellipsoid array to train a pre-built deep network model includes: Determining a target true label of the pre-built deep network model using the target dense ellipsoid array; The pre-built deep network model is trained according to the target true label.

3. The method according to claim 1, characterized in that The step of inputting the sparse point cloud data into the trained model to generate a predicted dense ellipsoid array comprises: Extracting a first target dimension feature of each point according to the position information and color information of each point in the sparse point cloud data; Perform feature concatenation on the first target dimension features of each point and the target neighboring points to obtain a target dimension feature vector of each point; Based on the target dimension feature vector and the target neighborhood information, generating a second target dimension feature for each point; The second target dimension feature is target processed by using a preset target fully connected layer to obtain the third target dimension feature of each point, and the predicted dense ellipsoid array is generated based on the third target dimension feature.

4. The method according to claim 1, characterized in that: The generating the predicted dense ellipsoid array comprises: Regularizing the opacity in the predicted dense ellipsoid array to obtain processed opacity; Regularizing the covariance matrix in the predicted dense ellipsoid array to obtain a processed covariance matrix; Inferring the positions of dense points in the predicted dense ellipsoid array using the positions in the predicted dense ellipsoid array; Performing incremental processing on the colors in the predicted dense ellipsoid array to obtain incremental colors; The predicted dense ellipsoid array is generated using the processed opacity, the processed covariance matrix, the positions of the dense points and the incremented colors.

5. The method according to claim 1, characterized in that The predicted dense ellipsoid array is expressed as: g n =(μ n ,S n ,a n ,C rgbn ( T*N Among them, μ n is the mean vector, Σ n is the covariance matrix, α n is opacity, C rgbn is the color, T is a hyperparameter, and N is the number of initial sparse point clouds.

6. A design device for a generalizable plug-and-play 3D Gaussian splash module, characterized in that: include: An acquisition module, used to acquire sparse point cloud data of a target image, and to acquire dense point cloud data of the target image using the sparse point cloud data; A generation module, used for inputting the dense point cloud data into a target 3D Gaussian splash to generate a target dense ellipsoid array; A module is designed for training a pre-built deep network model using the target dense ellipsoid array to obtain a trained model, and the sparse point cloud data is input into the trained model to generate a predicted dense ellipsoid array, and a target 3D Gaussian splash module is generated according to the predicted dense ellipsoid array.

7. The device according to claim 6, characterized in that The design module includes: A determination unit, configured to determine a target true label of the pre-built deep network model using the target dense ellipsoid array; A training unit is used to train the pre-built deep network model according to the target true label.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for designing a generalizable plug-and-play 3D Gaussian splash module as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the design method of a generalizable plug-and-play 3D Gaussian splash module as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the design method of a generalizable plug-and-play 3D Gaussian splash module as described in any one of claims 1 to 5.

Images