Monocular reconstruction method and device, electronic equipment and storage medium
Through the combination of the Octrene data structure and neural network, the Gaussian sputtering model is used to perform three-dimensional reconstruction of monocular images, which solves the problem of low accuracy in monocular reconstruction technology and realizes efficient three-dimensional scene reconstruction and rendering.
Patent Information
- Application Number
- CN202411042932.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-17
AI Technical Summary
Monocular reconstruction technology lacks geometric information, resulting in low accuracy and difficulty in maintaining the detailed structure and texture of objects.
The octree data structure and neural network are used to reconstruct the monocular image by a Gaussian sputtering model, and the octree Gaussian sputtering model includes a Gaussian ellipsoid, which structures and constrains three-dimensional Gaussian sputtering to reduce the computational complexity.
It significantly reduces the computational complexity without sacrificing accuracy, improves the processing efficiency of monocular image data, and significantly improves the visual quality and rendering efficiency of three-dimensional scene reconstruction.
Smart Images
Figure CN119152109A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of image processing, and in particular to a monocular reconstruction method, device, electronic device and storage medium. Background Art
[0002] 3D reconstruction is a classic problem in computer vision and has a wide range of applications in virtual reality, autonomous driving, robotics, film and television entertainment, and other fields. Monocular reconstruction is the task of generating a complete 3D model of a scene using image information captured by a single camera. Compared with multi-camera reconstruction, monocular reconstruction has the advantages of simple equipment and low cost.
[0003] Compared with multi-view reconstruction, monocular reconstruction lacks geometric information and has lower accuracy. Therefore, how to maintain the detailed structure and texture of the object after monocular reconstruction is a technical problem that needs to be solved urgently in this field. Summary of the invention
[0004] In view of this, the purpose of the present disclosure is to propose a monocular reconstruction method, device, electronic device and storage medium, which can solve the existing problems in a targeted manner.
[0005] Based on the above purpose, in a first aspect, the present disclosure proposes a monocular reconstruction method, comprising: determining image features for a monocular image; using an octree data structure to represent the image features to obtain octree image features; inputting the octree image features into an octree neural network to perform Gaussian sputtering on the objects or scenes in the monocular image to obtain an octree Gaussian sputtering model, wherein the Gaussian sputtering model includes a Gaussian ellipsoid corresponding to each leaf node in the octree image features; and reconstructing the three-dimensional scene of the monocular image through the octree Gaussian sputtering model.
[0006] In a second aspect, a monocular reconstruction device is also provided, comprising: determining image features for a monocular image; using an octree data structure to represent the image features to obtain octree image features; inputting the octree image features into an octree neural network to perform Gaussian sputtering on the objects or scenes in the monocular image to obtain an octree Gaussian sputtering model, wherein the Gaussian sputtering model includes a Gaussian ellipsoid corresponding to each leaf node in the octree image features; and reconstructing the three-dimensional scene of the monocular image through the octree Gaussian sputtering model.
[0007] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.
[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement any method described in the first aspect.
[0009] In a fifth aspect, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor to implement any method described in the first aspect.
[0010] In general, the present disclosure has at least the following beneficial effects: by integrating the image features of the octree structure into three-dimensional Gaussian sputtering, it can be conveniently applied to subsequent octree neural networks, significantly reducing the computational complexity without sacrificing accuracy, and more efficiently processing monocular image data. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.
[0012] Figure 1 A flow chart of a monocular reconstruction method according to an embodiment of the present disclosure is shown;
[0013] Figure 2a Another flow chart of the monocular reconstruction method according to an embodiment of the present disclosure is shown;
[0014] Figure 2b A schematic diagram of a monocular reconstruction device according to an embodiment of the present disclosure is shown;
[0015] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;
[0016] Figure 4 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0017] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0019] Figure 1The monocular reconstruction method of the present disclosure is shown. In an embodiment of the present disclosure, the method includes:
[0020] Step S101, determining image features for a monocular image.
[0021] In this embodiment, the execution subject of the monocular reconstruction method (such as any terminal or server) can use a deep learning model to extract pixel-by-pixel features of the input monocular image. For example, the deep learning model can be U-Net.
[0022] Step S102, using an octree data structure to represent the image features, and obtaining octree image features.
[0023] In this embodiment, the execution subject may use an octree data structure to represent image features, and the result of the representation is an octree image feature.
[0024] Octree is a tree-shaped data structure, usually used for space partitioning, so that the three-dimensional space can be recursively divided into eight sub-areas. Each node in the octree represents a spatial cube, and if the data inside the cube is not uniform, it is further subdivided into eight smaller cubes until the required level of detail is reached or the data is uniform.
[0025] Step S103, inputting the octree image features into an octree neural network to perform Gaussian sputtering on the object or scene in the monocular image to obtain an octree Gaussian sputtering model, wherein the Gaussian sputtering model includes a Gaussian ellipsoid corresponding to each leaf node in the octree image features.
[0026] In this embodiment, the octree neural network may be any neural network model that can predict the octree image features to obtain an octree Gaussian sputtering model, such as a convolutional neural network or a residual neural network.
[0027] The octree neural network is a convolutional neural network that can be used to process the octree data structure. The octree image features can be used as the input of the image, and the octree Gaussian sputtering model can be used as the output of the image to train the octree Gaussian sputtering model to be trained, so as to obtain an octree neural network that can predict the octree Gaussian sputtering model.
[0028] In terms of appearance, dense Gaussian ellipsoids are stored “one-to-one” in each child node of the deepest layer of the octree.
[0029] In traditional neural networks, processing large-scale or high-dimensional three-dimensional data often requires a lot of computing resources and time; at the same time, the three-dimensional Gaussian sputtering model has the characteristics of discreteness, unstructuredness, and high dimensionality, which makes it face huge challenges as an object of neural network prediction. In this regard, the present invention proposes a new representation "Octree-based Gaussian Splatting", which uses the compactness and multi-level characteristics of the octree to structure the three-dimensional Gaussian sputtering.
[0030] In the octree Gaussian sputtering model, the Gaussian ellipsoid G(x) is represented by a 3D covariance matrix Σ and its center position μ:
[0031]
[0032] Where T represents the transpose and x represents the three-dimensional position of the Gaussian ellipsoid.
[0033] To ensure the positive definiteness of the covariance matrix during the optimization process, Σ is further decomposed into a matrix S representing the scale of the Gaussian ellipsoid and a matrix R representing the rotation:
[0034] Σ=RSS T R T
[0035] in, Represented as a 3D vector, R∈SO(3) is represented as a quaternion. The Gaussian ellipsoid also contains other learnable attribute parameters, including opacity o∈(0,1), and spherical harmonic function coefficients k representing anisotropic color.
[0036] Step S104: reconstructing the three-dimensional scene of the monocular image through the octree Gaussian sputtering model.
[0037] In this embodiment, the execution subject can use various methods to reconstruct the three-dimensional scene of the monocular image through the octree Gaussian sputtering model. For example, the three-dimensional scene of the monocular image can be obtained by processing the octree Gaussian construction model using the model of the reconstructed three-dimensional scene.
[0038] This embodiment integrates the image features of the octree structure into three-dimensional Gaussian sputtering, which can be easily applied to the subsequent octree neural network, significantly reducing the computational complexity without sacrificing accuracy, and more efficiently processing monocular image data.
[0039] In some optional implementations of any embodiment of the present disclosure, performing Gaussian sputtering on the object or scene in the monocular image to obtain an octree Gaussian sputtering model includes: using a PBR material to generate a Gaussian sputtering model of the object or scene in the monocular image, and obtaining an octree Gaussian sputtering model with a Gaussian ellipsoid as the PBR material, wherein the material properties of the PBR material include at least one of the following: albedo color, normal, roughness, and metallicity.
[0040] The above execution entities can model PBR (Physically Based Rendering) materials in a generalizable reconstruction method. It can be widely used in downstream tasks such as relighting and material editing. Specifically, the octree Gaussian sputtering model not only contains the albedo color of the object, but also contains material-related attributes such as normal, roughness, and metalness. Given a camera perspective, rendering a view of related attributes can obtain the final output image through deferred rendering.
[0041] In some optional implementations of any one of the embodiments of the present disclosure, the image features are pixel-by-pixel features; the image features are represented by an octree data structure to obtain octree image features, including: determining a monocular depth for the monocular image; determining a monocular point cloud containing the image features based on the monocular depth and the image features; and determining the octree image features through the monocular point cloud.
[0042] In these optional implementations, the above-mentioned execution entity can determine the octree image features corresponding to the monocular point cloud through a calculation formula or a preset model.
[0043] In some optional implementations of any embodiment of the present disclosure, the monocular point cloud is a three-dimensional Gaussian sputtering dense point cloud; determining the octree image features through the monocular point cloud includes: converting the three-dimensional Gaussian sputtering dense point cloud into a multi-layer octree image feature by marking the child nodes.
[0044] Optionally, the monocular point cloud is a three-dimensional Gaussian sputtering dense point cloud; determining the octree image features through the monocular point cloud includes: converting the three-dimensional Gaussian sputtering dense point cloud into a multi-layer octree image feature by marking, and the marked content includes whether the sub-node feature has a next-level sub-node.
[0045] In the converted multi-layer octree image features, each child node feature has a corresponding mark, and the marked content is whether the child node feature has a next-layer child node.
[0046] In some optional implementations of any embodiment of the present disclosure, the octree neural network includes an encoder and a decoder; the inputting the octree image features into the octree neural network includes: inputting the octree image features into the encoder to limit the convolution calculation to non-empty sparse voxels, and using an octree to organize the non-empty sparse voxels to determine the child nodes corresponding to the non-empty sparse voxels.
[0047] The core of the octree neural network is to limit the convolution calculation to non-empty sparse voxels, and effectively organize and index these sparse voxels using the octree, thereby achieving spatial locality and hierarchy and greatly improving efficiency.
[0048] In some optional implementations of any embodiment of the present disclosure, reconstructing the three-dimensional scene of the monocular image through the octree Gaussian sputtering model includes: sorting the octree Gaussian sputtering model according to depth, projecting it into a two-dimensional space, and obtaining two-dimensional Gaussian ellipsoid information; mixing the opacity and color of the two-dimensional Gaussian ellipsoid information to reconstruct the three-dimensional scene of the monocular image at each pixel.
[0049] In these optional implementations, the process of reconstructing a three-dimensional scene using three-dimensional Gaussian sputtering is as follows: Figure 1 As shown. The parameter optimization and efficient rendering of three-dimensional Gaussian sputtering benefit from differentiable rasterization rendering: First, calculate the two-dimensional covariance matrix of the camera space and project the three-dimensional Gaussian into the two-dimensional space:
[0050] Σ′=JVΣV T J T
[0051] Among them, J is the Jacobian matrix of the approximate projection transformation, and V is the camera extrinsic parameter matrix.
[0052] Then, based on an efficient GPU acceleration algorithm, the two-dimensional Gaussians are sorted and mixed by depth to calculate the color of each pixel in the image:
[0053]
[0054] Among them, c i represents the i-th three-dimensional Gaussian G i The color, α i =o i G i ′,o i and G i ′ respectively represent G i opacity and 2D drop shadow.
[0055] In some optional implementations of any embodiment of the present disclosure, the method includes: during the training process of the octree neural network, three-dimensional Gaussian sputtering can adaptively control the density of the Gaussian ellipsoid, wherein controlling the density of the Gaussian ellipsoid includes: increasing the density of the Gaussian ellipsoid by splitting or replicating the Gaussian ellipsoid, or reducing the density of the Gaussian ellipsoid by periodically removing Gaussian ellipsoids that are close to transparent.
[0056] Nearly transparent means low transparency or high opacity.
[0057] During the optimization process, 3D Gaussian sputtering can also adaptively control the density of Gaussian ellipsoids, for example, splitting or replicating Gaussian spheres in 3D spaces where the reconstruction is not good enough, regularly removing Gaussian spheres that are close to transparent, and so on.
[0058] After Gaussian density adjustment and iterative optimization, 3D Gaussian sputtering can reconstruct the geometry and appearance of 3D scenes with high quality and render 1080P high-definition images at a speed of more than 100FPS. Compared with similar 3D reconstruction methods based on differentiable rendering, such as neural radiation field, 3D Gaussian sputtering has achieved the best visual quality and rendering efficiency.
[0059] The disclosed embodiment provides a monocular reconstruction device, which is used to execute the monocular reconstruction method described in the above embodiment. As shown in Figure 2, the device includes: a feature determination unit 201, configured to determine image features for a monocular image; a representation unit 202, configured to use an octree data structure to represent the image features and obtain octree image features; a prediction unit 203, configured to input the octree image features into an octree neural network to perform Gaussian sputtering on the object or scene in the monocular image to obtain an octree Gaussian sputtering model, wherein the Gaussian sputtering model includes a Gaussian ellipsoid corresponding to each leaf node in the octree image features; and a reconstruction unit 204, configured to reconstruct the three-dimensional scene of the monocular image through the octree Gaussian sputtering model.
[0060] The monocular reconstruction device provided in the above-mentioned embodiment of the present disclosure and the monocular reconstruction method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.
[0061] The embodiments of the present disclosure also provide an electronic device corresponding to the monocular reconstruction method provided in the above embodiments, so as to execute the above monocular reconstruction method. The embodiments of the present disclosure are not limited.
[0062] Please refer to Figure 3 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 3As shown, the electronic device 30 includes: a processor 300, a memory 301, a bus 302 and a communication interface 303, and the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and the processor 300 executes the method provided in any of the aforementioned embodiments of the present disclosure when running the computer program.
[0063] The memory 301 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0064] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store a program, and the processor 300 executes the program after receiving an execution instruction. The monocular reconstruction method disclosed in any implementation of the aforementioned embodiment of the present disclosure may be applied to the processor 300, or implemented by the processor 300.
[0065] The processor 300 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 300. The above processor 300 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.
[0066] The electronic device provided by the embodiment of the present disclosure and the monocular reconstruction method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.
[0067] The present disclosure also provides a computer-readable storage medium corresponding to the monocular reconstruction method provided in the above embodiment. Figure 4 The computer-readable storage medium shown is a CD 40 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the monocular reconstruction method provided by any of the aforementioned embodiments will be executed.
[0068] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.
[0069] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the monocular reconstruction method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0070] It should be noted that:
[0071] In the above text, the term "comprises", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.
[0073] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, which are only specific implementation modes of the present disclosure. However, the present disclosure is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the purpose of the present disclosure and the scope of protection of the claims, all of which are within the protection of the present disclosure.
Claims
1. A monocular reconstruction method, It is characterized in that include: Determine image features for a monocular image; An octree data structure is used to represent the image features, thereby obtaining an octree image feature; Inputting the octree image features into an octree neural network to perform Gaussian sputtering on the object or scene in the monocular image to obtain an octree Gaussian sputtering model, wherein the Gaussian sputtering model includes a Gaussian ellipsoid corresponding to each leaf node in the octree image features; The three-dimensional scene of the monocular image is reconstructed through the octree Gaussian sputtering model.
2. The method according to claim 1, It is characterized in that The performing Gaussian sputtering on the object or scene in the monocular image to obtain an octree Gaussian sputtering model includes: A PBR material is used to generate a Gaussian sputtering model of the object or scene in the monocular image, and an octree Gaussian sputtering model with a Gaussian ellipsoid as the PBR material is obtained, wherein the material properties of the PBR material include at least one of the following: albedo color, normal, roughness, and metallicity.
3. The method according to claim 1, It is characterized in that The image features are pixel-by-pixel features; The octree data structure is used to represent the image features, and the octree image features are obtained, including: Determining a monocular depth for the monocular image; Determining a monocular point cloud including the image feature according to the monocular depth and the image feature; The octree image features are determined through the monocular point cloud.
4. The method according to claim 3, It is characterized in that The monocular point cloud is a three-dimensional Gaussian sputtering dense point cloud; Determining the octree image features through the monocular point cloud includes: The three-dimensional Gaussian sputtering dense point cloud is converted into a multi-layer octree image feature by a marking method, and the marked content includes whether the sub-node feature has a sub-node of the next layer.
5. The method according to claim 1, It is characterized in that The octree neural network includes an encoder and a decoder; The step of inputting the octree image features into the octree neural network comprises: The octree image features are input into the encoder to limit the convolution calculation to non-empty sparse voxels, and the non-empty sparse voxels are organized using the octree to determine the child nodes corresponding to the non-empty sparse voxels.
6. The method according to claim 1, It is characterized in that The method of reconstructing the three-dimensional scene of the monocular image by using the octree Gaussian sputtering model includes: Projecting the octree Gaussian sputtering model into a two-dimensional space to obtain two-dimensional Gaussian ellipsoid information; The two-dimensional Gaussian ellipsoid information is sorted according to depth, and opacity and color are mixed on the sorting result to reconstruct the three-dimensional scene of the monocular image at each pixel.
7. The method according to claim 1, It is characterized in that The method comprises: During the training process of the octree neural network, three-dimensional Gaussian sputtering can adaptively control the density of the Gaussian ellipsoid, wherein the control of the density of the Gaussian ellipsoid includes: increasing the density of the Gaussian ellipsoid by splitting or replicating the Gaussian ellipsoid, or reducing the density of the Gaussian ellipsoid by regularly removing Gaussian ellipsoids that are close to transparent.
8. A monocular reconstruction device, It is characterized in that include: A feature determination unit is configured to determine image features for a monocular image; A representation unit is configured to use an octree data structure to represent the image feature and obtain an octree image feature; A prediction unit is configured to input the octree image features into an octree neural network to perform Gaussian sputtering on the object or scene in the monocular image to obtain an octree Gaussian sputtering model, wherein the Gaussian sputtering model includes a Gaussian ellipsoid corresponding to each leaf node in the octree image features; The reconstruction unit is configured to reconstruct the three-dimensional scene of the monocular image through the octree Gaussian sputtering model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that The processor runs the computer program to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that The program is executed by a processor to implement the method according to any one of claims 1 to 7.