Indoor three-dimensional object reconstruction method and apparatus, computer device and storage medium

Through improved NeRF model and monocular camera technology, the problems of 2D local feature matching errors and increased sampling time in 3D reconstruction are solved, and faster and more efficient 3D reconstruction is achieved, expanding the scope of application and reducing resource requirements.

WO2024230151A1PCT designated stage expired Publication Date: 2024-11-14SHENZHEN INST OF ADVANCED TECH

Patent Information

Application Number
PCT/CN2023/137200
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2023-12-07
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In the existing three-dimensional reconstruction technology, the SFM algorithm based on multiple images taken at different angles has problems such as the two-dimensional local feature matching error and the MVS algorithm lacks inference ability. The NeRF method is limited by binocular or RGB-D cameras and single-eye The camera is studied less and the sampling scheme increases the three-dimensional reconstruction time.

Method used

A monocular camera is used to collect indoor scene images from multiple different perspectives, and the improved NeRF model uses the MLP network to output bulk density and RGB colors. The combined rendering algorithm renders pictures from different perspectives, and uses hierarchical pixels of rough and delicate networks. Sampling reduces the number of samples and generates a three-dimensional reconstruction image.

Benefits of technology

It reduces the three-dimensional reconstruction time, expands the scope of application of NeRF, avoids the complex manual modeling process in traditional methods, and reduces the demand for storage space and computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023137200_14112024_PF_FP_ABST
    Figure CN2023137200_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to an indoor three-dimensional object reconstruction method and apparatus, a computer device and a storage medium. The method comprises: using a monocular camera to acquire indoor scene images under a plurality of different visual angles; inputting the indoor scene images into a trained neural radiation field model, the neural radiation field model outputting the volume density and the RGB color of each indoor scene image by means of an MLP network, and using a volume rendering algorithm to render, according to the volume density and the RGB color, a scene picture under each visual angle; and fusing the scene pictures under the plurality of different visual angles to generate three-dimensional reconstructed images of indoor scenes. The embodiments of the present application reduce the sampling times of a fitting space light field, thus reducing the time for three-dimensional reconstruction. The embodiments of the present application do not require storing a large amount of scene information, and can avoid the complex process needing manual modeling in traditional methods, thereby greatly reducing storage space and computing resources.
Need to check novelty before this filing date? Find Prior Art

Description

Indoor three-dimensional object reconstruction method, device, computer equipment and storage medium Technical Field

[0001] The present application belongs to the field of three-dimensional reconstruction technology, and in particular relates to a method, apparatus, computer equipment, and storage medium for reconstructing indoor three-dimensional objects. Background Art

[0002] 3D reconstruction refers to the creation of mathematical models suitable for computer representation and processing of three-dimensional objects. It serves as the foundation for processing, manipulating, and analyzing the properties of objects in a computer environment. It is also a key technology for creating virtual reality representations of the objective world within computers. After more than a decade of development, 3D reconstruction has achieved remarkable success. Because 3D images contain more information than 2D images and more intuitively reflect the real world, using computers to restore the 3D information of objects and construct 3D models of real scenes has become an essential step before in-depth research in many fields. For example, 3D CT images are used in medical imaging to diagnose medical conditions, and 3D restoration of cultural relics is performed in cultural heritage conservation.

[0003] Currently, researchers mainly use the following three methods to obtain 3D models of indoor objects:

[0004] 1. Traditional geometric modeling technology: Geometric modeling technology is the most mature and can be used to obtain very accurate three-dimensional models, but it is time-consuming and labor-intensive, and there are still differences from the real scene.

[0005] Second, three-dimensional modeling is performed using computer vision theory based on multiple images taken from different angles. Representative methods of this technology include the SFM (Structure from motion) and MVS (Multi-view stereo) algorithms. When using the SFM algorithm to reconstruct the shape of indoor three-dimensional objects, the matching of two-dimensional local features often results in serious errors in practice due to the small overlap between different views, missing object textures, or light reflections on the objects. The MVS algorithm, however, lacks inference capabilities and cannot solve the problem of completing three-dimensional structures when input information is limited. This method also suffers from issues such as long processing times and complex operations. For many irregular natural or man-made objects, the modeling results are still somewhat different from the real scene.

[0006] Third, use a 3D scanning device to scan the target and then reconstruct a 3D model of the target. This method is point cloud-based 3D reconstruction. Currently, NeRF (Neural Radiance Fields) is an emerging 3D reconstruction method that uses a neural network model to represent the radiance field of a 3D scene, thereby achieving 3D scene reconstruction. NeRF can use 2D posed images as supervision to represent complex 3D scenes. Currently, most 3D modeling methods based on NeRF use binocular cameras or RGB-D cameras to capture images, and then use the known depth information after the capture to model indoor 3D objects. However, there is less research on monocular cameras, which limits the scope of application of NeRF methods. In addition, current NeRF methods use a coarse and fine dual network cascade sampling scheme to fit the spatial light field. In this sampling scheme, it is equivalent to performing two uniform samplings and one non-uniform sampling, which will increase the time consumed for 3D reconstruction to a certain extent.

[0007] Summary of the Invention

[0008] The present application provides a method, apparatus, computer device, and storage medium for reconstructing indoor three-dimensional objects, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0009] In order to solve the above problems, this application provides the following technical solutions:

[0010] A method for reconstructing indoor three-dimensional objects, comprising:

[0011] Use a monocular camera to capture images of indoor scenes from multiple different perspectives;

[0012] Input the indoor scene image into the trained neural radiation field model, the neural radiation field model outputs the volume density and RGB color of each indoor scene image through the MLP network, and uses the volume rendering algorithm to render the scene image at each perspective according to the volume density and RGB color;

[0013] The scene images from multiple different perspectives are fused to generate a 3D reconstructed image of the indoor scene.

[0014] The technical solution adopted in the embodiment of the present application also includes: using a monocular camera to capture indoor scene images from multiple different perspectives also includes: recording the position and direction of each indoor scene image.

[0015] The technical solution adopted in the embodiment of the present application further includes: after using the monocular camera to capture multiple indoor scene images from different perspectives, it also includes:

[0016] The indoor scene image is preprocessed to obtain a corrected indoor scene image.

[0017] The technical solution adopted in the embodiment of the present application also includes: the pre-processing of the indoor scene image is specifically:

[0018] Obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the monocular camera;

[0019] The indoor scene image is subjected to distortion removal processing according to the intrinsic parameter matrix and the extrinsic parameter matrix to obtain a corrected indoor scene image.

[0020] The technical solution adopted in the embodiment of the present application further includes: before inputting the indoor scene image into the trained neural radiation field model, it also includes:

[0021] The corrected indoor scene image is input into the improved NeRF for training. The improved NeRF outputs the volume density and RGB color of each pixel in the indoor scene image through the MLP network. Hierarchical pixel sampling of the coarse network and the fine network is performed according to the volume density and RGB color. Three-dimensional reconstruction of the indoor scene image is performed according to the hierarchical pixel sampling result. The improved NeRF is optimized according to the three-dimensional reconstruction result to obtain a trained neural radiation field model.

[0022] The technical solution adopted in the embodiment of the present application also includes: the training process of the improved NeRF is specifically as follows:

[0023] For each indoor scene image r, it is processed into a 5D vector input The 5D vector input includes the 3D position coordinate x and the viewing direction d of each indoor scene image in the Cartesian coordinate system; the 5D vector is converted into Perform uniform sampling with a sampling interval of Δt, obtain the distribution of valid sampling positions, fit the probability density function of the valid sampling positions, and output the uniform sampling results

[0024] Based on the probability density function, the 5D vector is centered at the position with high probability through a delicate network. Perform non-uniform sampling and output non-uniform sampling results in, represents the bulk density;

[0025] The uniform sampling result of the rough network Non-uniform sampling results with delicate network output Mix and get the final RGB color output result C f (r).

[0026] The technical solution adopted in the embodiment of the present application also includes: the optimization of the improved NeRF according to the three-dimensional reconstruction result is specifically as follows:

[0027] The mean square error of the uniform sampling result of the coarse network and the non-uniform sampling result of the fine network is calculated at each pixel as the loss function to simultaneously optimize the coarse network and the fine network:

[0028] in, is the RGB color sampling result of the rough network for light r, C gt (r) is the true RGB color value of light r.

[0029] Another technical solution adopted in the embodiment of the present application is: a method and device for reconstructing indoor three-dimensional objects, comprising:

[0030] Image acquisition module: used to acquire indoor scene images from multiple different perspectives. The image acquisition module is a monocular camera;

[0031] Image rendering module: used to input the indoor scene image into the trained neural radiation field model, the neural radiation field model outputs the volume density and RGB color of each indoor scene image through the MLP network, and uses the volume rendering algorithm to render the scene image at each perspective according to the volume density and RGB color;

[0032] Image fusion module: used to fuse scene images from multiple different perspectives to generate a 3D reconstructed image of the indoor scene.

[0033] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein:

[0034] The memory stores program instructions for implementing the indoor three-dimensional object reconstruction method;

[0035] The processor is configured to execute the program instructions stored in the memory to control a method for reconstructing indoor three-dimensional objects.

[0036] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the indoor three-dimensional object reconstruction method.

[0037] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: the indoor three-dimensional object reconstruction method, device, computer equipment and storage medium of the embodiments of the present application collect indoor scene images from multiple perspectives through a monocular camera, and implicitly represent the scene as a continuous 5D radiation field through the improved NeRF model, and use the MLP network to output the volume density of each indoor scene image and the RGB color related to the input image, render pictures from different perspectives according to the volume density and RGB color, and fuse the pictures from multiple perspectives to generate a three-dimensional image of the indoor scene. The embodiments of the present application reduce the number of sampling times for fitting the spatial light field by improving the original NeRF, thereby reducing the three-dimensional reconstruction time. Secondly, the embodiments of the present application do not need to store a large amount of scene information, can avoid the complex process of manual modeling required in traditional methods, and greatly reduce storage space and computing resources. In addition, the embodiments of the present application use a monocular camera for image acquisition, which expands the scope of application of NeRF. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a flow chart of a method for reconstructing an indoor three-dimensional object according to an embodiment of the present application;

[0039] Figure 2 is an introduction to the original NeRF (Neural Radiance Field) principle and a schematic diagram of the rendering process;

[0040] FIG3 is a schematic diagram of the improved NeRF interpretation of model input data according to an embodiment of the present application;

[0041] FIG4 is a schematic diagram of the network structure of the improved NeRF according to an embodiment of the present application;

[0042] FIG5 is a schematic diagram of the structure of an apparatus for indoor three-dimensional object reconstruction method according to an embodiment of the present application;

[0043] FIG6 is a schematic diagram of the structure of a computer device according to an embodiment of the present application;

[0044] FIG7 is a schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications also change accordingly. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or computer device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products, or computer devices.

[0047] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] Please refer to FIG1 , which is a flow chart of a method for reconstructing an indoor three-dimensional object according to an embodiment of the present application. The method for reconstructing an indoor three-dimensional object according to an embodiment of the present application comprises the following steps:

[0049] S100: Use a monocular camera to capture indoor scene images from multiple perspectives and record the position and orientation of each indoor scene image;

[0050] S110: Preprocessing the indoor scene image to obtain a corrected indoor scene image;

[0051] In this step, more accurate three-dimensional information can be obtained by preprocessing the indoor scene image. The preprocessing of the indoor scene image specifically includes:

[0052] S111: Obtain the intrinsic parameter matrix and extrinsic parameter matrix of the monocular camera;

[0053] Among them, the intrinsic parameter matrix is ​​the parameters such as focal length and pixel size related to the camera's own characteristics; the extrinsic parameter matrix is ​​the parameters such as the camera's position and rotation direction in the world coordinate system.

[0054] S112: performing distortion removal processing on the indoor scene image according to the intrinsic parameter matrix and the extrinsic parameter matrix to obtain a corrected indoor scene image.

[0055] S120: Input the corrected indoor scene image into the improved NeRF for training. The NeRF outputs the volume density and RGB color of each pixel in the indoor scene image through the MLP network. Hierarchical pixel sampling of the coarse network and the fine network is performed based on the volume density and RGB color. The indoor scene is reconstructed 3D based on the hierarchical pixel sampling results. The NeRF is optimized based on the 3D reconstruction results to obtain a trained neural radiation field model.

[0056] In this step, please also refer to Figure 2, which provides an introduction to the original NeRF (Neural Radiance Field) principle and a schematic diagram of the rendering process. As shown in the multi-level sampling sections (c) and (d) of Figure 2, the NeRF rendering process is computationally intensive, requiring many points to be sampled for each ray (the line connecting the focus and the pixel). However, in reality, most of the area along each ray is empty or occluded, and thus has little contribution to the final 3D reconstructed color. The original NeRF method repeatedly samples these empty or occluded areas, reducing the efficiency of 3D reconstruction.

[0057] Based on this, the embodiment of the present application adopts the improved NeRF to represent the scene. The input data of the improved NeRF is a function of a 5D vector. As shown in Figure 3, it is a schematic diagram of the interpretation of the model input data by the improved NeRF in the embodiment of the present application. The 5D vector includes the 3D position coordinates x = (x, y, z) of each indoor scene image in the Cartesian coordinate system and the viewing direction Where θ represents the angle between the straight line from point p to the origin o and the z-axis. Represents the angle between the line oq formed by the projection point q and point o from point p to the xoy plane and the x-axis. A fully connected MLP neural network is used to implicitly learn a static three-dimensional scene. During the model training process, the 3D position coordinates and viewing direction of the corrected indoor scene image are input into the improved NeRF model. The model outputs the volume density σ of each indoor scene image and the RGB color related to the input image. The output volume density σ and RGB color are then projected into the image using the volume rendering algorithm, and images from different perspectives are rendered. The images from multiple perspectives are then fused to generate a three-dimensional image of the indoor scene image. At the same time, the network parameters are adjusted by optimizing the loss function so that the scene output by the model can be as close as possible to the real scene, in order to obtain the optimized neural radiation field model.

[0058] Specifically, please refer to FIG4 , which is a schematic diagram of the improved NeRF network structure of the embodiment of the present application. For each indoor scene image r, it is first processed into a 5D vector input d is the direction of all sampling positions; then the 5D vector is Perform uniform sampling with a sampling interval of Δt, obtain the distribution of valid sampling positions, fit the probability density function of the valid sampling positions, and output the uniform sampling results Based on the probability density function, the 5D vector is centered at the position with high probability through a delicate network. Perform denser non-uniform sampling and output the non-uniform sampling results in, After the two samplings are completed, the uniform sampling result of the coarse network is mixed with the non-uniform sampling result output by the fine network to obtain the final RGB color output result C f (r), thereby reducing the number of original NeRF sampling from three to two, speeding up NeRF 3D reconstruction. Specifically, the improved NeRF sampling derivation formula is as follows:

[0059] In formula (1), sort is the sorting function, x k represents the kth sampling position, represents the sampling position under uniform sampling, Indicates non-uniform sampling position, each x k Corresponding to their respective sampling results, the two sampling results are then merged to calculate the final RGB color:

[0060] In the above formula, x k -x k-1 is the distance between the kth sampling position and the previous sampling position, and the value of x0 is 0. σ k is x k The volume density at the location, c k is x k Sampling results of the location.

[0061] Finally, the mean square error of the two sampling results of the coarse network and the fine network is calculated at each pixel as the loss function to simultaneously optimize the coarse network and the fine network:

[0062] in, is the RGB color sampling result of the rough network for light r, C gt (r) is the true RGB color value of light r.

[0063] S130: Inputting the indoor scene image to be reconstructed into the trained neural radiation field model, and outputting a three-dimensional reconstructed image of the indoor scene through the neural radiation field model;

[0064] In this step, after obtaining a trained neural radiation field model, it can be used to reconstruct a 3D model. Specifically, for a new perspective, the neural radiation field model can be used to predict the volume density and RGB color of each pixel at that perspective. A volume rendering algorithm is then used to render the scene image at each perspective based on the volume density and RGB color. Finally, the scene images from multiple perspectives are fused to obtain a complete 3D reconstructed image of the indoor scene.

[0065] Based on the above, the indoor three-dimensional object reconstruction method of the second embodiment of the present application uses a monocular camera to collect indoor scene images from multiple different perspectives, and implicitly represents the scene as a continuous 5D radiation field through the improved NeRF model, and uses the MLP network to output the volume density and RGB color of each indoor scene image, and renders pictures from different perspectives according to the volume density and RGB color, and fuses the pictures from multiple perspectives to generate a three-dimensional reconstructed image of the indoor scene. The embodiment of the present application reduces the number of sampling times for fitting the spatial light field by improving the original NeRF, thereby reducing the three-dimensional reconstruction time. Secondly, the embodiment of the present application does not need to store a large amount of scene information, can avoid the complex process of manual modeling required in traditional methods, and greatly reduces storage space and computing resources. In addition, the embodiment of the present application uses a monocular camera for image acquisition, which expands the scope of application of NeRF.

[0066] Please refer to FIG5 , which is a schematic diagram of the structure of an indoor three-dimensional object reconstruction method apparatus according to an embodiment of the present application. The indoor three-dimensional object reconstruction method apparatus 40 according to the embodiment of the present application comprises:

[0067] Image acquisition module 41: used to acquire indoor scene images from multiple different perspectives, the image acquisition module is a monocular camera;

[0068] Image rendering module 42: used to input the indoor scene image into the trained neural radiation field model, the neural radiation field model outputs the volume density and RGB color of each indoor scene image through the MLP network, and uses the volume rendering algorithm to render the scene image at each perspective according to the volume density and RGB color;

[0069] Image fusion module 43: used to fuse scene images from multiple different perspectives to generate a 3D reconstructed image of the indoor scene.

[0070] Please refer to FIG6 , which is a schematic diagram of the computer device structure of an embodiment of the present application. The computer device 50 includes:

[0071] A memory 51 storing executable program instructions;

[0072] a processor 52 connected to the memory 51;

[0073] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: using a monocular camera to collect indoor scene images from multiple different perspectives; inputting the indoor scene images into a trained neural radiation field model, the neural radiation field model outputs the volume density and RGB color of each indoor scene image through the MLP network, and uses a volume rendering algorithm to render the scene picture from each perspective according to the volume density and RGB color; fusing the scene pictures from multiple different perspectives to generate a three-dimensional reconstructed image of the indoor scene.

[0074] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0075] Please refer to Figure 7, which is a schematic diagram of the structure of the storage medium of an embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 that can implement the following steps: using a monocular camera to collect indoor scene images from multiple different perspectives; inputting the indoor scene images into a trained neural radiation field model, the neural radiation field model outputs the volume density and RGB color of each indoor scene image through an MLP network, and using a volume rendering algorithm to render the scene image from each perspective according to the volume density and RGB color; fusing the scene images from multiple different perspectives to generate a three-dimensional reconstructed image of the indoor scene. Among them, the program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including several instructions for causing a computer device (which can be a personal computer, server, or network computer device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program instructions, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, or a terminal computer device such as a computer, a server, a mobile phone, or a tablet. Among them, the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0077] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for indoor three-dimensional object reconstruction, characterized in that: include: Use a monocular camera to collect images of indoor scenes from multiple different perspectives; Input the indoor scene image into the trained neural radiation field model, the neural radiation field model outputs the volume density and RGB color of each indoor scene image through the MLP network, and uses the volume rendering algorithm to render the scene image at each viewing angle according to the volume density and RGB color; The scene images from multiple different perspectives are fused to generate a 3D reconstructed image of the indoor scene.

2. The indoor three-dimensional object reconstruction method according to claim 1, characterized in that: The method of using a monocular camera to collect indoor scene images at multiple different viewing angles also includes: recording the position and direction of each indoor scene image.

3. The indoor three-dimensional object reconstruction method according to claim 2, characterized in that: After the monocular camera is used to collect indoor scene images at multiple different viewing angles, the method further includes: The indoor scene image is preprocessed to obtain a corrected indoor scene image.

4. The indoor three-dimensional object reconstruction method according to claim 3, characterized in that: The preprocessing of the indoor scene image is specifically as follows: Obtaining an intrinsic parameter matrix and an extrinsic parameter matrix of the monocular camera; The indoor scene image is subjected to distortion removal processing according to the intrinsic parameter matrix and the extrinsic parameter matrix to obtain a corrected indoor scene image.

5. The indoor three-dimensional object reconstruction method according to claim 3 or 4, characterized in that: Before inputting the indoor scene image into the trained neural radiation field model, the method further includes: The corrected indoor scene image is input into the improved NeRF for training, and the improved NeRF outputs the volume density and RGB color of each pixel in the indoor scene image through the MLP network. Color, hierarchical pixel sampling of coarse network and fine network is performed according to the volume density and RGB color, three-dimensional reconstruction of indoor scene image is performed according to the hierarchical pixel sampling result, and the improved NeRF is optimized according to the three-dimensional reconstruction result to obtain a trained neural radiation field model.

6. The indoor three-dimensional object reconstruction method according to claim 5, characterized in that: The training process of the improved NeRF is specifically as follows: For each indoor scene image r, it is processed into a 5D vector input The 5D vector input includes the 3D position coordinate x and the viewing direction d of each indoor scene image in the Cartesian coordinate system; the 5D vector is converted into Perform uniform sampling with a sampling interval of Δt, obtain the distribution of valid sampling positions, fit the probability density function of the valid sampling positions, and output the uniform sampling results Based on the probability density function, the 5D vector is centered at the position with high probability through a delicate network. Perform non-uniform sampling and output non-uniform sampling results in, represents the volume density; The uniform sampling result of the rough network Non-uniform sampling results with delicate network output Mix and get the final RGB color output result C f (r).

7. The indoor three-dimensional object reconstruction method according to claim 5 or 6, characterized in that: The optimization of the improved NeRF according to the three-dimensional reconstruction result is specifically as follows: The mean square error of the uniform sampling result of the coarse network and the non-uniform sampling result of the fine network is calculated at each pixel as the loss function to simultaneously optimize the coarse network and the fine network: in, is the RGB color sampling result of the rough network sampling the light r, C gt (r) is the true RGB color value of light r.

8. A method and device for indoor three-dimensional object reconstruction, characterized in that: include: Image acquisition module: used to acquire indoor scene images from multiple different perspectives, the image acquisition module is a monocular camera; Image rendering module: used for inputting the indoor scene image into the trained neural radiation field model, the neural radiation field model outputs the volume density and RGB color of each indoor scene image through the MLP network, and uses the volume rendering algorithm to render the scene image at each viewing angle according to the volume density and RGB color; Image fusion module: used to fuse scene images from multiple different perspectives to generate a three-dimensional reconstructed image of the indoor scene.

9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the indoor three-dimensional object reconstruction method according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control a method for reconstructing indoor three-dimensional objects.

10. A storage medium, characterized in that: The device stores program instructions executable by a processor, wherein the program instructions are used to execute the indoor three-dimensional object reconstruction method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Neural network light field method based on joint sampling structure

    CN113327299A

  • Image processing method and device based on neural radiation field

    CN114972632A

  • Fuel gas plant station three-dimensional reconstruction method and device based on nerve radiation field

    CN115035252A

  • Image generation method and device, electronic equipment and storage medium

    CN115100339A

  • Unmanned aerial vehicle aerial image synthesis method and system, computer equipment and storage medium

    CN116071278A

Cited By

  • Digital ground three-dimensional reconstruction method, device and equipment based on neural radiation field and bidirectional reflectance distribution function, and storage medium

    CN119229033A

  • Generalized neural radiation field reconstruction method based on multi-modal information fusion

    CN119359934A

  • Method and device for reconstructing scene grabbed by robot and electronic equipment

    CN119445029A

  • Transparent object reconstruction and new view angle synthesis method based on 3DGS

    CN119579796A

  • Three-dimensional reconstruction model training method and device

    CN119648899A