Method and system for three-dimensional reconstruction of static indoor scene based on neural radiance field

Abstract

The application provides a kind of static indoor scene three-dimensional reconstruction method and system based on neural radiation field, and the method comprises the following steps: obtaining the data of the indoor scene to be reconstructed;Based on neural radiation field algorithm, the indoor scene to be reconstructed is divided into two or more parameterized voxel grids, and the corresponding scene information of the voxel grid is described by parameters, and the neural radiation field implicit model of the scene is reconstructed;Based on particle swarm optimization algorithm, the performance of the parameter combination of each voxel grid of the current neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained;Obtain the trained neural radiation field implicit model.The application realizes that non-professional users can quickly and accurately complete the three-dimensional reconstruction of static indoor scene;At the same time, the efficiency and quality of three-dimensional reconstruction are improved;It is suitable for application scenarios that require instant data and fast response.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction technology in the field of artificial intelligence technology, more specifically, to a static indoor scene three-dimensional reconstruction method and system based on neural radiance field. BACKGROUND

[0002] With the rapid development of computer science and artificial intelligence technology, digitalization is increasingly widely used in performing arts, and various users including researchers, industry professionals and non-technical users have shown strong interest in innovative performing arts space development and utilization. Under this trend, it is particularly important to achieve high-precision and fast three-dimensional reconstruction of indoor scenes. However, the three-dimensional reconstruction technology on the market often faces the dual challenges of unsatisfactory reconstruction results and slow processing speed when dealing with indoor environments, which to a large extent restricts the promotion and application of real-time three-dimensional reconstruction technology.

[0003] A method and device for three-dimensional reconstruction of an indoor scene are disclosed in the prior art (publication number: CN118037942A), which obtains RGB images and depth images containing the indoor scene to be reconstructed by a depth camera, then performs point cloud conversion, registration and fusion to obtain a three-dimensional model containing the indoor scene to be reconstructed, and finally performs surface reconstruction and texture mapping to achieve three-dimensional reconstruction of the indoor scene. The above method has the following disadvantages: the data acquisition and processing, as well as the complexity of the iterative closest point algorithm and sparse convolution calculation, result in large consumption of computing resources and time during the reconstruction process in large-scale scenes. In addition, a method and device for three-dimensional reconstruction of an indoor scene are also disclosed (publication number: CN114004941A), which uses the powerful computing power of cloud devices for model training and three-dimensional model generation, and simultaneously converts the implicit model to an explicit model through an edge device. Although it can also be used for large-scale indoor scene three-dimensional reconstruction, it still has the following disadvantages: 1) there are challenges of high computing resource requirements, system complexity, device and cost problems, and long model training time; 2) it requires high professional requirements for devices and collection personnel, and is not suitable for application scenarios that require fast response or real-time data. A method and system for three-dimensional reconstruction of an indoor scene based on a single RGB image are also disclosed (publication number: CN111414923A), which improves the accuracy of depth estimation through an indoor depth estimation network based on structural knowledge, and effectively improves the overall effect of indoor reconstruction through an indoor three-dimensional reconstruction network based on implicit neural representation. However, the above method still has the following disadvantages: 1) when dealing with high-resolution images and large-scale scenes, the training and inference process of the depth estimation network requires a large amount of computing resources; 2) it involves multiple complex calculation steps such as normal vector calculation, principal direction estimation, and camera ray calculation, which puts high requirements on the implementation and maintenance of the system.

[0004] Therefore, there is an urgent need for a method for 3D reconstruction of indoor scenes that saves computational resources and computational steps. Summary of the Invention

[0005] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method and system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields.

[0006] This invention provides a method for three-dimensional reconstruction of a static indoor scene based on neural radiation fields, comprising: acquiring data of the indoor scene to be reconstructed; and performing data preprocessing on the indoor scene data; wherein, the data preprocessing includes performing image noise reduction processing on the indoor scene data using a partitioned joint bilateral filtering technique;

[0007] The neural radiation field algorithm divides the indoor scene space to be reconstructed into two or more parameterized voxel grids, and describes the scene information corresponding to the voxel grids through parameters to reconstruct the implicit model of the scene's neural radiation field.

[0008] Based on the particle swarm optimization algorithm, an initial particle swarm and iteration parameters are set, and the performance of the parameter combination of each voxel grid in the current neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is taken as the optimal parameters, and the trained neural radiation field implicit model is obtained.

[0009] One optional approach is to use a partitioned joint bilateral filtering technique to perform image noise reduction processing on the indoor scene data, including:

[0010] The static indoor scene image data is divided into regions, and different confidence weights are set for each region; accurate pixel values ​​are restored at pixels with inaccurate pixel values ​​and low confidence, and the error between the filtered pixel values ​​and the true values ​​is reduced;

[0011] Joint bilateral filtering is applied to each region to achieve image noise reduction processing of the indoor scene data.

[0012] One option is to divide the static indoor scene image data into regions and assign different confidence weights to each region, using the following formula:

[0013]

[0014] Where Conf(x,y) is the confidence level, α is the adjustment parameter, and W conf (x,y) represents the confidence weight;

[0015] A joint bilateral filter is applied to each region to achieve image noise reduction of the indoor scene data; this is achieved through the following formula:

[0016]

[0017] Among them, W conf (x+i, y+j) represents the confidence weight, K space (i,j) is the spatial filter kernel K range (D(x+i,y+j)), where D(x,y) is the range filter kernel. These are the filtered pixel values.

[0018] One optional approach is that the data preprocessing further includes,

[0019] Frame extraction processing is performed on the indoor scene data to obtain static indoor scene image data;

[0020] The static indoor scene image data was calibrated using the radial distortion model in the OpenCV library;

[0021] The calibrated static indoor scene image data is then subjected to noise reduction processing.

[0022] The static indoor scene image data, after noise reduction, is processed using COLMAP software to obtain a JSON file containing camera parameters and the pose of each image.

[0023] One optional approach is to use COLMAP software to process the static indoor scene image data after noise reduction. The method for obtaining a JSON file containing camera parameters and the pose of each image includes...

[0024] Use COLMAP software to estimate the pose of the camera in the scene;

[0025] Generate the corresponding transformation matrix based on the camera pose output by COLMAP;

[0026] Adjust the transformation matrix according to whether the coordinate system used by the COLMAP software needs to be maintained;

[0027] The center points of all cameras are calculated using the adjusted transformation matrix;

[0028] Based on the requirements of NeRF processing, the camera positions are fine-tuned; the adjusted camera positions and the center points of all cameras are then integrated and written into a JSON file.

[0029] One option is to dynamically adjust the size of the voxel grids according to the scene complexity in the step of dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiation field algorithm.

[0030] One possible approach is to use a particle swarm optimization algorithm, setting an initial particle swarm and iteration parameters, and iteratively evaluating the performance of the parameter combinations of each voxel grid in the implicit model of the neural radiation field until the parameter combination of each voxel grid corresponding to the globally optimal position is obtained. This implementation method includes...

[0031] Create a swarm of particles and initialize the particle swarm; where each particle represents a combination of parameters of a voxel grid in a multilayer perception model of an implicit neural radiation field model.

[0032] Create a group and randomly initialize the velocity and position of each particle; calculate the individual optimal position of each particle and find the minimum fitness value in the parameter space;

[0033] Find the particle with the best fitness value in the entire particle swarm, and set the parameters of the particle with the best fitness value as the global optimal position; update the velocity and position of each particle.

[0034] For each particle, the multilayer sensing model of the neural radiation field implicit model is trained using the current parameter combination corresponding to the particle, and the loss function value is calculated as the fitness value for fitness evaluation; wherein, when the current fitness value of a particle is better than the fitness value of the individual best position corresponding to the particle, the position corresponding to the current fitness value of the particle is updated to the individual best position; when the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, the individual best position of the particle is updated to the global best position.

[0035] Repeat the steps of velocity and position updates and fitness value evaluation until the set number of iterations is reached or the convergence condition is met; when the particle swarm optimization algorithm converges or reaches the number of iterations, the final combination of parameters of each voxel grid corresponding to the global optimal position is obtained.

[0036] One possible approach is to assume V ij X represents the velocity of particle i in dimension j. ij Let i represent the current position of particle i in dimension j. The velocity and position of each particle are updated using the following formula:

[0037] V ij =w·V ij +c1·r1·(Pbest ij -X ij )+c2·r2·(Gbestj -X ij )

[0038] X ij =X ij +V ij

[0039] Among them, Pbest ij This indicates that the optimal position of particle i is in dimension j, Gbest j This indicates that the global optimal position is in dimension j, w is the inertia weight, c1 and c2 are acceleration factors, and r1 and r2 are random numbers.

[0040] On the other hand, the present invention also provides a three-dimensional reconstruction system for static indoor scenes based on neural radiation fields, wherein the three-dimensional reconstruction based on the aforementioned three-dimensional reconstruction method for static indoor scenes based on neural radiation fields includes:

[0041] A preprocessing unit is used to acquire data of the indoor scene to be reconstructed and to perform data preprocessing on the indoor scene data; wherein, the data preprocessing includes performing image noise reduction processing on the indoor scene data using a partitioned joint bilateral filtering technique;

[0042] Voxelization unit is used to divide the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiation field algorithm, and to reconstruct the implicit model of the neural radiation field of the scene by describing the scene information corresponding to the voxel grids through parameters.

[0043] The model training unit is used to iteratively evaluate the performance of the parameter combinations of each voxel grid in the neural radiation field implicit model based on the particle swarm optimization algorithm, setting an initial particle swarm and iteration parameters, until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is taken as the optimal parameters to obtain the trained neural radiation field implicit model.

[0044] In another aspect, the present invention also provides an electronic device, the electronic device comprising:

[0045] At least one processor; and,

[0046] A memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the steps in the previously described method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields.

[0048] As can be seen from the above technical solution, the static indoor scene 3D reconstruction method and system based on neural radiation field provided by the present invention divides the indoor scene space to be reconstructed into two or more parameterized voxel meshes based on the neural radiation field algorithm; then, it uses the particle swarm optimization algorithm to train the implicit model of the neural radiation field. On the one hand, it reduces the professional requirements of the acquisition equipment and personnel; on the other hand, it reduces the computational resource requirements and computational system complexity; it enables non-professional users to quickly and accurately complete the 3D reconstruction of static indoor scenes; at the same time, it improves the efficiency and quality of 3D reconstruction. The static indoor scene 3D reconstruction method based on neural radiation field of the present invention is particularly suitable for application scenarios that require real-time data and rapid response. Attached Figure Description

[0049] Other objects and results of the invention will become more apparent and readily understood by referring to the following description taken in conjunction with the accompanying drawings, and with a more complete understanding of the invention. In the drawings:

[0050] Figure 1 This is a flowchart illustrating a method for three-dimensional reconstruction of a static indoor scene based on a neural radiation field according to an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the logical structure of a static indoor scene 3D reconstruction system based on neural radiation field according to an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0053] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description of one or more embodiments.

[0054] To address the existing problems, this invention provides a method and system for 3D reconstruction of static indoor scenes based on neural radiation fields. This not only solves the problem of computational complexity in existing 3D reconstruction schemes for static indoor scenes based on neural radiation fields, but also enables users to quickly and accurately complete 3D reconstruction of static indoor scenes using indoor scene data captured by non-professional devices such as mobile phones or cameras.

[0055] To better illustrate the technical solution of the present invention, some of the technical terms involved in the present invention will be briefly explained below.

[0056] Neural Radiance Field (NeRF) is a method based on Multilayer Perceptron (MLP) and volume rendering techniques for applications such as 3D reconstruction and novel perspective synthesis. It implicitly represents a scene by taking 3D point coordinates and viewpoint as input and outputting the color and density of those points. Then, by sampling several points along each ray and performing volume integration based on their color and density, an image from the target viewpoint is generated. NeRF is a deep learning method for 3D scene reconstruction. It encodes the 3D scene using a Multilayer Perceptron (MLP). NeRF models can generate high-quality images from novel perspectives and achieve end-to-end rendering of 3D scenes.

[0057] Joint Bilateral Filtering (JBF) is a nonlinear filtering technique that combines the bilateral characteristics of the spatial and intensity domains to simultaneously consider the spatial proximity and similarity of pixel values. In JBF, the spatial and range filter kernels are typically Gaussian functions.

[0058] Multilayer Perceptron (MLP) is a common artificial neural network architecture, often used for deep learning tasks. An MLP consists of multiple neurons distributed across multiple layers, including an input layer, at least one or more hidden layers, and an output layer.

[0059] Particle Swarm Optimization (PSO) is a swarm intelligence-based optimization technique proposed by Eberhart and Kennedy in 1995. Inspired by the foraging behavior of flocks of birds, it solves problems by simulating the social behavior of groups of organisms such as flocks of birds or schools of fish. In PSO, each solution is considered a "particle" in the search space, representing a potential solution to the problem. Each particle has a velocity that guides its movement within the search space. Each particle updates its position in the search space based on its velocity and position. Each particle remembers the optimal solution it finds, called its individual best (pBest). The global best (gBest) is the optimal solution found by all particles in the entire swarm.

[0060] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0061] It should be noted that the following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Technologies and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such technologies and equipment should be considered part of the specification.

[0062] To illustrate the method and system for 3D reconstruction of static indoor scenes based on neural radiation fields provided by this invention, Figure 1 The flowchart of the static indoor scene three-dimensional reconstruction method based on neural radiation field in this embodiment of the invention is illustrated by way of example.

[0063] like Figure 1 As shown, the method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields provided by the present invention mainly includes steps S110 to S130.

[0064] S110. Acquire data of the indoor scene to be reconstructed; and perform data preprocessing on the indoor scene data; wherein, the data preprocessing includes performing image noise reduction processing on the indoor scene data using a partitioned joint bilateral filtering technique.

[0065] S120. Based on the neural radiation field algorithm, the indoor scene space to be reconstructed is divided into two or more parameterized voxel grids, and the scene information corresponding to the voxel grids is described by parameters to reconstruct the implicit model of the neural radiation field of the scene.

[0066] S130. Based on the particle swarm optimization algorithm, set the initial particle swarm and iteration parameters, and iteratively evaluate the performance of the parameter combination of each voxel grid in the current neural radiation field implicit model until the parameter combination of each voxel grid corresponding to the global best position is obtained; take the parameter combination of each voxel grid corresponding to the global best position as the optimal parameters to obtain the trained neural radiation field implicit model.

[0067] The following will describe in detail, step by step, the method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields provided by the present invention.

[0068] First, in step S110, data of the indoor scene to be reconstructed is acquired; and the data of the indoor scene is preprocessed; wherein, the data preprocessing includes image noise reduction processing of the data of the indoor scene using a partitioned joint bilateral filtering technique.

[0069] Specifically, this refers to a method of acquiring indoor scene data using portable devices such as smartphones, cameras, and tablets—non-professional equipment. In particular, the process of acquiring indoor scene data can be, but is not limited to, using portable devices to photograph indoor scenes to obtain image data; and utilizing the device's built-in sensors, such as accelerometers and gyroscopes, to collect the device's spatial positioning information.

[0070] The data for the indoor scene to be reconstructed includes RGB images of the static indoor scene to be reconstructed and a JSON file containing camera parameters and the pose of each image.

[0071] The data preprocessing described in step S110 includes not only image noise reduction, but the entire preprocessing process also includes the following steps S111 to S114.

[0072] S111. Frame extraction processing is performed on the indoor scene data to obtain static indoor scene image data. Specifically, if the input indoor scene data is video data, the ffmpeg library is used to extract frames from the video sequence to obtain static indoor scene image data. Users can select video frame extraction parameters. S112. The static indoor scene image data is calibrated using the radial distortion model in the OpenCV library. S113. Noise reduction processing is performed on the calibrated static indoor scene image data. Partitioned joint bilateral filtering technology is used to perform image noise reduction processing on the indoor scene data. Specifically, partitioned joint bilateral filtering is used as an image noise reduction technique, considering the spatial information of the image and the grayscale differences between pixels, reducing noise while preserving details and edge information in the image. S114. COLMAP software is used to process the denoised static indoor scene image data to obtain a JSON file containing camera parameters and the pose of each image. Users can specify the file input and output paths and output the processed image file to a specified folder. In summary, static indoor scene image data is calibrated using the radial distortion model in the OpenCV library. COLMAP software estimates the camera pose, and after generating the transformation matrix, it is adjusted based on whether COLMAP coordinates are preserved. The center points of all cameras are calculated. Camera positions are adjusted to make the scene suitable for NeRF processing, and finally, the processed data is written to a JSON file. Camera parameters include intrinsic parameters (such as focal length, principal point coordinates, and pixel size) and extrinsic parameters (i.e., camera position and rotation, which can be represented as a rotation matrix and a translation vector). Intrinsic parameters determine the projection method of the image, while extrinsic parameters describe the camera's pose in the world coordinate system. Image pose refers to the position and orientation of each image relative to a reference coordinate system (usually the world coordinate system). Pose includes translation (the camera's position when the image is captured) and rotation (the camera's orientation when the image is captured), which determines the portion of the scene captured in the image; that is, the camera position is part of the image pose.

[0073] For step S114, the method of obtaining a JSON file containing camera parameters and the pose of each image based on the denoised static indoor scene image data using COLMAP software includes: S1141, estimating the pose of the cameras in the scene using COLMAP software; S1142, generating a corresponding transformation matrix based on the camera pose output by COLMAP; S1143, adjusting the transformation matrix according to whether the coordinate system used by COLMAP software needs to be maintained; S1144, calculating the center points of all cameras using the adjusted transformation matrix; it should be noted that the center point of the camera usually refers to the geometric center or centroid of all camera positions, which can be used as a reference point for the scene or for estimating the scope and layout of the scene; S1145, fine-tuning the camera positions according to the requirements of NeRF processing; integrating the adjusted camera positions and the center points of all cameras and writing them into a JSON file; wherein, the camera parameters output by COLMAP are automatically imported into the JSON file.

[0074] Specifically, in step S110, the present invention enables users to capture indoor scene data using non-professional equipment such as mobile phones or cameras. Compared with existing technologies, which typically require highly specialized equipment to capture the same type of data, the present invention significantly lowers the technical threshold, making 3D scene capture more convenient and widespread.

[0075] The method for image noise reduction processing of the indoor scene data using partitioned joint bilateral filtering technology in step S113 includes the following steps S1131 to S1132.

[0076] S1131. The static indoor scene image data is divided into regions, and different confidence weights are set for each region; accurate pixel values ​​are restored at pixels with inaccurate pixel values ​​and low confidence, and the error between the filtered pixel values ​​and the true values ​​is reduced. To correctly handle the confidence differences in different regions of the static indoor scene image, the static indoor scene image data is divided into regions, and different confidence weights are set for each region. This allows for better handling of the characteristics of different regions in the joint bilateral filtering.

[0077] Specifically, the static indoor scene image data is divided into regions, and different confidence weights are assigned to each region, which is achieved through the following formula:

[0078]

[0079] Where Conf(x,y) is the confidence level, α is the adjustment parameter, and W conf (x,y) represents the confidence weight.

[0080] To recover accurate pixel values ​​at pixels with inaccurate or low confidence levels; assuming Dfiltered is the filtered pixel value and Dtrue is the true value, to effectively improve the accuracy of 3D reconstruction of indoor scenes, it is necessary to reduce the error between the filtered pixel value and the true value.

[0081]

[0082] It should be noted that confidence level partitioning is performed first, followed by pixel restoration. This invention employs an innovative image processing technique, namely a partitioned joint bilateral filtering method, to improve the ability to handle noise and missing pixels. This method fully utilizes the advantages of bilateral filtering while considering the confidence differences in different regions of the image. Specifically, it first divides the image into multiple regions and then assigns different weights to each region. The purpose of this is to more accurately handle the unique characteristics of each region during joint bilateral filtering. In this process, the method maintains high sensitivity for pixels with low confidence, ensuring that these pixels are processed appropriately. By setting different confidence weights for different regions, this method can better adapt to the characteristics of different regions in the image during the filtering process, thereby achieving higher quality image restoration results. This partitioned joint bilateral filtering method effectively improves the accuracy and efficiency of image processing by intelligently adjusting weights and filtering strategies.

[0083] S1132. Perform joint bilateral filtering on each region to achieve image noise reduction processing on the indoor scene data. That is, by partitioning the static indoor scene image data, setting different weights for each partition and performing joint bilateral filtering on each partition, sensitivity to low-confidence pixels is maintained.

[0084] Specifically, a joint bilateral filter is applied to each region to achieve image noise reduction of the indoor scene data; this is achieved through the following formula:

[0085]

[0086] Among them, W conf (x+i, y+j) represents the confidence weight, K space (i,j) is the spatial filter kernel K range (D(x+i,y+j)), where D(x,y) is the range filter kernel. These are the filtered pixel values.

[0087] In step S120, the indoor scene space to be reconstructed is divided into two or more parameterized voxel grids based on the Neural Radiation Field algorithm. The scene information corresponding to each voxel grid is described by parameters, and the implicit neural radiation field model of the scene is reconstructed. In other words, based on the Neural Radiation Field algorithm, the scene is divided into multiple parameterized voxel grids. Each small block is described by a small number of parameters, and the scene containing the preset three-dimensional spatial information of the static indoor scene to be reconstructed is reconstructed using the NeRF model.

[0088] The 3D reconstruction of an indoor target scene using neural radiation fields mainly involves several steps: data collection, network training, voxelization, scene reconstruction, and mesh reconstruction. The first step, data collection, primarily involves gathering a set of sample images taken by a camera from multiple viewpoints. These images should cover various angles and viewpoints within the target scene. The second step, network training, mainly involves training a NeRF model using the collected sample images. The model's input is the direction and origin of the light rays, and its output is the radiation intensity and color at that ray. NeRF models typically employ a neural network-like structure, such as a multilayer perceptron (MLP) or convolutional neural network (CNN). The third step, voxelization, involves dividing the scene space into voxel grids and generating a set of rays within each voxel grid. These rays are used to query the NeRF model to obtain the radiation intensity and color of each point in the scene. The fourth step, scene reconstruction, mainly involves obtaining the radiation intensity and color of the rays in each voxel grid by querying the NeRF model. Based on the obtained information, dense sampling of points in each voxel grid can be generated, and the color of each point can be calculated. The final step, mesh reconstruction, is used to convert the reconstructed point cloud into a mesh model and perform post-processing operations such as texture mapping. Methods such as triangulation can be used to convert the point cloud into a mesh model.

[0089] In the step of dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiation field algorithm, the size of the voxel grids is dynamically adjusted according to the scene complexity to improve reconstruction efficiency and accuracy. Specifically, by using a multi-scale hash table based on neural radiation fields and optimized data structures, fast retrieval of neural network weights is achieved, effectively reducing the number of model parameters. Specifically, by utilizing multi-scale representation, NeRF can more efficiently handle input data at different resolutions. This method allows the model to capture scene details at different levels, thereby improving rendering quality and reducing computational load. Efficient data structures, such as KD-trees or octrees, are used to organize and query data in 3D space. These data structures allow NeRF to quickly locate and process specific regions in the scene, thereby reducing unnecessary computation. By introducing hash tables, NeRF can quickly index the weights in the neural network. By using hash tables and efficient data structures, the NeRF model can significantly reduce the number of network parameters. The reduction in the number of parameters not only reduces the model's memory footprint but may also improve the model's training and inference speed. The fast indexing capability means that NeRF can quickly retrieve the required weights from the neural network, which is crucial for real-time rendering and novel perspective synthesis. The reduced number of parameters and efficient data structures help optimize memory usage and computational resource allocation, allowing NeRF models to be deployed in resource-constrained environments. Furthermore, this method accelerates the forward propagation speed of the neural network. Through fine-tuning and optimization of voxel parameters, the technology of this invention can adapt to varying scene and perspective requirements, enhancing matching accuracy and processing speed, and supporting real-time 3D scene updates and reconstruction. In short, this invention provides powerful support for real-time 3D vision tasks by reducing computational burden and improving processing efficiency.

[0090] In step S130, during the PSO algorithm optimization process, an initial particle swarm and iteration parameters are set, and the performance of the current parameter combination is evaluated in each iteration to gradually approach the globally optimal parameter settings. Specifically, the method for implementing the PSO algorithm, setting an initial particle swarm and iteration parameters, and iteratively evaluating the performance of the parameter combinations of each voxel grid in the current implicit model of the neural radiation field until the parameter combination of each voxel grid corresponding to the globally optimal position includes the following steps.

[0091] S131. Create a group of particles and initialize the particle swarm; where each particle represents a combination of parameters of each voxel grid of the multilayer perception model of the neural radiation field implicit model.

[0092] Assume P i Let n represent the combination of parameters for particle i, and n represent the number of parameters. The initial particle swarm is represented as:

[0093] P i =(p i1 ,p i2 ,...,p in )

[0094] S132. Create a group and randomly initialize the velocity and position of each particle.

[0095] Assume X is the initial velocity, x ij To initialize the position, the initial velocity and position are represented as follows:

[0096] X ij =(x ij1 ,x ij2 ,...,x ijn )

[0097] Calculate the optimal position for each particle and find the minimum fitness value in the parameter space.

[0098] Assume Pbest is the optimal position for initializing an individual, and f(Pi) is the optimal parameter position.

[0099] Pbest i =argmin(f(P) i ))

[0100] S133. Find the particle with the best fitness value in the entire particle swarm, and set the parameters of the particle with the best fitness value as the global optimal position.

[0101] Assuming Gbest is the initial global best position, the initial global best position is represented as:

[0102] Gbest=argmin(f(P i ))

[0103] For each particle, update its velocity and position.

[0104] Specifically, let's assume V ij X represents the velocity of particle i in dimension j. ij Let i represent the current position of particle i in dimension j. The velocity and position of each particle are updated using the following formula:

[0105] V ij =w·V ij +c1·r1·(Pbest ij -X ij )+c2·r2·(Gbest j -X ij )

[0106] Xij =X ij +V ij

[0107] Among them, Pbest ij This indicates that the optimal position of particle i is in dimension j, Gbest j This indicates that the global optimal position is in dimension j, w is the inertia weight, c1 and c2 are acceleration factors, and r1 and r2 are random numbers.

[0108] S134. For each particle, the current parameter combination corresponding to the particle is used to train the multilayer sensing model of the neural radiation field implicit model, and the loss function value is calculated as the fitness value for fitness evaluation. That is, for each particle, its current parameter combination is used to train the MLP model, and its prediction error or loss function value is calculated as the fitness value for fitness evaluation.

[0109] f(P i ) = Loss Function (P) i )

[0110] Specifically, if a particle's current fitness value is better than the fitness value of its corresponding best position, then the position corresponding to the particle's current fitness value is updated to the best position. In other words, if a particle's current fitness value is better than the fitness value of its best position, then the best position is updated.

[0111] Pbest i =argmin(f(P) i ))

[0112] If the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, then the individual best position of that particle is updated to the global best position. In other words, if the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, then the global best position is updated.

[0113] Gbest=argmin(f(P i ))

[0114] S135, repeat the steps of velocity and position update and fitness value evaluation until the set number of iterations is reached or the convergence condition is met; when the particle swarm optimization algorithm converges or reaches the number of iterations, the parameter combination of each voxel grid corresponding to the global best position is finally obtained.

[0115] In the specific implementation process, after obtaining the trained neural radiation field implicit model, volume rendering technology is used to visualize the trained neural radiation field implicit model. A user management module can also be used to manage user permissions, process user input, and output 3D reconstruction results, allowing users to easily operate the system and view the reconstruction effects. Additionally, a real-time web viewer can be set up, allowing users to visualize the results in real time while training the NeRF model; geometric export is supported, which is beneficial for artists and creators who need to export 3D reconstructions to game engines or other downstream tools. The module supports multiple export methods and output formats, including point clouds and meshes.

[0116] The combination of the three steps S110 to S130 described above constitutes the static indoor scene 3D reconstruction method based on neural radiation fields provided by this invention. This method uses neural radiation fields as its core technology, effectively achieving high-quality scene reconstruction, rendering, and inference, while also possessing real-time 3D scene update and reconstruction capabilities. The comprehensive advantages of this invention make it widely applicable in fields such as virtual reality, augmented reality, and game development.

[0117] Corresponding to the above-mentioned method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields, the present invention also provides a system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields, which is used for video production based on the above-mentioned method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields. Figure 2 The logical framework structure of a static indoor scene 3D reconstruction system based on neural radiation field according to an embodiment of the present invention is shown.

[0118] like Figure 2 As shown, the present invention provides a static indoor scene 3D reconstruction system 200 based on neural radiation field, which mainly includes a preprocessing unit 210, a voxelization unit 220 and a model training unit 230.

[0119] The preprocessing unit 210 is used to acquire data of the indoor scene to be reconstructed and to perform data preprocessing on the indoor scene data; wherein, the data preprocessing includes performing image noise reduction processing on the indoor scene data using a partitioned joint bilateral filtering technique;

[0120] Voxelization unit 220 is used to divide the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiation field algorithm, and to reconstruct the implicit model of the neural radiation field of the scene by describing the scene information corresponding to the voxel grids through parameters.

[0121] The model training unit 230 is used to iteratively evaluate the performance of the parameter combinations of each voxel grid in the neural radiation field implicit model based on the particle swarm optimization algorithm, setting an initial particle swarm and iteration parameters, until the parameter combination of each voxel grid corresponding to the globally optimal position is obtained; the parameter combination of each voxel grid corresponding to the globally optimal position is taken as the optimal parameters to obtain the trained neural radiation field implicit model.

[0122] The above-mentioned static indoor scene 3D reconstruction system based on neural radiation field is an implementation method corresponding to the aforementioned static indoor scene 3D reconstruction method based on neural radiation field. Other more specific implementation embodiments can refer to the specific embodiments of the above-mentioned static indoor scene 3D reconstruction method based on neural radiation field, which will not be described in detail here.

[0123] As can be seen from the above embodiments, the static indoor scene 3D reconstruction method and system based on neural radiation fields proposed in this invention allows users to capture indoor scene images using readily available non-professional devices such as smartphones or cameras. It breaks through the limitations of traditional 3D reconstruction and has the following significant features:

[0124] 1) Speed: Users can quickly complete the entire process from shooting to generating a 3D model, greatly shortening the reconstruction time.

[0125] 2) Accuracy: Advanced image processing techniques, such as noise reduction algorithms, ensure high accuracy and high quality of the reconstruction results.

[0126] 3) Optimized voxel mesh generation: Parametric voxel mesh generation technology is adopted, which improves the efficiency and accuracy of scene spatial representation.

[0127] 4) Particle parameter optimization algorithm: Combining the parameterized voxel grid partitioning of the neural radiation field of this invention with advanced algorithms such as particle swarm optimization, the model parameters are intelligently optimized, which further improves the efficiency and effect of model training.

[0128] 5) Solving existing problems: This invention provides an effective solution to the problems of low reconstruction quality, slow speed and high consumption of computing resources in the existing technology.

[0129] 6) User-friendly: Especially suitable for all types of users, including non-technical users, making 3D reconstruction technology easier to access and use.

[0130] 7) Real-time performance: Supports efficient and real-time 3D reconstruction of indoor scenes, suitable for application scenarios that require indoor scene updates and dynamic reconstruction.

[0131] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 3As shown, the present invention also provides an electronic device, the electronic device comprising:

[0132] At least one processor; and,

[0133] A memory that is communicatively connected to at least one processor; wherein,

[0134] The memory stores a computer program that can be executed by at least one processor to enable the at least one processor to perform the steps in the aforementioned method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields.

[0135] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0136] Electronic devices include, but are not limited to, Extended Reality (XR) devices, Virtual Reality (VR) devices, Augmented Reality (AR) devices, smartphones, tablets, laptops, and desktop computers. It should be noted that the executing entity in this embodiment can be an electronic device or a server. If the executing entity is a server, after completing model fusion, the server can send the target scene model to the electronic device that needs to render the target scene for deployment.

[0137] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0138] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0139] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0140] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0141] The memory 11 in the electronic device 1 stores a static indoor scene 3D reconstruction program 12 based on neural radiation fields, which is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0142] S110. Acquire data of the indoor scene to be reconstructed; and perform data preprocessing on the indoor scene data; wherein, the data preprocessing includes performing image noise reduction processing on the indoor scene data using a partitioned joint bilateral filtering technique;

[0143] S120. Based on the neural radiation field algorithm, the indoor scene space to be reconstructed is divided into two or more parameterized voxel grids, and the scene information corresponding to the voxel grids is described by parameters to reconstruct the implicit model of the neural radiation field of the scene.

[0144] S130. Based on the particle swarm optimization algorithm, set the initial particle swarm and iteration parameters, and iteratively evaluate the performance of the parameter combination of each voxel grid in the current neural radiation field implicit model until the parameter combination of each voxel grid corresponding to the global best position is obtained; take the parameter combination of each voxel grid corresponding to the global best position as the optimal parameters to obtain the trained neural radiation field implicit model.

[0145] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figure 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0146] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0147] The method and system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields, according to the present invention, have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the method and system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the contents of the appended claims.

Claims

1. A method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields, characterized in that, include: Acquire data of the indoor scene to be reconstructed; The data of the indoor scene is preprocessed; wherein, the data preprocessing includes image noise reduction processing of the indoor scene data using a partitioned joint bilateral filtering technique; the method of image noise reduction processing of the indoor scene data using a partitioned joint bilateral filtering technique includes dividing the static indoor scene image data into regions and setting different confidence weights for each region; The neural radiation field algorithm divides the indoor scene space to be reconstructed into two or more parameterized voxel grids, and describes the scene information corresponding to the voxel grids through parameters to reconstruct the implicit model of the scene's neural radiation field. Based on the particle swarm optimization algorithm, an initial particle swarm and iteration parameters are set, and the performance of the parameter combination of each voxel grid in the current neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is taken as the optimal parameters to obtain the trained neural radiation field implicit model. The static indoor scene image data is divided into regions, and different confidence weights are assigned to each region using the following formula: in, For confidence level, To adjust the parameters, Confidence weights; A joint bilateral filter is applied to each region to achieve image noise reduction of the indoor scene data; this is achieved through the following formula: in, As the confidence level weight, For spatial filtering kernel, For the range of filter kernels, These are the filtered pixel values.

2. The method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields as described in claim 1, characterized in that, The method for image denoising of the indoor scene data using partitioned joint bilateral filtering technology further includes: It restores accurate pixel values ​​at pixels with inaccurate or low confidence levels, and reduces the error between filtered pixel values ​​and true values. Joint bilateral filtering is applied to each region to achieve image noise reduction processing of the indoor scene data.

3. The method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields as described in claim 1, characterized in that, The data preprocessing also includes, Frame extraction processing is performed on the indoor scene data to obtain static indoor scene image data; The static indoor scene image data was calibrated using the radial distortion model in the OpenCV library; The calibrated static indoor scene image data is then subjected to noise reduction processing. The static indoor scene image data, after noise reduction, is processed using COLMAP software to obtain a JSON file containing camera parameters and the pose of each image.

4. The method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields as described in claim 3, characterized in that, A method for obtaining a JSON file containing camera parameters and the pose of each image based on the denoised static indoor scene image data using COLMAP software includes the following steps: Use COLMAP software to estimate the pose of the camera in the scene; Generate the corresponding transformation matrix based on the camera pose output by COLMAP; Adjust the transformation matrix according to whether the coordinate system used by the COLMAP software needs to be maintained; The center points of all cameras are calculated using the adjusted transformation matrix; Based on the requirements of NeRF processing, the camera positions are fine-tuned; the adjusted camera positions and the center points of all cameras are then integrated and written into a JSON file.

5. The method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields as described in claim 1, characterized in that, In the step of dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiation field algorithm, the size of the voxel grids is dynamically adjusted according to the scene complexity.

6. The method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields as described in claim 1, characterized in that, Based on the particle swarm optimization algorithm, the performance of the parameter combinations of each voxel grid in the implicit model of the neural radiation field is iteratively evaluated by setting an initial particle swarm and iteration parameters until the parameter combination of each voxel grid corresponding to the globally optimal position is obtained. The implementation method includes... Create a swarm of particles and initialize the particle swarm; where each particle represents a combination of parameters of a voxel grid in a multilayer perception model of an implicit neural radiation field model. Create a group and randomly initialize the velocity and position of each particle; calculate the individual optimal position of each particle and find the minimum fitness value in the parameter space; Find the particle with the best fitness value in the entire particle swarm, and set the parameters of the particle with the best fitness value as the global optimal position; update the velocity and position of each particle. For each particle, the multilayer sensing model of the neural radiation field implicit model is trained using the current parameter combination corresponding to the particle, and the loss function value is calculated as the fitness value for fitness evaluation; wherein, when the current fitness value of a particle is better than the fitness value of the individual best position corresponding to the particle, the position corresponding to the current fitness value of the particle is updated to the individual best position; when the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, the individual best position of the particle is updated to the global best position. Repeat the steps of velocity and position updates and fitness value evaluation until the set number of iterations is reached or the convergence condition is met; when the particle swarm optimization algorithm converges or reaches the number of iterations, the final combination of parameters of each voxel grid corresponding to the global optimal position is obtained.

7. The method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields as described in claim 6, characterized in that, Assumption Represents particles In dimensions The speed on, Represents particles In dimensions The current position of the particle is updated by performing velocity and position updates for each particle using the following formula: in, Represents particles The optimal position of an individual in dimension superior, Indicates the global optimal position in dimension superior, It is inertial weight. and It is an accelerator. and It is a random number.

8. A three-dimensional reconstruction system for static indoor scenes based on neural radiation fields, comprising three-dimensional reconstruction based on the three-dimensional reconstruction method for static indoor scenes based on neural radiation fields as described in any one of claims 1-7, including: The preprocessing unit is used to acquire data of the indoor scene to be reconstructed; The data of the indoor scene is preprocessed; wherein, the data preprocessing includes image noise reduction processing of the indoor scene data using a partitioned joint bilateral filtering technique; the method of image noise reduction processing of the indoor scene data using a partitioned joint bilateral filtering technique includes dividing the static indoor scene image data into regions and setting different confidence weights for each region; Voxelization unit is used to divide the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiation field algorithm, and to reconstruct the implicit model of the neural radiation field of the scene by describing the scene information corresponding to the voxel grids through parameters. The model training unit, based on the particle swarm optimization algorithm, sets an initial particle swarm and iteration parameters, and iteratively evaluates the performance of the parameter combinations of each voxel grid in the current neural radiation field implicit model until the parameter combination of each voxel grid corresponding to the globally optimal position is obtained. The parameter combination of each voxel grid corresponding to the globally optimal position is taken as the optimal parameters to obtain the trained neural radiation field implicit model. The static indoor scene image data is divided into regions, and different confidence weights are set for each region using the following formula: in, For confidence level, To adjust the parameters, Confidence weights; A joint bilateral filter is applied to each region to achieve image noise reduction of the indoor scene data; this is achieved through the following formula: in, As the confidence level weight, Spatial filter kernel For the range of filter kernels, These are the filtered pixel values.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the steps in the method for three-dimensional reconstruction of a static indoor scene based on a neural radiation field as described in any one of claims 1-7.

Images