Static indoor scene three-dimensional reconstruction method and system based on neural radiation field

Abstract

The invention provides a static indoor scene three-dimensional reconstruction method and system based on a neural radiation field. The method comprises the following steps: acquiring data of an indoor scene to be reconstructed; dividing the indoor scene space to be reconstructed into more than two parameterized voxel grids based on a neural radiation field algorithm, describing scene information corresponding to the voxel grids through parameters, and reconstructing a neural radiation field implicit model of the scene; based on a particle swarm optimization algorithm, performing iterative evaluation on the performance of the parameter combination of each current voxel grid of the neural radiation field implicit model until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; and obtaining a trained neural radiation field implicit model. According to the invention, a non-professional user can quickly and accurately complete three-dimensional reconstruction of a static indoor scene; meanwhile, the efficiency and the quality of three-dimensional reconstruction are improved; the method is suitable for application scenes requiring instant data and quick response.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology under the field of artificial intelligence technology, and more specifically, to a method and system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields. Background Art

[0002] With the rapid development of computer science and artificial intelligence technology, the application of digitalization in performing arts activities is becoming more and more extensive. Various users, including researchers, industrial professionals and users with non-technical backgrounds, have shown great interest in the development and utilization of innovative performing arts spaces. Under this trend, it is particularly important to achieve high-precision and fast 3D reconstruction of indoor scenes. However, the 3D reconstruction technology currently available on the market often faces the dual challenges of unsatisfactory reconstruction results and slow processing speed when dealing with indoor environments, which largely restricts the promotion and application of real-time 3D reconstruction technology.

[0003] The prior art discloses a method and device for 3D reconstruction of indoor scenes (publication number: CN118037942A), which uses a depth camera to obtain an RGB image and a depth image containing the indoor scene to be reconstructed, and then performs point cloud conversion, registration and fusion to obtain a 3D model containing the indoor scene to be reconstructed, and finally performs surface reconstruction and texture mapping to achieve 3D reconstruction of the indoor scene. The above method has the following disadvantages: high complexity in data acquisition and processing, as well as calculation of iterative nearest point algorithm and sparse convolution, resulting in large consumption of computing resources and time in the reconstruction process in large-scale scenes. In addition, a method and device for three-dimensional reconstruction of indoor scenes are also disclosed (publication number: CN114004941A), which utilizes the powerful computing power of cloud devices to perform model training and three-dimensional model generation, and converts implicit models into explicit models through edge devices; although it can also be used for large-scale three-dimensional reconstruction of indoor scenes, it still has the following disadvantages: 1) There are challenges such as high computing resource requirements, system complexity, equipment and cost issues, and long model training time; 2) It has high requirements on the professionalism of equipment and collection personnel, and is not suitable for application scenarios that require fast response or real-time data. Also disclosed is a method and system for three-dimensional reconstruction of indoor scenes based on a single RGB image (publication number: CN111414923A), which improves the accuracy of depth estimation through an indoor depth estimation network based on structural knowledge, and effectively improves the overall effect of indoor reconstruction through an indoor three-dimensional reconstruction network based on implicit neural expression; however, the above method still has the following disadvantages: 1) When processing high-resolution images and large-scale scenes, the training and reasoning process of the depth estimation network requires a lot of computing resources; 2) It involves multiple complex calculation steps such as normal vector calculation, main direction estimation, camera light calculation, etc., which puts high requirements on the implementation and maintenance of the system.

[0004] Therefore, there is an urgent need for a three-dimensional reconstruction method for indoor scenes that saves computing resources and calculation steps. Summary of the invention

[0005] In view of the above problems existing in the prior art, an object of the present invention is to provide a method and system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields.

[0006] The present invention provides a method for three-dimensional reconstruction of a static indoor scene based on a neural radiation field, comprising: acquiring data of an indoor scene to be reconstructed; and performing data preprocessing on the data of the indoor scene; wherein the data preprocessing comprises performing image noise reduction processing on the data of the indoor scene using a partition joint bilateral filtering technique;

[0007] Based on the neural radiation field algorithm, the indoor scene space to be reconstructed is divided into two or more parameterized voxel grids, and the scene information corresponding to the voxel grids is described by parameters to reconstruct the neural radiation field implicit model of the scene;

[0008] Based on the particle swarm optimization algorithm, the initial particle swarm and iteration parameters are set, and the performance of the current parameter combination of each voxel grid of the neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is used as the optimal parameter to obtain a trained neural radiation field implicit model.

[0009] Among them, an optional solution is that the method of performing image noise reduction processing on the data of the indoor scene using the partition joint bilateral filtering technology includes:

[0010] Divide the static indoor scene image data into regions and set different confidence weights for each region; restore accurate pixel values ​​at pixels with inaccurate pixel values ​​and low confidence, and reduce the error between the filtered pixel values ​​and the true values;

[0011] Joint bilateral filtering is performed on each of the regions to achieve image noise reduction processing on the data of the indoor scene.

[0012] Among them, an optional solution is to divide the static indoor scene image data into regions and set different confidence weights for each region, which is achieved by the following formula:

[0013]

[0014] Among them, Conf(x,y) is the confidence, α is the adjustment parameter, W conf (x,y) is the confidence weight;

[0015] Performing joint bilateral filtering on each region to achieve image noise reduction processing on the data of the indoor scene; this is achieved by the following formula:

[0016]

[0017] Among them, W conf (x+i,y+j) is the confidence weight, K space (i,j) is the spatial filter kernel K range (D(x+i,y+j)), D(x,y) is the filter kernel of the range, is the pixel value after filtering.

[0018] Among them, an optional solution is that the data preprocessing also includes:

[0019] Performing frame extraction processing on the data of the indoor scene to obtain static indoor scene image data;

[0020] Calibrate the static indoor scene image data using a radial distortion model in the OpenCV library;

[0021] Performing noise reduction processing on the calibrated static indoor scene image data;

[0022] Through COLMAP software processing, a JSON file containing camera parameters and the pose of each image is obtained based on the static indoor scene image data after noise reduction processing.

[0023] Among them, an optional solution is that, by processing with COLMAP software, a method for obtaining a JSON file containing camera parameters and the pose of each image based on the static indoor scene image data after noise reduction processing includes:

[0024] Use COLMAP software to estimate the pose of the camera in the scene;

[0025] Generate the corresponding transformation matrix based on the camera pose output by COLMAP;

[0026] Adjust the transformation matrix according to whether it is necessary to maintain the coordinate system used by the COLMAP software;

[0027] Using the adjusted transformation matrix, calculate the center points of all cameras;

[0028] According to the requirements of NeRF processing, the camera position is fine-tuned; the adjusted camera position and the center point of all cameras are integrated and written into the JSON file.

[0029] Among them, an optional solution is that in the step of dividing the indoor scene space to be reconstructed into more than two parameterized voxel grids based on the neural radiation field algorithm, the size of the voxel grid is dynamically adjusted according to the complexity of the scene.

[0030] Among them, an optional scheme is to set an initial particle swarm and iteration parameters based on a particle swarm optimization algorithm, and iteratively evaluate the performance of the parameter combination of each current voxel grid of the neural radiation field implicit model until the parameter combination of each voxel grid corresponding to the global optimal position is obtained. The implementation method includes:

[0031] Create a group of particles and initialize the particle group; each particle represents a parameter combination of each voxel grid of the multi-layer perception model of the neural radiation field implicit model;

[0032] Create a group of randomly initialized speeds and positions for each particle; calculate the individual best position of each particle and find the minimum fitness value in the parameter space;

[0033] Find the particle with the best fitness value in the entire particle group, and set the parameters of the particle with the best fitness value as the global optimal position; update the speed and position of each particle;

[0034] For each particle, the multi-layer perception model of the neural radiation field implicit model is trained using the current parameter combination corresponding to the particle, and the loss function value is calculated as the fitness value to perform fitness evaluation; wherein, when the current fitness value of a particle is better than the fitness value of the individual best position corresponding to the particle, the position corresponding to the current fitness value of the particle is updated to the individual best position; when the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, the individual best position of the particle is updated to the global best position;

[0035] Repeat the steps of speed and position update and fitness value evaluation until the set number of iterations is reached or the convergence condition is reached; when the particle swarm optimization algorithm converges or reaches the number of iterations, the parameter combination of each voxel grid corresponding to the global optimal position is finally obtained.

[0036] Among them, the optional solution is to assume that V ij represents the velocity of particle i in dimension j, X ij Represents the current position of particle i in dimension j. The speed and position of each particle are updated using the following formula:

[0037] V ij =w·V ij +c1·r1·(Pbest ij -X ij )+c2·r2·(Gbestj -X ij )

[0038] X ij =X ij +V ij

[0039] Among them, Pbest ij Indicates the individual best position of particle i on dimension j, Gbest j Indicates that the global optimal position is in dimension j, w is the inertia weight, c1 and c2 are acceleration factors, and r1 and r2 are random numbers.

[0040] On the other hand, the present invention also provides a static indoor scene 3D reconstruction system based on neural radiation field, and the 3D reconstruction based on the static indoor scene 3D reconstruction method based on neural radiation field as described above comprises:

[0041] A preprocessing unit, used to obtain data of an indoor scene to be reconstructed; and perform data preprocessing on the data of the indoor scene; wherein the data preprocessing includes performing image noise reduction processing on the data of the indoor scene using a partition joint bilateral filtering technique;

[0042] A voxelization unit, used for dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on a neural radiation field algorithm, and reconstructing a neural radiation field implicit model of the scene by describing the scene information corresponding to the voxel grids through parameters;

[0043] The model training unit is used to set the initial particle swarm and iteration parameters based on the particle swarm optimization algorithm, and iteratively evaluate the performance of the parameter combination of each current voxel grid of the neural radiation field implicit model until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is used as the optimal parameter to obtain a trained neural radiation field implicit model.

[0044] In another aspect, the present invention further provides an electronic device, comprising:

[0045] at least one processor; and,

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the steps in the method for three-dimensional reconstruction of a static indoor scene based on a neural radiation field as described above.

[0048] From the above technical scheme, it can be seen that the method and system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields provided by the present invention divide the indoor scene space to be reconstructed into more than two parameterized voxel grids based on the neural radiation field algorithm; and then use the particle swarm optimization algorithm to train the neural radiation field implicit model; on the one hand, it reduces the professional requirements for acquisition equipment and acquisition personnel; on the other hand, it reduces the computing resource requirements and the complexity of the computing system; it enables non-professional users to quickly and accurately complete the three-dimensional reconstruction of static indoor scenes; and at the same time improves the efficiency and quality of three-dimensional reconstruction; the three-dimensional reconstruction method of static indoor scenes based on neural radiation fields of the present invention is particularly suitable for application scenarios that require real-time data and rapid response. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings:

[0050] Figure 1 A schematic diagram of a process of a method for three-dimensional reconstruction of a static indoor scene based on a neural radiation field according to an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the logical structure of a static indoor scene 3D reconstruction system based on neural radiation field according to an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In the following description, for the purpose of illustration, in order to provide a comprehensive understanding of one or more embodiments, many specific details are set forth. However, it is apparent that these embodiments may also be implemented without these specific details. In other examples, for ease of describing one or more embodiments, known structures and devices are shown in the form of block diagrams.

[0054] In view of the existing problems, the present invention provides a method and system for three-dimensional reconstruction of static indoor scenes based on neural radiation fields, which not only solves the problem of complex calculations in the existing three-dimensional reconstruction schemes of static indoor scenes based on neural radiation fields, but also enables users to use indoor scene data captured by non-professional equipment such as mobile phones or cameras to quickly and accurately complete the technical effect of three-dimensional reconstruction of static indoor scenes.

[0055] In order to better illustrate the technical solution of the present invention, some technical terms involved in the present invention are briefly explained below.

[0056] Neural Radiance Field (NeRF) is a method based on multi-layer perceptron (MLP) and volume rendering technology that can be used for applications such as 3D reconstruction and new perspective synthesis. By taking the 3D point coordinates and perspective as input, the color and density of the point are output to implicitly represent the scene. Then, by sampling several points along each ray and performing volume integration based on their color and density, an image at the target perspective is generated. NeRF is a deep learning method for 3D scene reconstruction. It encodes the 3D scene through a multi-layer perceptron (MLP). The NeRF model can generate high-quality images of new perspectives and achieve end-to-end rendering of 3D scenes.

[0057] Joint Bilateral Filtering (JBF): It is a nonlinear filtering technique that combines the bilateral characteristics of the spatial domain and the intensity domain to simultaneously consider the spatial proximity between pixels and the similarity of pixel values. In joint bilateral filtering, the spatial filter kernel and the range filter kernel usually appear in the form of Gaussian functions.

[0058] Multilayer Perceptron (MLP) is a common artificial neural network architecture, commonly used for deep learning tasks. MLP consists of multiple neurons distributed in multiple layers, including an input layer, at least one or more hidden layers, and an output layer.

[0059] Particle Swarm Optimization (PSO) is an optimization technique based on swarm intelligence, proposed by Eberhart and Kennedy in 1995. It is inspired by the foraging behavior of bird flocks and solves problems by simulating the social behavior of biological groups such as bird flocks or fish schools. Among them, particle swarm: Each solution in the algorithm is regarded as a "particle" in the search space, representing a potential solution to the problem. Speed: Each particle has a speed that guides its movement in the search space. Position: Each particle updates its position in the search space according to its speed and position. Individual extreme value: Each particle remembers the optimal solution it has found, called the individual extreme value (pBest). Global extreme value: The optimal solution found by all particles in the entire particle swarm is called the global extreme value (gBest).

[0060] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0061] It should be noted that the following description of the exemplary embodiments is merely illustrative and is not intended to limit the present invention and its application or use. Technologies and devices known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the technologies and devices should be considered as part of the specification.

[0062] In order to illustrate the method and system for 3D reconstruction of static indoor scenes based on neural radiation fields provided by the present invention, Figure 1 The process of the method for three-dimensional reconstruction of a static indoor scene based on a neural radiation field according to an embodiment of the present invention is exemplarily indicated.

[0063] like Figure 1 As shown, the method for 3D reconstruction of static indoor scenes based on neural radiation fields provided by the present invention mainly includes steps S110 to S130.

[0064] S110, acquiring data of an indoor scene to be reconstructed; and performing data preprocessing on the data of the indoor scene; wherein the data preprocessing includes performing image noise reduction processing on the data of the indoor scene using a partitioned joint bilateral filtering technique.

[0065] S120. Divide the indoor scene space to be reconstructed into two or more parameterized voxel grids based on a neural radiation field algorithm, and describe the scene information corresponding to the voxel grids through parameters to reconstruct a neural radiation field implicit model of the scene.

[0066] S130. Based on the particle swarm optimization algorithm, the initial particle swarm and iteration parameters are set, and the performance of the current parameter combination of each voxel grid of the neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is used as the optimal parameter to obtain a trained neural radiation field implicit model.

[0067] The following will describe in detail the three-dimensional reconstruction method of static indoor scenes based on neural radiation fields provided by the present invention in steps.

[0068] First, in step S110, data of the indoor scene to be reconstructed is obtained; and data preprocessing is performed on the data of the indoor scene; wherein the data preprocessing includes performing image denoising processing on the data of the indoor scene using a partitioned joint bilateral filtering technique.

[0069] In particular, it refers to a method of acquiring indoor scene data using non-professional devices such as portable devices such as smart phones, cameras, tablet computers, etc. Specifically, the process of acquiring indoor scene data can be, but is not limited to, using a portable device to shoot an indoor scene to acquire image data of the scene; and using the built-in sensors of the device, such as an accelerometer and a gyroscope, to collect the spatial positioning information of the device.

[0070] The data of the indoor scene to be reconstructed includes an RGB image of the static indoor scene to be reconstructed and a JSON file containing camera parameters and the pose of each image.

[0071] The data preprocessing in step S110 not only includes image noise reduction processing, but the entire preprocessing process includes the following steps S111 to S114.

[0072] S111. Frame extraction is performed on the data of the indoor scene to obtain static indoor scene image data; specifically, if the input indoor scene data is video data, the ffmpeg library is used to extract the video sequence to obtain static indoor scene image data. The user can select the video frame extraction parameters by himself; S112. The static indoor scene image data is calibrated by the radial distortion model in the OpenCV library; S113. The calibrated static indoor scene image data is subjected to noise reduction processing; the partition joint bilateral filtering technology is used to perform image noise reduction processing on the data of the indoor scene; specifically, the partition joint bilateral filtering is used as the image noise reduction technology, considering the spatial information of the image and the grayscale difference between pixels, reducing noise and retaining the details and edge information in the image. S114. Through COLMAP software processing, a JSON file containing camera parameters and the pose of each image is obtained based on the static indoor scene image data after noise reduction processing. The user can specify the file input and output paths, and output the processed image files to the specified folder. In general, the static indoor scene image data is calibrated through the radial distortion model in the OpenCV library, and the COLMAP software estimates the camera pose. After generating the transformation matrix, the transformation matrix is ​​adjusted according to whether the COLMAP coordinates are maintained, and the center points of all cameras are calculated. The camera position is adjusted to make the scene suitable for NeRF processing, and finally the processed data is written to the JSON file. Among them, the camera parameters include the intrinsic parameters of the camera (such as focal length, principal point coordinates, pixel size) and extrinsic parameters (i.e. camera position and rotation, which can be expressed as rotation matrix and translation vector). The intrinsic parameters determine how the image is projected, while the extrinsic parameters describe the pose of the camera in the world coordinate system. The pose of an image refers to the position and orientation of each image relative to a reference coordinate system (usually the world coordinate system). The pose includes translation (the position of the camera when the image is taken) and rotation (the direction of the camera when the image is taken), which determines the part of the scene captured in the image; that is, the camera position is part of the image pose.

[0073] For the method of obtaining a JSON file containing camera parameters and the pose of each image based on the static indoor scene image data after noise reduction processing by COLMAP software in step S114, S1141, using COLMAP software to estimate the pose of the camera in the scene; S1142, generating a corresponding transformation matrix according to the camera pose output by COLMAP; S1143, adjusting the transformation matrix according to whether it is necessary to maintain the coordinate system used by the COLMAP software; S1144, using the adjusted transformation matrix, calculating the center point of all cameras; it should be noted that the center point of the camera usually refers to the geometric center or center of mass of all camera positions, which can be used as a reference point for the scene or for estimating the scope and layout of the scene. S1145, fine-tuning the position of the camera according to the requirements of NeRF processing; integrating the adjusted camera position and the center points of all cameras and writing them into the JSON file; wherein the camera parameters output by COLMAP are automatically imported into the JSON file.

[0074] Specifically, in step S110, the present invention enables users to use non-professional devices such as mobile phones or cameras to capture indoor scene data. Compared with the prior art, which usually requires highly professional equipment to capture the same type of data, the present invention significantly reduces the technical threshold, making three-dimensional scene capture more convenient and popular.

[0075] In step S113, the method of performing image noise reduction processing on the data of the indoor scene using the partition joint bilateral filtering technology includes the following steps S1131-S1132.

[0076] S1131, the static indoor scene image data is divided into regions, and different confidence weights are set for each region; accurate pixel values ​​are restored at pixels with inaccurate pixel values ​​and low confidence, and the error between the filtered pixel value and the true value is reduced. In order to correctly deal with the confidence differences in different regions in the static indoor scene image, the static indoor scene image data is divided into regions, and different confidence weights are set for each region. Thereby, the characteristics of different regions can be better processed in the joint bilateral filtering.

[0077] Specifically, the static indoor scene image data is divided into regions, and different confidence weights are set for each region, which is achieved by the following formula:

[0078]

[0079] Among them, Conf(x,y) is the confidence, α is the adjustment parameter, W conf (x,y) is the confidence weight.

[0080] Restore accurate pixel values ​​at pixels with inaccurate pixel values ​​and low confidence; assuming that Dfiltered is the filtered pixel value and Dtrue is the true value, in order to effectively improve the accuracy of 3D reconstruction of indoor scenes, it is necessary to reduce the error Error between the filtered pixel value and the true value:

[0081]

[0082] It should be noted that confidence is partitioned first, and then pixels are restored. The present invention adopts an innovative image processing technology, namely a partitioned joint bilateral filtering method, to improve the ability to cope with noise and pixel loss. This method fully utilizes the advantages of bilateral filtering while taking into account the confidence differences in different regions of the image. Specifically, it first divides the image into multiple regions, and then assigns different weights to each region. The purpose of this is to be able to more accurately process the unique characteristics of each region when performing joint bilateral filtering. In this process, for pixels with low confidence, the method can maintain a high sensitivity to ensure that these pixels are properly processed. By setting different confidence weights for different regions, the method can better adapt to the characteristics of different regions in the image during the filtering process, thereby achieving a higher quality image restoration effect. This partitioned joint bilateral filtering method effectively improves the accuracy and efficiency of image processing by intelligently adjusting weights and filtering strategies.

[0083] S1132, performing joint bilateral filtering on each of the regions to achieve image noise reduction processing on the data of the indoor scene. That is, by partitioning the static indoor scene image data, setting different weights for each partition and performing joint bilateral filtering respectively, sensitivity to low confidence pixels is maintained.

[0084] Specifically, a joint bilateral filtering is performed on each region to achieve image noise reduction processing on the data of the indoor scene; this is achieved by the following formula:

[0085]

[0086] Among them, W conf (x+i,y+j) is the confidence weight, K space (i,j) is the spatial filter kernel K range (D(x+i,y+j)), D(x,y) is the filter kernel of the range, is the pixel value after filtering.

[0087] For step S120, the indoor scene space to be reconstructed is divided into two or more parameterized voxel grids based on the neural radiation field algorithm, and the scene information corresponding to the voxel grids is described by parameters to reconstruct the neural radiation field implicit model of the scene. That is to say, based on the neural radiation field algorithm, the scene is divided into multiple parameterized voxel grids, each small block is described by a small number of parameters to describe the scene information inside it, and the NeRF model is used to reconstruct the scene containing the preset three-dimensional space information of the static indoor scene to be reconstructed.

[0088] The three-dimensional reconstruction of indoor target scenes using neural radiation fields mainly includes several steps: data collection, network training, voxelization, reconstruction of scenes, and grid reconstruction. The first step, data collection, is mainly to collect a set of sample images taken by the camera at multiple perspectives. These images should cover all angles and perspectives in the target scene. The second step, network training, is mainly to use the collected sample images to train the NeRF model. The input of the model is the direction and starting point of the light, and the output is the radiation intensity and color at the light. The NeRF model usually adopts a structure similar to a neural network, such as a multi-layer perceptron (MLP) or a convolutional neural network (CNN). The third step, voxelization, is mainly used to divide the scene space into voxel grids and generate a set of light in each voxel grid. These light will be used to query the NeRF model to obtain the radiation intensity and color of each point in the scene. The fourth step, reconstructing the scene, is mainly based on the light in each voxel grid, and its radiation intensity and color are obtained by querying the NeRF model. Based on the information obtained, a dense sampling of points in each voxel grid can be generated, and the color of each point can be calculated. The last step, mesh reconstruction, is used to convert the reconstructed point cloud into a mesh model and perform post-processing operations such as texture mapping. Point clouds can be converted into mesh models using methods such as triangulation.

[0089] In the step of dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiation field algorithm, the size of the voxel grid is dynamically adjusted according to the scene complexity to improve the reconstruction efficiency and accuracy; specifically, through the multi-scale hash table based on the neural radiation field and the optimized data structure, the fast retrieval of the neural network weights is realized, and the number of model parameters is effectively reduced. Specifically, using multi-scale representation, NeRF can process input data of different resolutions more efficiently. This method allows the model to capture the details of the scene at different levels, thereby improving the rendering quality and reducing the amount of calculation. Efficient data structures, such as KD trees or octrees, are used to organize and query data in 3D space. These data structures allow NeRF to quickly locate and process specific areas in the scene, thereby reducing unnecessary calculations. By introducing hash tables, NeRF can quickly index weights in the neural network. By using hash tables and efficient data structures, the NeRF model can significantly reduce the number of parameters of the network. The reduction in the number of parameters not only reduces the memory usage of the model, but also may increase the training speed and inference speed of the model. The fast indexing capability means that NeRF can quickly retrieve the required weights from the neural network, which is crucial for real-time rendering and synthesis of new perspectives. The reduced number of parameters and efficient data structures help optimize memory usage and computing resource allocation, allowing the NeRF model to be deployed in resource-constrained environments. In addition, the method also speeds up the forward propagation speed of the neural network. By fine-tuning and optimizing the voxel parameters, the technology of the present invention can adapt to changing scene and perspective requirements, enhance the matching accuracy and processing speed, and support real-time three-dimensional scene updates and reconstruction. In short, the present invention provides strong support for real-time three-dimensional vision tasks by reducing the computational burden and improving processing efficiency.

[0090] In step S130, during the PSO algorithm optimization process, an initial particle group and iteration parameters are set, and the performance of the current parameter combination is evaluated in each iteration to gradually approach the global optimal parameter setting. Specifically, based on the particle swarm optimization algorithm, an initial particle group and iteration parameters are set, and the performance of the parameter combination of each current voxel grid of the neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained. The implementation method includes the following steps.

[0091] S131. Create a group of particles and initialize the particle group; wherein each particle represents a parameter combination of each voxel grid of the multi-layer perception model of the neural radiation field implicit model.

[0092] Assume P i represents the parameter combination of particle i, n represents the number of parameters, and the initialized particle swarm is expressed as:

[0093] P i =(p i1 ,p i2 ,...,p in )

[0094] S132. Create a group of randomly initialized speeds and positions for each particle.

[0095] Assume X is the initialization speed, x ij To initialize the position, the initial speed and position are expressed as:

[0096] X ij =(x ij1 ,x ij2 ,...,x ijn )

[0097] Calculate the individual best position for each particle and find the minimum fitness value in parameter space.

[0098] Assume that Pbest is the optimal position of the initialized individual and f(Pi) is the optimal parameter position.

[0099] Pbest i =argmin(f(P i ))

[0100] S133, finding a particle with the best fitness value in the entire particle swarm, and setting the parameters of the particle with the best fitness value as the global optimal position.

[0101] Assume that Gbest is the initial global best position, and the initial global best position is expressed as:

[0102] Gbest = argmin(f(P i ))

[0103] For each particle, the velocity and position are updated.

[0104] Specifically, assuming that V ij represents the velocity of particle i in dimension j, X ij Represents the current position of particle i in dimension j. The speed and position of each particle are updated using the following formula:

[0105] V ij =w·V ij +c1·r1·(Pbest ij -X ij )+c2·r2·(Gbest j -X ij )

[0106] Xij =X ij +V ij

[0107] Among them, Pbest ij Indicates the individual best position of particle i on dimension j, Gbest j Indicates that the global optimal position is in dimension j, w is the inertia weight, c1 and c2 are acceleration factors, and r1 and r2 are random numbers.

[0108] S134. For each particle, use the current parameter combination corresponding to the particle to train the multi-layer perception model of the neural radiation field implicit model, and calculate the loss function value as the fitness value to perform fitness evaluation. That is, for each particle, use its current parameter combination to train the MLP model, and calculate its prediction error or loss function value as the fitness value to perform fitness evaluation.

[0109] f(P i )=Loss function(P i )

[0110] Among them, when the current fitness value of a particle is better than the fitness value of the individual best position corresponding to the particle, the position corresponding to the current fitness value of the particle is updated to the individual best position. In other words, if the current fitness value of a particle is better than the fitness value of its individual best position, then the individual best position is updated.

[0111] Pbest i =argmin(f(P i ))

[0112] When the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, the individual best position of the particle is updated to the global best position. In other words, if the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, then the global best position is updated.

[0113] Gbest = argmin(f(P i ))

[0114] S135, repeat the steps of speed and position updating and fitness value evaluation until the set number of iterations is reached or the convergence condition is reached; when the particle swarm optimization algorithm converges or reaches the number of iterations, the parameter combination of each voxel grid corresponding to the global optimal position is finally obtained.

[0115] In the specific implementation process, after obtaining the trained neural radiation field implicit model, the trained neural radiation field implicit model is visualized using volume rendering technology. The user management module can also be used to manage user permissions, process user input, and output three-dimensional reconstruction results, so that users can easily operate the system and view the reconstruction effect. In addition, a real-time web viewer can be set up to allow users to visualize the results in real time while training the NeRF model; support geometry export, which is for artists and creators who need to export 3D reconstructions to game engines or other downstream tools. The module supports multiple export methods and output formats, including point clouds and meshes.

[0116] The combination of the above three steps S110 to S130 constitutes the static indoor scene 3D reconstruction method based on neural radiation field provided by the present invention. The method takes neural radiation field as the core technology, can effectively realize high-quality scene reconstruction, rendering and reasoning, and has real-time 3D scene update and reconstruction functions. The comprehensive advantages of the present invention make it have broad application prospects in the fields of virtual reality, augmented reality, game development, etc.

[0117] Corresponding to the above-mentioned three-dimensional reconstruction method of static indoor scenes based on neural radiation fields, the present invention also provides a three-dimensional reconstruction system of static indoor scenes based on neural radiation fields, and performs video production based on the above-mentioned three-dimensional reconstruction method of static indoor scenes based on neural radiation fields. Figure 2 The logical framework structure of a static indoor scene 3D reconstruction system based on neural radiation field according to an embodiment of the present invention is shown.

[0118] like Figure 2 As shown, the present invention provides a static indoor scene three-dimensional reconstruction system 200 based on neural radiation field, which mainly includes a preprocessing unit 210, a voxelization unit 220 and a model training unit 230.

[0119] The preprocessing unit 210 is used to obtain data of the indoor scene to be reconstructed; and perform data preprocessing on the data of the indoor scene; wherein the data preprocessing includes performing image noise reduction processing on the data of the indoor scene using a partition joint bilateral filtering technique;

[0120] A voxelization unit 220 is used to divide the indoor scene space to be reconstructed into two or more parameterized voxel grids based on a neural radiation field algorithm, and describe the scene information corresponding to the voxel grids through parameters to reconstruct a neural radiation field implicit model of the scene;

[0121] The model training unit 230 is used to set the initial particle swarm and iteration parameters based on the particle swarm optimization algorithm, iteratively evaluate the performance of the parameter combination of each voxel grid of the neural radiation field implicit model, until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is used as the optimal parameter to obtain the trained neural radiation field implicit model.

[0122] The above-mentioned static indoor scene three-dimensional reconstruction system based on neural radiation field is an implementation method corresponding to the above-mentioned static indoor scene three-dimensional reconstruction method based on neural radiation field. Its other more specific implementation embodiments can refer to the specific embodiments of the above-mentioned static indoor scene three-dimensional reconstruction method based on neural radiation field, which will not be described in detail here.

[0123] From the above embodiments, it can be seen that the method and system for 3D reconstruction of static indoor scenes based on neural radiation fields proposed by the present invention allow users to capture indoor scene images using popular non-professional devices such as smartphones or cameras. It breaks through the limitations of traditional 3D reconstruction and has the following significant features:

[0124] 1) Speed: Users can quickly complete the entire process from shooting to 3D model generation, greatly shortening the reconstruction time.

[0125] 2) Accuracy: Advanced image processing techniques, such as noise reduction algorithms, ensure high accuracy and quality of the reconstruction results.

[0126] 3) Optimized voxel grid division: The parameterized voxel grid division technology is used to improve the efficiency and accuracy of the spatial representation of the scene.

[0127] 4) Particle parameter optimization algorithm: In combination with the parameterized voxel grid division of the neural radiation field of the present invention, advanced algorithms such as the particle swarm optimization algorithm are introduced to perform intelligent optimization of model parameters, further improving the efficiency and effect of model training.

[0128] 5) Solving existing problems: The present invention provides an effective solution to the problems existing in the prior art, such as low reconstruction quality, slow speed and large consumption of computing resources.

[0129] 6) User-friendly: Especially suitable for all kinds of users, including non-technical users, making 3D reconstruction technology more accessible and easy to use.

[0130] 7) Real-time: Supports efficient and real-time 3D reconstruction of indoor scenes, suitable for application scenarios that require indoor scene updates and dynamic reconstruction.

[0131] Figure 3 Schematic diagram of an electronic device according to an embodiment of the present invention. Figure 3As shown, the present invention also provides an electronic device, the electronic device comprising:

[0132] at least one processor; and,

[0133] a memory communicatively connected to at least one processor; wherein,

[0134] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the steps in the aforementioned method for three-dimensional reconstruction of static indoor scenes based on neural radiation fields.

[0135] It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0136] Electronic devices include, but are not limited to, extended reality (XR) devices, virtual reality (VR) devices, augmented reality (AR) devices, smart phones, tablet computers, laptops, and desktop computers. It should be noted that the execution subject of this embodiment can be an electronic device or a server. If the execution subject is a server, after completing the model fusion, the server can send the target scene model to the electronic device that has the rendering requirement for the target scene for deployment.

[0137] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0138] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0139] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0140] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0141] The static indoor scene 3D reconstruction program 12 based on neural radiation field stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0142] S110, acquiring data of an indoor scene to be reconstructed; and performing data preprocessing on the data of the indoor scene; wherein the data preprocessing includes performing image noise reduction processing on the data of the indoor scene using a partition joint bilateral filtering technique;

[0143] S120, dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on a neural radiation field algorithm, and describing scene information corresponding to the voxel grids by parameters to reconstruct a neural radiation field implicit model of the scene;

[0144] S130. Based on the particle swarm optimization algorithm, the initial particle swarm and iteration parameters are set, and the performance of the current parameter combination of each voxel grid of the neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is used as the optimal parameter to obtain a trained neural radiation field implicit model.

[0145] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figure 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0146] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0147] As described above with reference to the accompanying drawings, the method and system for 3D reconstruction of static indoor scenes based on neural radiation fields proposed in accordance with the present invention are described by way of example. However, those skilled in the art should understand that various improvements can be made to the method and system for 3D reconstruction of static indoor scenes based on neural radiation fields proposed in the present invention without departing from the content of the present invention. Therefore, the scope of protection of the present invention should be determined by the contents of the attached claims.

Claims

1. A method for 3D reconstruction of static indoor scenes based on neural radiation fields, characterized in that: include: Acquire data of the indoor scene to be reconstructed; and performing data preprocessing on the data of the indoor scene; wherein the data preprocessing includes performing image noise reduction processing on the data of the indoor scene using a partition joint bilateral filtering technique; Based on the neural radiation field algorithm, the indoor scene space to be reconstructed is divided into two or more parameterized voxel grids, and the scene information corresponding to the voxel grids is described by parameters to reconstruct the neural radiation field implicit model of the scene; Based on the particle swarm optimization algorithm, the initial particle swarm and iteration parameters are set, and the performance of the current parameter combination of each voxel grid of the neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is used as the optimal parameter to obtain a trained neural radiation field implicit model.

2. The method for 3D reconstruction of static indoor scenes based on neural radiation fields according to claim 1, characterized in that: The method of performing image noise reduction processing on the data of the indoor scene by using the partition joint bilateral filtering technology includes: The static indoor scene image data is divided into regions, and different confidence weights are set for each region; accurate pixel values ​​are restored at pixels with inaccurate pixel values ​​and low confidence, and the error between the filtered pixel values ​​and the true values ​​is reduced; Joint bilateral filtering is performed on each of the regions to achieve image noise reduction processing on the data of the indoor scene.

3. The method for 3D reconstruction of static indoor scenes based on neural radiation fields according to claim 2, characterized in that: The static indoor scene image data is divided into regions, and different confidence weights are set for each region, which is achieved by the following formula: Among them, Conf(x,y) is the confidence, α is the adjustment parameter, W conf (x,y) is the confidence weight; Performing joint bilateral filtering on each region to achieve image noise reduction processing on the data of the indoor scene; this is achieved by the following formula: Among them, W conf (x+i,y+j) is the confidence weight, K space (i,j) is the spatial filter kernel K range (D(x+i,y+j)), D(x,y) is the filter kernel of the range, is the pixel value after filtering.

4. The method for 3D reconstruction of static indoor scenes based on neural radiation fields according to claim 1, characterized in that: The data preprocessing also includes: Performing frame extraction processing on the data of the indoor scene to obtain static indoor scene image data; Calibrate the static indoor scene image data using a radial distortion model in the OpenCV library; Performing noise reduction processing on the calibrated static indoor scene image data; Through COLMAP software processing, a JSON file containing camera parameters and the pose of each image is obtained based on the static indoor scene image data after noise reduction processing.

5. The method for 3D reconstruction of static indoor scenes based on neural radiation fields according to claim 1, characterized in that: The method of obtaining a JSON file containing camera parameters and each image pose based on the static indoor scene image data after noise reduction by COLMAP software includes: Use COLMAP software to estimate the pose of the camera in the scene; Generate the corresponding transformation matrix based on the camera pose output by COLMAP; Adjust the transformation matrix according to whether it is necessary to maintain the coordinate system used by the COLMAP software; Using the adjusted transformation matrix, calculate the center points of all cameras; According to the requirements of NeRF processing, the camera position is fine-tuned; the adjusted camera position and the center point of all cameras are integrated and written into the JSON file.

6. The method for 3D reconstruction of static indoor scenes based on neural radiation fields according to claim 1, characterized in that: In the step of dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on the neural radiance field algorithm, the size of the voxel grid is dynamically adjusted according to the complexity of the scene.

7. The method for 3D reconstruction of static indoor scenes based on neural radiation fields according to claim 1, characterized in that: Based on the particle swarm optimization algorithm, the initial particle swarm and iteration parameters are set, and the performance of the parameter combination of each voxel grid of the neural radiation field implicit model is iteratively evaluated until the parameter combination of each voxel grid corresponding to the global optimal position is obtained. The implementation method includes: Create a group of particles and initialize the particle group; each particle represents a parameter combination of each voxel grid of the multi-layer perception model of the neural radiation field implicit model; Create a group of randomly initialized speeds and positions for each particle; calculate the individual best position of each particle and find the minimum fitness value in the parameter space; Find the particle with the best fitness value in the entire particle group, and set the parameters of the particle with the best fitness value as the global optimal position; update the speed and position of each particle; For each particle, the multi-layer perception model of the neural radiation field implicit model is trained using the current parameter combination corresponding to the particle, and the loss function value is calculated as the fitness value to perform fitness evaluation; wherein, when the current fitness value of a particle is better than the fitness value of the individual best position corresponding to the particle, the position corresponding to the current fitness value of the particle is updated to the individual best position; when the fitness value corresponding to the individual best position of a particle is better than the fitness value of the global best position, the individual best position of the particle is updated to the global best position; Repeat the steps of speed and position update and fitness value evaluation until the set number of iterations is reached or the convergence condition is reached; when the particle swarm optimization algorithm converges or reaches the number of iterations, the parameter combination of each voxel grid corresponding to the global optimal position is finally obtained.

8. The method for 3D reconstruction of static indoor scenes based on neural radiation fields according to claim 7, characterized in that: Assume V ij represents the velocity of particle i in dimension j, X ij Represents the current position of particle i in dimension j. The speed and position of each particle are updated using the following formula: V ij =w·V ij +c1·r1·(Pbest ij -X ij )+c2·r2·(Gbest j -X ij ) X ij =X ij +V ij Among them, Pbest ij Indicates the individual best position of particle i on dimension j, Gbest j Indicates that the global optimal position is in dimension j, w is the inertia weight, c1 and c2 are acceleration factors, and r1 and r2 are random numbers.

9. A system for 3D reconstruction of static indoor scenes based on neural radiation fields, which performs 3D reconstruction based on the method for 3D reconstruction of static indoor scenes based on neural radiation fields as claimed in any one of claims 1 to 8, comprising: A preprocessing unit, used for acquiring data of an indoor scene to be reconstructed; and performing data preprocessing on the data of the indoor scene; wherein the data preprocessing includes performing image noise reduction processing on the data of the indoor scene using a partition joint bilateral filtering technique; A voxelization unit, used for dividing the indoor scene space to be reconstructed into two or more parameterized voxel grids based on a neural radiation field algorithm, and reconstructing a neural radiation field implicit model of the scene by describing the scene information corresponding to the voxel grids through parameters; The model training unit is used to set the initial particle swarm and iteration parameters based on the particle swarm optimization algorithm, and iteratively evaluate the performance of the parameter combination of each current voxel grid of the neural radiation field implicit model until the parameter combination of each voxel grid corresponding to the global optimal position is obtained; the parameter combination of each voxel grid corresponding to the global optimal position is used as the optimal parameter to obtain a trained neural radiation field implicit model.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the steps in the method for three-dimensional reconstruction of a static indoor scene based on a neural radiation field as described in any one of claims 1-8.

Images