Scene three-dimensional reconstruction method and device based on neural radiation field, medium and product

Abstract

The invention discloses a scene three-dimensional reconstruction method and device based on a neural radiation field, a medium and a product, and relates to the technical field of scene three-dimensional reconstruction, and the method comprises the steps: collecting a plurality of color images of a to-be-reconstructed scene; inputting each color image into a pre-training depth model to obtain a corresponding scene depth map; dividing the scene to be reconstructed into a plurality of grid units; initializing a multi-layer perceptron of each grid unit; sampling rays corresponding to each pixel point on each color image to obtain a plurality of sampling points of each ray, and determining spatial parameters of each sampling point; performing multi-round training on all initial multi-layer perceptron by utilizing the depth values of multiple groups of random pixel point groups on all scene depth maps and the space parameters of all sampling points on the corresponding rays to obtain multiple target multi-layer perceptron; and performing three-dimensional reconstruction on the scene to be reconstructed based on the scene representation of all sampling points output by each target perceptron. According to the invention, the speed of scene three-dimensional reconstruction is improved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional scene reconstruction, and in particular to a method, device, medium and product for three-dimensional scene reconstruction based on neural radiation field. Background Art

[0002] 3D reconstruction can transform real-world scenes into digital models, enabling machines to understand the surrounding environment in three dimensions, just like humans. 3D reconstruction is a key technology in the fields of virtual reality, augmented reality, and robot navigation. It is also the data basis for computer graphics algorithms and technologies, and plays a key role in engineering modeling and simulation. Traditional 3D reconstruction methods are mainly based on structured light, binocular vision, depth sensors, lidar, etc. Despite great achievements, they still face many challenges and limitations. Traditional algorithms have high requirements on the quality and diversity of input data. Algorithms usually need to obtain high-quality images or point cloud data from multiple angles. If the data quality is not high, the reconstruction results are often affected, resulting in insufficient accuracy or reconstruction failure. Secondly, when dealing with small objects or complex textured surfaces, traditional reconstruction algorithms have difficulty capturing the geometric structure or surface texture of complex objects. In comparison, image-based 3D reconstruction is cheaper and easier to implement.

[0003] The introduction of Neural Radiance Field (NeRF) has brought a strong driving force to the research of three-dimensional scene reconstruction. The output color and density are projected into the image using a multilayer perceptron (MLP) with an implicit neural scene representation and a classic volume rendering method to obtain a rendering result with complex geometry and appearance. However, the original neural radiation field model uses an MLP network to process the entire scene. This processing method will not only reduce the model's ability to extract local details as the scene scale increases, but also reduce the model's rendering speed. The original neural radiation field model has the problem of shape-radiance ambiguity in scene reconstruction, which makes it impossible to reconstruct accurate geometry. Zhang et al. proposed NeRF++, which divides the entire scene into two parts. The inner volume contains the close view and all cameras, while the outer volume contains the rest of the environment, which greatly avoids this problem. The two parts are modeled by two independent NeRFs, and finally the two parts are synthesized. Rebain et al. proposed DeRF, which uses a Voronoi diagram to decompose the scene and then uses a smaller network to render each part separately, achieving an inference efficiency that is 3 times faster than NeRF. Kilo NeRF divides the scene into thousands of smaller networks and trains them using a teacher-student distillation network, which significantly improves the rendering speed. Wei et al. combined traditional sparse reconstruction with learning-based deep priors into neural radiance fields. Deng et al. introduced deep supervision into neural radiance fields for the first time. SFM obtains the depth of some sparse point clouds when calculating the camera pose of the scene image, which can be used as scene depth supervision information. Although there have been many improvement methods, there are still challenges in the extraction of local detail information and reconstruction rendering. However, most of the above-mentioned various improvement methods for neural radiance fields are improvements in overall quality. They still use an MLP to extract scene information. Complex scenes will cause the MLP to forget the problem, which makes the scene 3D reconstruction slower. Summary of the invention

[0004] The purpose of this application is to provide a method, device, medium and product for three-dimensional reconstruction of a scene based on a neural radiation field, so as to solve the problem of slow speed of three-dimensional reconstruction of a scene using the existing three-dimensional reconstruction method based on a neural radiation field.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for three-dimensional reconstruction of a scene based on a neural radiation field, comprising:

[0007] Using a camera to collect multiple color images of the scene to be reconstructed;

[0008] Input each color image into a pre-trained depth model to obtain a corresponding scene depth map; the pre-trained depth model is obtained by training a deep network using a training set, and the deep network is built based on Transformer;

[0009] Divide the scene to be reconstructed into multiple grid units;

[0010] Initializing the multilayer perceptron of each of the grid units to obtain a plurality of initial multilayer perceptrons; the grid units and the multilayer perceptrons correspond one to one;

[0011] The rays of the camera corresponding to each pixel point on each color image are sampled respectively to obtain multiple sampling points of each ray, and the spatial parameters of each sampling point are determined; one grid unit corresponds to multiple pixels, and one pixel corresponds to one ray; the spatial parameters include: three-dimensional space coordinates and observation direction;

[0012] Using the depth values ​​of multiple groups of random pixel points on all scene depth maps and the spatial parameters of all sampling points on the rays corresponding to each random pixel point group, all initial multi-layer perceptrons are trained for multiple rounds to obtain a target multi-layer perceptron for each grid unit; the random pixel point group is two pixels randomly selected on the scene depth map;

[0013] Based on the scene representation of all sampling points output by each target sensor, the scene to be reconstructed is three-dimensionally reconstructed; the scene representation includes: RGB color and volume density.

[0014] Optionally, the training process of the pre-trained deep model includes:

[0015] Acquire a training set; the training set includes: color images of multiple training scenes and corresponding actual scene depth maps;

[0016] Constructing the deep network; the deep network includes: an embedding layer, multiple Transformers, multiple reorganization layers, multiple fusion modules and a detection head;

[0017] The deep network is trained with the color image of each training scene as input and the corresponding actual scene depth map as output to obtain the pre-trained deep model.

[0018] Optionally, all initial multi-layer perceptrons are trained multiple times using the depth values ​​of multiple groups of random pixel points on all scene depth maps and the spatial parameters of all sampling points on the rays corresponding to each random pixel point group to obtain a target multi-layer perceptron for each grid unit, including:

[0019] The training process of the multilayer perceptron of each grid unit in any current round includes:

[0020] Determine any pixel point on any color image as the current pixel point, and determine the ray corresponding to the current pixel point as the current ray;

[0021] Determine the updated multilayer perceptron of each grid unit in the previous round as the multilayer perceptron of each grid unit in the current round;

[0022] Determine the scene representation of each sampling point in the current round based on the spatial parameters of each sampling point on the current ray and the multi-layer perceptron in the current round of the grid unit corresponding to the current ray;

[0023] Determine the pixel color and depth value of the current ray in the current round based on the scene representation in the current round of all sampling points on the current ray;

[0024] Determine the loss value in the current round based on the pixel colors of all pixels on all color images and the pixel colors of the corresponding rays in the current round, and the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of the corresponding rays in the current round;

[0025] Determine whether the training stop condition is met; the training stop condition is that the current round reaches a preset number of training times or the loss value in the current round is less than a preset loss;

[0026] If yes, the multilayer perceptron of each grid unit in the current round is determined as the target multilayer perceptron of each grid unit;

[0027] If not, the multilayer perceptron of each grid unit in the current round is updated to obtain the updated multilayer perceptron of each grid unit in the current round, and the updated multilayer perceptron of each grid unit in the previous round is updated to the updated multilayer perceptron of each grid unit in the current round, and the process returns to "determine the updated multilayer perceptron of each grid unit in the previous round as the multilayer perceptron of each grid unit in the current round" until the target multilayer perceptron of each grid unit is obtained.

[0028] Optionally, based on the spatial parameters of each sampling point on the current ray and the multi-layer perceptron of the grid unit corresponding to the current ray in the current round, determining the scene representation of each sampling point in the current round includes:

[0029] Perform dimension-upgrading encoding on the spatial parameters of each sampling point on the current ray to obtain the dimension-upgraded spatial parameters of each sampling point on the current ray;

[0030] The dimensionally upgraded spatial parameters of each sampling point on the current ray are respectively input into the multi-layer perceptron of the current round of the grid unit corresponding to the current ray to obtain the scene representation of each sampling point in the current round.

[0031] Optionally, based on the scene representation of all sampling points on the current ray in the current round, determining the pixel color and depth value of the current ray in the current round includes:

[0032] Calculate the pixel color of the current ray in the current round according to the RGB color and volume density of all sampling points on the current ray in the current round;

[0033] According to the volume density of all sampling points on the current ray in the current round, the depth value of the current ray in the current round is calculated.

[0034] Optionally, based on the pixel colors of all pixels on all color images and the pixel colors of corresponding rays in the current round and the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of corresponding rays in the current round, determining the loss value in the current round includes:

[0035] Based on the pixel colors of all pixels on all color images and the pixel colors of the corresponding rays in the current round, the color difference value in the current round is calculated;

[0036] Calculate the depth constraint value in the current round based on the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of the corresponding rays in the current round;

[0037] Based on the color difference value and the depth constraint value in the current round, the loss value in the current round is determined.

[0038] Optionally, based on the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of corresponding rays in the current round, the depth constraint value in the current round is calculated, including:

[0039] Determine any random pixel point group on any scene depth map as the current pixel point group;

[0040] Determine that a relationship between depth values ​​of two pixels in the current pixel group on the scene depth map is a first relationship; the first relationship is greater than, less than or equal to;

[0041] Determine that the relationship between the depth values ​​of the rays corresponding to the two pixel points in the current pixel point group in the current round is a second relationship; the second relationship is greater than, less than or equal to;

[0042] The depth constraint value in the current round is calculated based on the depth values ​​in the current round of the rays corresponding to all random pixel point groups for which the first relationship and the second relationship are inconsistent.

[0043] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for three-dimensional reconstruction of a scene based on a neural radiation field.

[0044] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for three-dimensional reconstruction of a scene based on a neural radiation field.

[0045] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for three-dimensional reconstruction of a scene based on a neural radiation field.

[0046] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0047] The present application discloses a method, device, medium and product for scene three-dimensional reconstruction based on neural radiation field. First, a camera is used to collect multiple color images of the scene to be reconstructed; each color image is input into a pre-trained depth model respectively to obtain a corresponding scene depth map; secondly, the scene to be reconstructed is divided into multiple grid units; a multi-layer perceptron of each grid unit is initialized to obtain multiple initial multi-layer perceptrons; the grid units and the multi-layer perceptrons correspond one to one; then, the rays of the camera corresponding to each pixel point on each color image are sampled respectively to obtain multiple sampling points of each ray, and the spatial parameters of each sampling point are determined; one grid unit corresponds to multiple pixels, and one pixel corresponds to a ray; the spatial parameters include: three-dimensional space coordinates and observation direction; finally, all initial multi-layer perceptrons are trained for multiple rounds using the depth values ​​of multiple groups of random pixel points on all scene depth maps and the spatial parameters of all sampling points on the rays corresponding to each random pixel point group to obtain a target multi-layer perceptron of each grid unit; the random pixel point group is two pixels randomly selected on the scene depth map; based on the scene representation of all sampling points output by each target perceptron, the scene to be reconstructed is three-dimensionally reconstructed. The neural radiation field in the related art only uses a multi-layer perceptron to extract scene information. Complex scenes cause the multi-layer perceptron to forget, which slows down the speed of scene 3D reconstruction. This application divides the scene to be reconstructed into multiple grid units, and sets a multi-layer perceptron for each grid unit, which improves the speed of scene 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a process flow of a scene 3D reconstruction method based on a neural radiation field provided in an embodiment of the present application;

[0050] Figure 2 Schematic diagram of the scene 3D reconstruction architecture based on neural radiation field;

[0051] Figure 3 This is a schematic diagram of the pre-trained deep model structure;

[0052] Figure 4 It is a schematic diagram of the structure of a multi-layer perceptron;

[0053] Figure 5 It is a global schematic diagram of a color image;

[0054] Figure 6 It is a local schematic diagram of a color image;

[0055] Figure 7 A global schematic diagram of the 3D reconstruction result obtained using a multi-layer perceptron;

[0056] Figure 8 A partial schematic diagram of the 3D reconstruction result obtained using a multi-layer perceptron;

[0057] Fig. 9 A global schematic diagram of a three-dimensional reconstruction result obtained using the method of the present application;

[0058] Fig.10 A partial schematic diagram of a three-dimensional reconstruction result obtained using the method of the present application;

[0059] Fig.11 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] The purpose of this application is to provide a method, device, medium and product for three-dimensional reconstruction of a scene based on a neural radiation field, aiming to increase the speed of three-dimensional reconstruction of the scene.

[0062] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0063] In an exemplary embodiment, Figure 1 and Figure 2 As shown, the scene 3D reconstruction method based on neural radiation field in this embodiment includes:

[0064] Step 1: Use a camera to capture multiple color images of the scene to be reconstructed.

[0065] Step 2: Input each color image into the pre-trained depth model to obtain the corresponding scene depth map.

[0066] Among them, the pre-trained deep model is obtained by training the deep network using the training set, and the deep network is built based on Transformer.

[0067] As an optional implementation, the training process of the pre-trained deep model includes:

[0068] Step 21: Obtain a training set; the training set includes: color images of multiple training scenes and corresponding actual scene depth maps.

[0069] Step 22: Build a deep network; the deep network includes: an embedding layer, multiple Transformers, multiple reorganization layers, multiple fusion modules and a detection head.

[0070] Step 23: Using the color image of each training scene as input and the corresponding actual scene depth map as output, the deep network is trained to obtain a pre-trained deep model.

[0071] Specifically, any color image is input into Figure 3 The process of generating a scene depth map in the pre-trained depth model shown includes:

[0072] First, the color image is input into the embedding layer (Embed), which converts the color image into an image tag (orange) and adds a tag (red) to represent the global information of the entire image.

[0073] Secondly, these image tags and tags representing the global information of the entire image are transformed through multiple Transformer layers into higher-level image feature tags.

[0074] Then, the higher-level image feature labels output from different Transformer layers are reassembled into image feature maps of different resolutions by the reassembly layers (Reassemble4, Reassemble8, Reassemble16, Reassemble32).

[0075] Thirdly, the fusion module gradually fuses and upsamples image feature maps from different resolutions to generate high-resolution feature maps.

[0076] Finally, the Head module converts the high-resolution feature map into a high-resolution scene depth map.

[0077] Step 3: Divide the scene to be reconstructed into multiple grid cells.

[0078] Specifically, let the minimum and maximum boundaries of the scene to be reconstructed be z n and z f , divide the scene to be reconstructed into pieces with a resolution of r = (r x ,r y ,r z ) uniform network unit. The three-dimensional coordinates of the i-th grid unit are X i , and corresponds to a small multilayer perceptron with a single set of parameters θ(i).

[0079] Step 4: Initialize the multilayer perceptron of each grid unit to obtain multiple initial multilayer perceptrons; the grid units and multilayer perceptrons correspond one to one.

[0080] Specifically, Figure 4 As shown in the figure, the multilayer perceptron includes 4 hidden layers, each with 64 channels. The input of the first hidden layer is the dimensionalized three-dimensional space coordinate x, the observation direction d and the volume density σ output by the third hidden layer are provided as input to the last hidden layer, and the last hidden layer outputs RGB color, which is expressed as:

[0081] (c,σ)=f θ(g(x)) (x,d).

[0082] Where c is the RGB color; σ is the volume density; f θ(g(x)) For MLP.

[0083] Step 5: Sample the camera's rays corresponding to each pixel point on each color image respectively, obtain multiple sampling points of each ray, and determine the spatial parameters of each sampling point; one grid unit corresponds to multiple pixels, and one pixel corresponds to one ray; the spatial parameters include: three-dimensional space coordinates and observation direction.

[0084] Specifically, the expression of the camera ray r(o, t, d) corresponding to any pixel point is:

[0085] r(o,t,d)=o+td.

[0086] Among them, o is the origin (that is, the position of the camera); t is the distance from the origin to any position on the ray along the direction of the ray.

[0087] Step 6: Use the depth values ​​of multiple groups of random pixel points on all scene depth maps and the spatial parameters of all sampling points on the rays corresponding to each random pixel point group to perform multiple rounds of training on all initial multi-layer perceptrons to obtain the target multi-layer perceptron for each grid unit; the random pixel point group is two pixels randomly selected on the scene depth map.

[0088] As an optional implementation, step 6 includes:

[0089] The training process of the multilayer perceptron of each grid unit in any current round includes:

[0090] Step 61: Determine any pixel point on any color image as the current pixel point, and determine the ray corresponding to the current pixel point as the current ray.

[0091] Step 62: Determine the updated multilayer perceptron of each grid unit in the previous round as the multilayer perceptron of each grid unit in the current round.

[0092] Step 63: Based on the spatial parameters of each sampling point on the current ray and the multi-layer perceptron of the current round of the grid unit corresponding to the current ray, determine the scene representation of each sampling point in the current round. The scene representation includes: RGB color and volume density.

[0093] As an optional implementation, step 63 includes:

[0094] Step 631: Perform dimension-upgrading encoding on the spatial parameters of each sampling point on the current ray to obtain the dimension-upgraded spatial parameters of each sampling point on the current ray.

[0095] Specifically, the expression of dimension-up coding is:

[0096] γ(p)=(sin(2 0 πp),cos(2 0 πp),···,sin(2 L-1 πp),cos(2 L-1 πp)).

[0097] Among them, γ(p) is the dimension-upgraded p; p is any one of the five-dimensional vectors in the three-dimensional space coordinate x and the observation direction d, x = (x, y, z), x is the x-axis component of the three-dimensional space coordinate x, y is the y-axis component of the three-dimensional space coordinate x, z is the z-axis component of the three-dimensional space coordinate x, θ is the pitch angle of the observation direction, is the horizontal angle of the observation direction; L is the dimension of γ(p).

[0098] Step 632: Input the dimension-upgraded spatial parameters of each sampling point on the current ray into the multi-layer perceptron of the current round of the grid unit corresponding to the current ray to obtain the scene representation of each sampling point in the current round.

[0099] Step 64: Determine the pixel color and depth value of the current ray in the current round based on the scene representation of all sampling points on the current ray in the current round.

[0100] As an optional implementation, step 64 includes:

[0101] Step 641: Calculate the pixel color of the current ray in the current round according to the RGB colors and volume density of all sampling points on the current ray in the current round.

[0102] Specifically, the calculation formula for the pixel color of any ray is:

[0103]

[0104] in, is the pixel color of ray r; M is the total number of sampling points on the ray; T i is the cumulative value of the i-th sampling point, σ j is the volume density of the jth sampling point, δ j is the distance difference between the j+1th sampling point and the jth sampling point, δ j =t j+1 -t j , t j+1 is the distance from the origin to the j+1th sampling point on the ray along the ray direction, t j is the distance from the origin to the jth sampling point on the ray along the ray direction; c i is the RGB color of the i-th sampling point.

[0105] Step 642: Calculate the depth value of the current ray in the current round according to the volume density of all sampling points on the current ray in the current round.

[0106] Specifically, the calculation formula for the depth value of any ray is:

[0107]

[0108] Among them, w i is the weight of the i-th sampling point, w i =T i (1-exp(-σ i δ i )),δ i is the distance difference between the i+1th sampling point and the ith sampling point, δ i =t i+1 -t i , t i+1 is the distance from the origin to the i+1th sampling point on the ray along the ray direction, t i It is the distance from the origin to the i-th sampling point on the ray along the ray direction.

[0109] Step 65: Determine the loss value in the current round based on the pixel colors of all pixels on all color images and the pixel colors of the corresponding rays in the current round, and the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of the corresponding rays in the current round.

[0110] As an optional implementation, step 65 includes:

[0111] Step 651: Calculate the color difference value in the current round based on the pixel colors of all pixels on all color images and the pixel colors of the corresponding rays in the current round.

[0112] Specifically, the calculation formula for the color difference value is:

[0113]

[0114] in, is the color difference value; R is the set of rays corresponding to all pixels on all color images; C(r) is the pixel color of the pixel on the color image corresponding to the ray r; ||·|| is the absolute value symbol.

[0115] Step 652: Calculate the depth constraint value in the current round based on the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of corresponding rays in the current round.

[0116] As an optional implementation, step 652 includes:

[0117] Step 6521: Determine any random pixel point group on any scene depth map as the current pixel point group.

[0118] Step 6522: Determine that the relationship between the depth values ​​of two pixels in the current pixel group on the scene depth map is a first relationship; the first relationship is greater than, less than or equal to.

[0119] Step 6523: Determine that the relationship between the depth values ​​of the rays corresponding to two pixel points in the current pixel point group in the current round is a second relationship; the second relationship is greater than, less than or equal to.

[0120] Step 6524: Calculate the depth constraint value in the current round based on the depth values ​​of the rays corresponding to all random pixel point groups for which the first relationship and the second relationship are inconsistent in the current round.

[0121] Specifically, the calculation formula for the depth constraint value is:

[0122]

[0123] Among them, R rank is the depth constraint value; N is the total number of scene depth maps; R n is the depth constraint component of the nth scene depth map; k 1,n is the first pixel in any random pixel group whose first relationship and second relationship are inconsistent in the nth scene depth map; k 2,n is the second pixel in any random pixel group whose first relationship and second relationship in the nth scene depth map are inconsistent; g n is the set of all random pixel groups whose first relationship and second relationship are inconsistent in the nth scene depth map; k 1,n The depth value of the corresponding ray; k 2,n The depth value of the corresponding ray; h is the allowable depth sorting error margin.

[0124] Step 653: Determine the loss value in the current round based on the color difference value and the depth constraint value in the current round.

[0125] Specifically, the loss value is calculated as follows:

[0126]

[0127] in, is the loss value; λ is the weight of the depth constraint value, λ=0.2.

[0128] Step 66: Determine whether the training stop condition is met; the training stop condition is that the current round reaches the preset number of training times or the loss value in the current round is less than the preset loss.

[0129] Step 67: If yes, the multilayer perceptron of each grid unit in the current round is determined as the target multilayer perceptron of each grid unit.

[0130] Step 68: If not, the multilayer perceptron of each grid unit in the current round is updated to obtain the updated multilayer perceptron of each grid unit in the current round, and the updated multilayer perceptron of each grid unit in the previous round is updated to the updated multilayer perceptron of each grid unit in the current round, and return to step 62 until the target multilayer perceptron of each grid unit is obtained.

[0131] Step 7: Based on the scene representation of all sampling points output by each target sensor, perform three-dimensional reconstruction of the scene to be reconstructed; the scene representation includes: RGB color and volume density.

[0132] Furthermore, the method of the present application was verified. Figure 5-Figure 10 As shown, a color image of a scene to be reconstructed, a three-dimensional reconstruction result obtained using a multi-layer perceptron, and a global and local graph of the three-dimensional reconstruction result obtained using the method of the present application are shown. The three-dimensional reconstruction result obtained by the method of the present application is better than the three-dimensional reconstruction result obtained using a multi-layer perceptron.

[0133] Advantages of this application:

[0134] (1) This application adopts scene decomposition technology to decompose the scene to be reconstructed into multiple grid units and set up unique MLPs for each unit. By adding scene decomposition technology and depth information constraints, the local detail extraction capability is improved and the rendering speed is accelerated.

[0135] (2) This application uses a depth information constrained rendering process to input the color image of the scene to be reconstructed into a pre-trained depth model to obtain a scene depth map. The local depth sorting method is used to make the depth value of the ray obtained based on the MLP consistent with the depth value output by the pre-trained depth model, thereby avoiding the problem of geometric radiation ambiguity and improving the quality of scene reconstruction.

[0136] In an exemplary embodiment, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for three-dimensional reconstruction of a scene based on a neural radiation field.

[0137] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a method for three-dimensional reconstruction of a scene based on a neural radiation field is implemented.

[0138] In an exemplary embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements a method for three-dimensional reconstruction of a scene based on a neural radiation field.

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a scene three-dimensional reconstruction method based on a neural radiation field is implemented.

[0140] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0142] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0143] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for scene 3D reconstruction based on neural radiation field, characterized in that: The method for scene 3D reconstruction based on neural radiation field includes: Using a camera to collect multiple color images of the scene to be reconstructed; Input each color image into a pre-trained depth model to obtain a corresponding scene depth map; the pre-trained depth model is obtained by training a deep network using a training set, and the deep network is built based on Transformer; Divide the scene to be reconstructed into multiple grid units; Initializing the multilayer perceptron of each of the grid units to obtain a plurality of initial multilayer perceptrons; the grid units and the multilayer perceptrons correspond one to one; The rays of the camera corresponding to each pixel point on each color image are sampled respectively to obtain multiple sampling points of each ray, and the spatial parameters of each sampling point are determined; one grid unit corresponds to multiple pixels, and one pixel corresponds to one ray; the spatial parameters include: three-dimensional space coordinates and observation direction; Using the depth values ​​of multiple groups of random pixel points on all scene depth maps and the spatial parameters of all sampling points on the rays corresponding to each random pixel point group, all initial multi-layer perceptrons are trained for multiple rounds to obtain a target multi-layer perceptron for each grid unit; the random pixel point group is two pixels randomly selected on the scene depth map; Based on the scene representation of all sampling points output by each target sensor, the scene to be reconstructed is three-dimensionally reconstructed; the scene representation includes: RGB color and volume density.

2. The method for scene 3D reconstruction based on neural radiation field according to claim 1, characterized in that: The training process of the pre-trained deep model includes: Acquire a training set; the training set includes: color images of multiple training scenes and corresponding actual scene depth maps; Constructing the deep network; the deep network includes: an embedding layer, multiple Transformers, multiple reorganization layers, multiple fusion modules and a detection head; The deep network is trained with the color image of each training scene as input and the corresponding actual scene depth map as output to obtain the pre-trained deep model.

3. The method for scene 3D reconstruction based on neural radiation field according to claim 1, characterized in that: Using the depth values ​​of multiple groups of random pixel points on all scene depth maps and the spatial parameters of all sampling points on the rays corresponding to each random pixel point group, all initial multi-layer perceptrons are trained for multiple rounds to obtain the target multi-layer perceptron of each grid unit, including: The training process of the multilayer perceptron of each grid unit in any current round includes: Determine any pixel point on any color image as the current pixel point, and determine the ray corresponding to the current pixel point as the current ray; Determine the updated multilayer perceptron of each grid unit in the previous round as the multilayer perceptron of each grid unit in the current round; Determine the scene representation of each sampling point in the current round based on the spatial parameters of each sampling point on the current ray and the multi-layer perceptron in the current round of the grid unit corresponding to the current ray; Determine the pixel color and depth value of the current ray in the current round based on the scene representation in the current round of all sampling points on the current ray; Determine the loss value in the current round based on the pixel colors of all pixels on all color images and the pixel colors of the corresponding rays in the current round, and the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of the corresponding rays in the current round; Determine whether the training stop condition is met; the training stop condition is that the current round reaches a preset number of training times or the loss value in the current round is less than a preset loss; If yes, the multilayer perceptron of each grid unit in the current round is determined as the target multilayer perceptron of each grid unit; If not, the multilayer perceptron of each grid unit in the current round is updated to obtain the updated multilayer perceptron of each grid unit in the current round, and the updated multilayer perceptron of each grid unit in the previous round is updated to the updated multilayer perceptron of each grid unit in the current round, and the process returns to "determine the updated multilayer perceptron of each grid unit in the previous round as the multilayer perceptron of each grid unit in the current round" until the target multilayer perceptron of each grid unit is obtained.

4. The method for scene 3D reconstruction based on neural radiation field according to claim 3, characterized in that: Based on the spatial parameters of each sampling point on the current ray and the multi-layer perceptron of the grid unit corresponding to the current ray in the current round, the scene representation of each sampling point in the current round is determined, including: Perform dimension-upgrading encoding on the spatial parameters of each sampling point on the current ray to obtain the dimension-upgraded spatial parameters of each sampling point on the current ray; The dimensionally upgraded spatial parameters of each sampling point on the current ray are respectively input into the multi-layer perceptron of the current round of the grid unit corresponding to the current ray to obtain the scene representation of each sampling point in the current round.

5. The method for scene 3D reconstruction based on neural radiation field according to claim 3, characterized in that: Based on the scene representation of all sampling points on the current ray in the current round, the pixel color and depth value of the current ray in the current round are determined, including: Calculate the pixel color of the current ray in the current round according to the RGB color and volume density of all sampling points on the current ray in the current round; According to the volume density of all sampling points on the current ray in the current round, the depth value of the current ray in the current round is calculated.

6. The method for scene 3D reconstruction based on neural radiation field according to claim 3, characterized in that: Based on the pixel colors of all pixels on all color images and the pixel colors of the corresponding rays in the current round, as well as the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of the corresponding rays in the current round, the loss value in the current round is determined, including: Based on the pixel colors of all pixels on all color images and the pixel colors of the corresponding rays in the current round, the color difference value in the current round is calculated; Calculate the depth constraint value in the current round based on the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of the corresponding rays in the current round; Based on the color difference value and the depth constraint value in the current round, the loss value in the current round is determined.

7. The method for scene 3D reconstruction based on neural radiation field according to claim 6, characterized in that: Based on the depth values ​​of multiple groups of random pixel points on all scene depth maps and the depth values ​​of the corresponding rays in the current round, the depth constraint value in the current round is calculated, including: Determine any random pixel point group on any scene depth map as the current pixel point group; Determine that a relationship between depth values ​​of two pixels in the current pixel group on the scene depth map is a first relationship; the first relationship is greater than, less than or equal to; Determine that the relationship between the depth values ​​of the rays corresponding to the two pixel points in the current pixel point group in the current round is a second relationship; the second relationship is greater than, less than or equal to; The depth constraint value in the current round is calculated based on the depth values ​​in the current round of the rays corresponding to all random pixel point groups for which the first relationship and the second relationship are inconsistent.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for three-dimensional reconstruction of a scene based on a neural radiation field as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for three-dimensional reconstruction of a scene based on a neural radiation field as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for three-dimensional reconstruction of a scene based on a neural radiation field as described in any one of claims 1 to 7 is implemented.

Images