Nerve radiation field three-dimensional reconstruction method based on space mixed attention
By introducing spatial mixed attention and linear gating structures into the neural radiation field method, the problems of error and boundary blur in three-dimensional scene reconstruction are solved, and the reconstruction quality and accuracy are significantly improved.
Patent Information
- Application Number
- CN202410946308.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
AI Technical Summary
The existing neural radiation field methods have errors in three-dimensional scene reconstruction, resulting in artifacts and boundary blurring, affecting the accuracy and sense of reality of the reconstruction.
Using the neural radiation field method based on spatial mixed attention, the correlation between color and density is enhanced through spatial vector mixing and linear gating structure, and the spatial mixed multi-layer perceptron and linear attention mechanism is used to optimize the body in the scene. Density and color information.
With the addition of a small number of network parameters, the quality of three-dimensional reconstruction and quantitative indicators are significantly improved, and the reconstruction accuracy and sense of reality are improved.
Smart Images

Figure CN118918250A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a three-dimensional reconstruction method of neural radiation field based on spatial mixed attention. Background Art
[0002] 3D scene reconstruction is an important problem in the field of computer vision and graphics, which aims to restore the geometry and appearance information of a 3D scene from a series of 2D images. This problem has wide applications in fields such as virtual reality, augmented reality, and 3D printing. However, since 2D images only contain limited perspective and lighting information, while 3D scenes have infinite complexity and diversity, 3D scene reconstruction has always been a very challenging task.
[0003] In recent years, deep learning methods have made remarkable progress in the field of three-dimensional scene reconstruction. Among them, neural radiation field, as a novel and effective method, can recover three-dimensional geometry and appearance information from two-dimensional images, but it still faces some limitations. One of them is that the model often makes errors when generating scene blocks, resulting in the appearance of bodies that do not exist in the actual scene. This error manifests itself as artifacts similar to cloud effects in the scene, which brings a certain sense of unreality to the reconstruction results. In addition, the neural radiation field model has challenges in dealing with object boundaries, resulting in obvious blurring effects on the boundaries, which in turn affects the accuracy and realism of the overall reconstruction. In order to solve the above problems, the present invention proposes a method based on spatial hybrid attention, which enhances and fuses the correlation between color and density in the scene through spatial vector mixing and linear gating structure to better capture its features and optimize the volume density and color information in the scene. It enables the reconstruction quality and quantitative indicators of the scene to be significantly improved with only a small increase in network parameters. Summary of the invention
[0004] The purpose of the present invention is to provide a neural radiation field three-dimensional reconstruction method based on spatial mixed attention, which can significantly improve the reconstruction quality and quantitative indicators of the scene with only a small amount of network parameters added.
[0005] To achieve the above object, the technical solution of the present invention is: a method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention, comprising the following steps:
[0006] Step S1, using a handheld camera to capture views of a real scene from various perspectives, and using an algorithm to calculate the spatial position and direction of the camera for the view set, and then arranging them into a data set for preprocessing;
[0007] Step S2: input each view and camera pose, and perform sampling in the scene, and then obtain the basic color and density information corresponding to each sampling point in the scene through a spatial hybrid multi-layer perceptron;
[0008] Step S3, input the obtained sampling point information into a gated structure with a linear attention mechanism, obtain the correlation between the color and density of each sampling point in the space by performing attention fusion on the color vector and the density vector, and combine it with the original color vector and the density vector to obtain a sampling point information feature vector containing the correlation between color and density;
[0009] Step S4: determine the area where the object exists in the scene through the density information of each sampling point in the space and perform three-dimensional scene modeling, then perform volume rendering on the generated three-dimensional volume through the color information of the sampling points, and finally reconstruct the complete three-dimensional scene.
[0010] In one embodiment of the present invention, the step S1 is specifically as follows:
[0011] Step S11, using a handheld camera to capture images from various perspectives in a real scene to ensure that different angles and fields of view are covered;
[0012] Step S12, calculating the spatial position and direction of the camera according to the motion trajectory of the camera and the posture information during shooting;
[0013] Step S13: sort and preprocess the calculated spatial position and direction of the camera, remove abnormal values and erroneous data, and convert the camera coordinate system into the world coordinate system.
[0014] In one embodiment of the present invention, step S2 is specifically as follows:
[0015] Step S21, input each view and the corresponding camera pose information into the neural radiation field, including the position coordinates Position and the viewing direction Direction of the camera in the world coordinate system;
[0016] Step S22, performing random sampling in the scene, taking a ray Ray in a random direction, taking a sampling point Sample on the ray Ray, and outputting the position information of all the sampling points;
[0017] Step S23: Input Position and Direction into the spatial hybrid multilayer perceptron MLP, and output the color RGB and density Density corresponding to the spatial position of each sampling point.
[0018] In one embodiment of the present invention, the spatial hybrid multi-layer perceptron MLP structure is specifically:
[0019] The 1×4-dimensional feature vector Feature that describes the color RGB and density information of the sampling point SampleAs the input of the spatial hybrid multilayer perceptron, each dimension represents R, G, B and density in RGB, Feature Sample After linear projection, it is expanded into an n×4-dimensional extended vector Feature expand_Sample , and Feature expand_Sample Transpose to get n-dimensional mixed feature Feature mix_Sample , and thus obtain n vectors with color RGB and density information; Feature mix_Sample Through a linear layer projection and transposition, the final sampling point attribute feature Feature is obtained raw_Sample ; The specific operation is expressed as:
[0020] Feature mix_Sample =Transpose(W n*1 *Feature Sample +b1)
[0021] Feature raw_Sample =Transpose(W 1*4 *Feature mix_Sample +b2)
[0022] Where W n *1 represents an n×1 dimensional matrix, W 1*4 Represents a 1×4 dimensional matrix, b1 and b2 represent the bias values of the corresponding projections, and Transpose represents the matrix transpose operation.
[0023] In one embodiment of the present invention, step S3 is specifically as follows:
[0024] Step S31: The color RGB and density information of each sampling point Sample, i.e., Feature raw_Sample Input to
[0025] In a gated structure with a linear attention mechanism;
[0026] Step S32: feature vector Feature raw_Sample Split into two vectors representing color and density, and then perform linear projection on these two vectors and fuse them to get Feature. mid_Sample ;
[0027] Step S33: For Feature mid_Sample , project it to the same raw_Sample The same dimension is used for residual connection to obtain the final sampling point information feature vector Feature containing the correlation between color and density output_Sample .
[0028] In one embodiment of the present invention, in step S32, firstly, the input 1×4 Feature raw_Sample The first three dimensions and the last dimension are split and transposed to obtain the vector Feature representing RGB and Density RGB_Sample With Feature Density_Sample ; Feature RGB_Sample With Feature Density_Sample Through linear projection and activation, we get two vector features of the same shape. expand_RGB With Feature expand_Density ; Then Feature expand_RGB With Feature expand_Density After a dot product operation, a vector feature containing the correlation between color and density is obtained. mid_Sample ; Finally, through a residual connection and linear projection layer and transposition; in step S33, Feature raw_Sample And the obtained Feature mid_Sample Connect to get the final sampling point information feature vector Feature output_Sample ; The specific operation is expressed as:
[0029] Feature RGB_Sample , Feature Density_Sample =split(Transpose(Feature raw_Sample ))
[0030] Feature expand_RGB =act(W m*3 *Feature RGB _ Sample +b3)
[0031] Feature expand_Density =act(W m*1 *Feature Density_Sample +b4)
[0032] Feature mid_Sample =W 4*m *(Feature expand_RGB ⊙Feature expand_Density )+b5
[0033] Feature output_Sample =Transpose(Feature mid_Sample )+Feature raw_Sample
[0034] Where split represents the segmentation operation, act represents the activation function, and the ReLU activation function is used here. m*3 Represents an m×3 dimensional matrix, W m*1 represents an m×1 dimensional matrix, W 4*m represents a 4×m-dimensional matrix, b3, b4, and b5 represent the bias values of the corresponding projections, and Transpose represents the matrix transpose operation.
[0035] In one embodiment of the present invention, step S4 is specifically as follows:
[0036] Step S41: The sampling point information feature vector Feature containing the color and density correlation obtained in step S3 is output_Sample Extract the density information corresponding to the sampling point, determine the area where the object exists in the scene, and generate the three-dimensional model Volume of the scene;
[0037] Step S42: From Feature output_Sample The color information corresponding to the sampling point is extracted, and the color information of the 3D model surface is mapped to the Volume through volume rendering;
[0038] Step S43: Given a viewing angle, output the view of the rendered 3D model at the corresponding viewing angle, and perform MSE loss with the input supervision image. After back propagation and gradient descent algorithms, optimize the color and density values of the sampling points in the whole process, and finally render a complete 3D model.
[0039] In one embodiment of the present invention, the volume rendering method is as follows:
[0040] Given the camera position coordinates Position and the viewing direction Direction, the color COLOR of the object surface at the corresponding position viewing angle is obtained through the input sampling point color RGB. The specific method is as follows:
[0041]
[0042] Where RGB i is the color of the i-th sampling point along the direction, Weight i is the weight of the color of the i-th sampling point along the direction, Density i Represents the density of the i-th sampling point along the direction, Density j Represents the density of the jth sampling point along the direction, Distance i Represents the distance between the i+1th and ith sampling points, Distance jRepresents the distance between the j+1th and jth sampling points, S represents the upper limit of the sampling points, which is 65536 here. Finally, on the surface of the Volume in the scene to be synthesized, a complete three-dimensional model is rendered by tracing the camera light.
[0043] The present invention also provides a three-dimensional reconstruction system of a neural radiation field based on spatial mixed attention, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0044] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0045] Compared with the prior art, the present invention has the following beneficial effects:
[0046] 1. Compared with the existing neural radiation field method, the present invention reconstructs the multi-layer perceptron structure of the neural radiation field while maintaining the inference speed as much as possible, improves the reconstruction accuracy of the method, and makes it more suitable for real scenes;
[0047] 2. The present invention innovatively uses a linear attention-based gating structure. Different from the traditional neural radiation field method, the present invention innovatively enhances and integrates the correlation between color and opacity in the scene, so that the multi-layer perceptron can more accurately learn the objects in the scene and optimize their volume density and color.
[0048] 3. Inspired by the Transformer model, the present invention redesigns the attention mechanism suitable for linear networks and combines it with a spatial hybrid multilayer perceptron, which can improve the three-dimensional reconstruction performance of the neural radiation field model without significantly increasing the number of parameters. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0050] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.
[0051] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0053] Please refer to Figure 1 The present invention provides a method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention, comprising the following steps:
[0054] Step S1, using a handheld camera to capture views of a real scene from various perspectives, and using an algorithm to calculate the spatial position and direction of the camera for the view set, and then arranging the data set into a preprocessing data set;
[0055] Step S2: input each view and camera pose, and perform sampling in the scene, and then obtain the basic color and density information corresponding to each sampling point in the scene through a spatial hybrid multi-layer perceptron;
[0056] Step S3, input the obtained sampling point information into a gated structure with a linear attention mechanism, obtain the correlation between the color and density of each sampling point in the space by performing attention fusion on the color vector and the density vector, and combine them with the original vector to obtain the color and density vector containing the associated attention information;
[0057] Step S4: determine the area where the object exists in the scene through the density information of each sampling point in the space and perform three-dimensional scene modeling, then perform volume rendering on the generated three-dimensional volume through the color information of the sampling points, and finally reconstruct the complete three-dimensional scene.
[0058] In this embodiment, the step S1 is specifically as follows:
[0059] Step S11, using a handheld camera to capture images from various perspectives in a real scene to ensure that different angles and fields of view are covered;
[0060] Step S12, calculating the position and direction of the camera in space according to the motion trajectory of the camera and the posture information during shooting;
[0061] Step S13: sort and preprocess the calculated camera position and direction data, remove abnormal values and erroneous data, and convert the camera coordinate system into the world coordinate system.
[0062] In this embodiment, the step S2 is specifically as follows:
[0063] Step S21, input each view and the corresponding camera pose information into the neural radiation field, including the position coordinates Position and the viewing direction Direction of the camera in the world coordinate system;
[0064] Step S22: Perform random sampling in the scene, take a ray Ray in a random direction, and take a sampling point on the ray Ray Sample , and output the location information of all sampling points;
[0065] Step S23: Input Position and Direction into the spatial hybrid multilayer perceptron MLP, and output the color RGB and density Density corresponding to the spatial position of each sampling point; the structure of the spatial hybrid multilayer perceptron is specifically as follows:
[0066] The 1×4-dimensional feature vector Feature that describes the color RGB and density information of the sampling point Sample (where each dimension represents R, G, B and density in RGB) as the input of the spatial hybrid multilayer perceptron, Feature Sample After linear projection, it is expanded into an n×4-dimensional extended vector Feature expand_Sample , and Feature expand_Sample Transpose to get n-dimensional mixed feature Feature mix_Sample , and thus obtain n vectors with color RGB and density information. mix_Sample By projecting through a linear layer and transposing each layer, the final sampling point attribute feature Feature is obtained. raw_Sample The specific operation can be expressed as:
[0067] Feature mix_Sample =Transpose(W n*1 *Feature Sample +b1)
[0068] Feature raw_Sample =Transpose(W 1*4 *FeatMre mix_Sample +b2)
[0069] Where W n *1 represents an n×1 dimensional matrix, W 1*4 represents a 1×4 dimensional matrix, b1 and b2 represent the bias values of its projection respectively, where n is 16 here, and Transpose represents the matrix transpose operation.
[0070] In this embodiment, step S3 is specifically as follows:
[0071] Step S31: Each sampling point Sample The color RGB and density information will be the feature vector Feature obtained from the above preliminary sampling point information raw_Sample Input into a gated structure with a linear attention mechanism;
[0072] Step S32: feature vector Feature raw_Sample Split into two vectors representing color and density, and then perform linear projection on these two vectors, and then fuse them through the linear attention gating unit to obtain the feature. mid_Sample ;
[0073] Step S33: For Feature mid_Sample , project it to the same raw_Sample The same dimension is used for residual connection to obtain the final sampling point information feature vector Feature containing the correlation between color and density output_Sample .
[0074] The gating structure is as follows:
[0075] First, input the 1×4 feature raw_Sample The first three dimensions and the last dimension are split and transposed to obtain the vector Feature representing RGB and Density RGB_Sample With Feature Density_Sample . RGB_Sample With Feature Density_Sample Through linear projection and activation, we get two vector features of the same shape. expand_RGB With Feature expand_Density Then, Feature expand_RGB With Feature expand_Density After a dot product operation, a vector feature containing the correlation between color and density is obtained. mid_Sample Finally, through a residual connection and linear projection layer and transposition, the feature input by the linear gating structure is raw_Sample And the obtained Feature mid_Sample Connect to get the final sampling point information feature vector Feature output_Sample The specific operation can be expressed as:
[0076] Feature RGB_Sample , Feature Density_Sample =split(Transpose(Feature raw_Sample ))
[0077] Feature expand_RGB =act(W m*3 *Feature RGB_Sample +b3)
[0078] Feature expand_Density =act(W m*1 *Feature Density_Sample +b4)
[0079] Feature mid_sample =W 4*m *(Feature expand_RGB ⊙Feature expand_Density )+b5
[0080] Feature output_Sample =Transpose(Feature mid_Sample )+Feature raw_Sample
[0081] Where split represents the segmentation operation, act represents the activation function, and the ReLU activation function is used here. m*3 Represents an m×3 dimensional matrix, W m*1 represents an m×1 dimensional matrix, W 4*m represents a 4×m-dimensional matrix, b3, b4, and b5 represent the bias values of their projections, respectively, where m is 256, and Transpose represents the matrix transpose operation.
[0082] In this embodiment, step S4 is specifically as follows:
[0083] Step S41: Use the Feature obtained from step S3 output_Sample Extract the density information corresponding to the sampling point, determine the area where the object exists in the scene, and generate the three-dimensional model Volume of the scene;
[0084] Step S42: Using the Feature obtained from step S3 output_Sample The color information corresponding to the sampling point is taken out, and the color information of the 3D model surface is mapped to the Volume through volume rendering; the specific method of volume rendering is:
[0085] Given the camera position coordinates Position and the viewing direction Direction, the color COLOR of the object surface at the viewing angle of the position can be obtained by inputting the sampling point color RGB. The specific method is as follows:
[0086]
[0087] Where RGB iis the color of the i-th sampling point along the direction, Weight i is the weight of the color of the i-th sampling point along the direction, Density i Represents the density of the i-th sampling point along the direction, Density j Represents the density of the jth sampling point along the direction, Dustance j Represents the distance between the i+1th and ith sampling points, Distance j It represents the distance between the j+1th and jth sampling points, and S represents the upper limit of the sampling points, which is 65536 here. Finally, on the surface of the Volume in the scene to be synthesized, a complete three-dimensional model can be rendered by tracing the camera light.
[0088] Step S43: Given a viewing angle, output the view of the rendered 3D model at that viewing angle, and perform MSE loss on the input supervision image. Through back propagation and gradient descent algorithms, the color and density values of the sampling points in the whole process are optimized, and finally a high-quality rendering result is obtained.
[0089] The present invention also provides a three-dimensional reconstruction system of a neural radiation field based on spatial mixed attention, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0090] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0091] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0092] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0093] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0095] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. A method for 3D reconstruction of neural radiation fields based on spatial mixed attention, characterized in that: The following steps are involved: Step S1, using a handheld camera to capture views of a real scene from various perspectives, and using an algorithm to calculate the spatial position and direction of the camera for the view set, and then arranging them into a data set for preprocessing; Step S2: input each view and camera pose, and perform sampling in the scene, and then obtain the basic color and density information corresponding to each sampling point in the scene through a spatial hybrid multi-layer perceptron; Step S3, input the obtained sampling point information into a gated structure with a linear attention mechanism, obtain the correlation between the color and density of each sampling point in the space by performing attention fusion on the color vector and the density vector, and combine it with the original color vector and the density vector to obtain a sampling point information feature vector containing the correlation between color and density; Step S4: determine the area where the object exists in the scene through the density information of each sampling point in the space and perform three-dimensional scene modeling, then perform volume rendering on the generated three-dimensional volume through the color information of the sampling points, and finally reconstruct the complete three-dimensional scene.
2. The method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention according to claim 1, characterized in that: The step S1 is specifically as follows: Step S11, using a handheld camera to capture images from various perspectives in a real scene to ensure that different angles and fields of view are covered; Step S12, calculating the spatial position and direction of the camera according to the motion trajectory of the camera and the posture information during shooting; Step S13: sort and preprocess the calculated spatial position and direction of the camera, remove abnormal values and erroneous data, and convert the camera coordinate system into the world coordinate system.
3. The method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention according to claim 1, characterized in that: The step S2 is specifically as follows: Step S21, input each view and the corresponding camera pose information into the neural radiation field, including the position coordinates Position and the viewing direction Direction of the camera in the world coordinate system; Step S22, performing random sampling in the scene, taking a ray Ray in a random direction, taking a sampling point Sample on the ray Ray, and outputting the position information of all the sampling points; Step S23: Input Position and Direction into the spatial hybrid multilayer perceptron MLP, and output the color RGB and density Density corresponding to the spatial position of each sampling point.
4. The method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention according to claim 3, characterized in that: The spatial hybrid multi-layer perceptron MLP structure is specifically as follows: The 1×4-dimensional feature vector Feature that describes the color RGB and density information of the sampling point Sample As the input of the spatial hybrid multilayer perceptron, each dimension represents R, G, B and density in RGB, Feature Sample After linear projection, it is expanded into an n×4-dimensional extended vector Feature expand_Sample , and Feature expand_Sample Transpose to get n-dimensional mixed feature Feature mix_Sample , and thus obtain n vectors with color RGB and density information; Feature mix_Sample Through a linear layer projection and transposition, the final sampling point attribute feature Feature is obtained raw_Sample ; The specific operation is expressed as: Feature mix_Sample =Transpose(W n*1 *Feature Sample +b1) Feature raw_Sample =Transpose(W 1*4 *Feature mix_Sample +b2) Where W n*1 represents an n×1 dimensional matrix, W 1*4 Represents a 1×4 dimensional matrix, b1 and b2 represent the bias values of the corresponding projections, and Transpose represents the matrix transpose operation.
5. The method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention according to claim 4, characterized in that: The step S3 is specifically as follows: Step S31: The color RGB and density information of each sampling point Sample, i.e., Feature raw_Sample Input into a gated structure with a linear attention mechanism; Step S32: feature vector Feature raw_Sample Split into two vectors representing color and density, and then perform linear projection on these two vectors and fuse them to get Feature. mid_Sample ; Step S33: For Feature mid_Sample , project it to the same raw_Sample The same dimension is used for residual connection to obtain the final sampling point information feature vector Feature containing the correlation between color and density output_Sample .
6. The method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention according to claim 5, characterized in that: In step S32, first input 1×4 Feature raw_Sample The first three dimensions and the last dimension are split and transposed to obtain the vector Feature representing RGB and Density RGB_Sample With Feature Density_Sample ; Feature RGB_Sample With Feature Density_Sample Through linear projection and activation, we get two vector features of the same shape. expand_RGB With Feature expand_Density ; Then Feature expand_RGB With Feature expand_Density After a dot product operation, a vector feature containing the correlation between color and density is obtained. mid_Sample ; Finally, through a residual connection and linear projection layer and transposition; in step S33, Feature raw_Sample And the obtained Feature mid_Sample Connect to get the final sampling point information feature vector Feature output_Sample ; The specific operation is expressed as: Feature RGB_Sample ,Feature Density_Sample =split(Transpose(Feature raw_Sample )) Feature expand_RGB =act(W m*3 *Feature RGB_Sample +b3) Feature expand_Density =act(W m*1 *Feature Density_Sample +b4) Feature mid_Sample =W 4*m *(Feature expand_RGB ⊙Feature expand_Density )+b5 Feature output_Sample =Transpose(Feature mid_Sample) +Feature raw_Sample Where split represents the segmentation operation, act represents the activation function, and the ReLU activation function is used here. m*3 Represents an m×3 dimensional matrix, W m*1 represents an m×1 dimensional matrix, W4*m represents a 4×m dimensional matrix, b3, b4, b5 represent the bias values of the corresponding projections, and Transpose represents the matrix transpose operation.
7. The method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention according to claim 1, characterized in that: The step S4 is specifically as follows: Step S41: The sampling point information feature vector Feature containing the color and density correlation obtained in step S3 is output_Sample Extract the density information corresponding to the sampling point, determine the area where the object exists in the scene, and generate the three-dimensional model Volume of the scene; Step S42: From Feature output_Sample The color information corresponding to the sampling point is extracted, and the color information of the 3D model surface is mapped to the Volume through volume rendering; Step S43: Given a viewing angle, output the view of the rendered 3D model at the corresponding viewing angle, and perform MSE loss with the input supervision image. After back propagation and gradient descent algorithms, optimize the color and density values of the sampling points in the whole process, and finally render a complete 3D model.
8. The method for three-dimensional reconstruction of neural radiation field based on spatial mixed attention according to claim 7, characterized in that: The specific method of volume rendering is: Given the camera position coordinates Position and the viewing direction Direction, the color COLOR of the object surface at the corresponding position viewing angle is obtained through the input sampling point color RGB. The specific method is as follows: Where RGB i is the color of the i-th sampling point along the direction, Weight i is the weight of the color of the i-th sampling point along the direction, Density i Represents the density of the i-th sampling point along the direction, Density j Represents the density of the jth sampling point along the direction, Distance i Represents the distance between the i+1th and ith sampling points, Distance j Represents the distance between the j+1th and jth sampling points, S represents the upper limit of the sampling points, which is 65536 here. Finally, on the surface of the Volume in the scene to be synthesized, a complete three-dimensional model is rendered by tracing the camera light.
9. A neural radiation field 3D reconstruction system based on spatial mixed attention, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.
10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.
Citation Information
Cited By
Sparse view outdoor scene three-dimensional reconstruction method based on multi-view stereo and grid
CN120259559A
3D reconstruction of outdoor scenes with sparse views based on multi-view stereo and mesh
CN120259559B
