A deep learning-based three-dimensional reconstruction method and system

Abstract

The application discloses a kind of three-dimensional reconstruction method and system based on deep learning, comprising: obtaining scene image data;Build improved neural radiance field three-dimensional reconstruction model;The improved neural radiance field three-dimensional reconstruction model is trained, and reconstruction result is obtained;Finally, the reconstruction result is evaluated, and the reconstruction result is comprehensively evaluated by performance index.It relieves the feature information loss caused by the conversion from position coding to fixed-length vector;Relieve local confusion to improve the ability of model fitting coarse sampling features, correct the wrong part in three-dimensional features in the coarse sampling stage;Overcome the spectral bias of neural radiance field;It can be applied to three-dimensional scene live rendering of mobile robot in complex environment, only monocular camera can be used to reconstruct three-dimensional scene, and can help mobile robot to perceive more spatial feature information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of view reconstruction using deep learning, in particular to a three-dimensional reconstruction method and system based on deep learning. BACKGROUND

[0002] Inferring three-dimensional scene representation from two-dimensional observations is a fundamental problem in computer graphics and computer vision. Novel view synthesis aims to render a view of a scene from an unknown viewpoint given a set of existing multi-view images and camera poses as input, and is widely used in VR / AR, medical imaging, robot navigation, smart city and other fields and industries, and is the basis for realizing tasks in complex environments. Although three-dimensional scanners based on structured light principles can achieve high-precision three-dimensional reconstruction, such devices are expensive and cumbersome to operate, so obtaining high-resolution reconstructed views through algorithms is of great significance to the development of three-dimensional reconstruction.

[0003] Traditional computer graphics generates high-quality reconstructed views through modeling geometric surface properties and machine learning methods from a physical perspective, but the reconstruction quality depends on the physical correctness of the constructed model and the quality of the training data. In recent years, people have integrated physics-based multi-view geometry techniques into deep learning-based methods for three-dimensional scene reconstruction tasks. Among them, explicit methods include voxel-based, point cloud-based, and curved mesh-based. Although these methods can effectively represent the three-dimensional characteristics of objects, they usually only densely sample and reconstruct local areas, and in addition to the need for a large number of image inputs, the way of querying 3D geometry priors also increases the memory requirements for view reconstruction. In comparison, view reconstruction methods that rely on implicit neural representations have broad application prospects due to their small scene representation storage and independence from view resolution. Neural Radiance Fields is a new view reconstruction technique based on implicit scene representation, which uses a coarse-to-fine sampling strategy to sample multi-view scene features through a multi-layer perceptron, and integrates differentiable rendering into network training, thereby achieving high-quality view reconstruction. A key step in this model is to learn the complex mapping from the captured view to the reconstructed view using a multi-layer perceptron, but the simple linear fully connected layer sampling method of neural radiance fields can cause the loss of local information, resulting in blurred and aliased reconstructed views. Some research methods express local details by selecting appropriate network architectures, but there is still little research on multi-scale feature information extraction and fusion in network training, and the internal dependence between global information and local information has not been combined, making it difficult to solve the problem of feature sparsity or loss when local details appear less frequently. At the same time, some three-dimensional scenes contain large empty spaces or disordered background spaces, and directly using MLPs for color and density inference can cause ambiguous encoding features to adversely affect the reconstruction results. SUMMARY

[0004] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification of the present application in order to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions are not used to limit the scope of the present application.

[0005] In view of the above problems, the present application is proposed.

[0006] Therefore, the technical problem solved by the present application is that the existing three-dimensional construction method has not yet combined the internal dependence between global information and local information, it is difficult to solve the problem of feature sparsity or absence when the frequency of local details is low, and how to solve the optimization problem that directly using MLP for color and density inference will cause ambiguous coding features to adversely affect the reconstruction result.

[0007] To solve the above technical problems, the present application provides the following technical solutions: a three-dimensional reconstruction method based on deep learning, comprising:

[0008] Obtaining scene image data;

[0009] Building an improved neural radiance field three-dimensional reconstruction model;

[0010] Training the improved neural radiance field three-dimensional reconstruction model to obtain a reconstruction result;

[0011] Finally, the reconstruction result is evaluated, and the reconstruction result is comprehensively evaluated by performance indicators.

[0012] As the three-dimensional reconstruction method based on deep learning described in the present application, characterized in that: the structure of the improved neural radiance field three-dimensional reconstruction model includes: a sine-cosine position encoding network, a feature extraction network and a volume rendering network.

[0013] As the three-dimensional reconstruction method based on deep learning described in the present application, characterized in that: the feature extraction network includes: a coarse sampling network and a fine sampling network, both of which contain two MLP models, one is a 5-layer fully connected sampling MLP model, and the other is a 3-layer fully connected inference MLP model.

[0014] As the three-dimensional reconstruction method based on deep learning described in the present application, characterized in that: the improved neural radiance field three-dimensional reconstruction model includes: the neural radiance field uses position encoding γ() to map the three-dimensional position X of each pixel point in the observation image and the pose d of the corresponding camera to a high-dimensional space before sampling, so that the light sample carrying image information is converted to high-dimensional sine and cosine signals in the Fourier space:

[0015] γ(p)=(sin(20 sin(2 0 πp), cos(2 L-1 πp), sin(2 L-1 πp)

[0016] where p is the parametric representation of the three-dimensional position X and pose d, L represents the dimension of the position encoding, and L=10 is set for the three-dimensional position X in the neural radiance field, and L=4 is set for the position encoding γ(d) of the camera pose d.

[0017] The three-dimensional reconstruction method based on deep learning according to the application has the characteristics that:

[0018] In the coarse sampling network, a multi-feature joint learning module is designed.

[0019] The input of the network is the X position encoding γ(X) of the three-dimensional coordinates, one branch first divides each feature map with a size of CxHxW into n equal parts, then uses DCT filtering and completes the frequency components not extracted in the feature spectrum, focuses on the relatively important local information in the image, and finally obtains higher frequency domain feature aggregation by embedding more frequency component information in the network; the MLP bottleneck structure is constructed as a feature decoder by using the strategy of dimension reduction first and then dimension increase, and ReLU is used as the activation function; the joint learning feature γ'(X) is obtained by calculating the energy function en r of each neuron:

[0020]

[0021] In the formula, γ(X) and Norm(γ(X)) represent the original feature map, Sigmoid() represents the neuron attention weight calculation, DCT() represents the local feature filtering, and nMLP() represents the global feature sampling.

[0022] In the fine sampling network, the feature vector with an output resolution of Nx256 is first adjusted in a unified range through layer normalization and reweighting; the feature vector dimension is expanded to twice the original, and the output resolution is Nx512; then the nonlinearity is increased through the Relu activation function; finally, the feature vector dimension is restored, the output resolution is Nx256, and the errors in the three-dimensional features in the coarse sampling stage are corrected.

[0023] The three-dimensional reconstruction method based on deep learning according to the application has the characteristics that: the improved neural radiance field three-dimensional reconstruction model further comprises: in the feature inference stage, a gated channel transformation multilayer perception module is designed; an l2-norm normalization method is used to establish the competition or cooperation relationship between channels, and the feature information is selectively transmitted through the gating mechanism F to realize the reweighting of the features in the channel dimension.

[0024] When the gating weight of a channel is positively activated, the GCT will promote the features of this channel and the features of other channels to compete;

[0025] When the gating weight of a channel is negatively activated, the GCT will promote the features of this channel and the features of other channels to cooperate, and finally output the feature screening result

[0026]

[0027] where g represents the input features of the gating unit, are three trainable parameters, the embedding weight α adjusts the encoding output by eliminating local ambiguity, and the gating weight γ and the gating bias β control the activation state of the channel features.

[0028] As the three-dimensional reconstruction method based on deep learning provided by the application, the reconstruction result is obtained by using volume rendering technology to reconstruct a new view according to a test set image not participating in training, and the qualitative comparison result of the reconstructed view and the original view can be used as an auxiliary evaluation standard for model performance.

[0029] As the three-dimensional reconstruction method based on deep learning provided by the application, the reconstruction result is obtained by using volume rendering technology to reconstruct a new view according to a test set image not participating in training, and the qualitative comparison result of the reconstructed view and the original view can be used as an auxiliary evaluation standard for model performance.

[0030] The evaluation of the reconstruction result includes comprehensive evaluation of the reconstruction result by three performance indicators of peak signal-to-noise ratio, structural similarity index and learning perception image block similarity:

[0031]

[0032]

[0033]

[0034]

[0035] where MSE represents the mean square error of the original image I and the comparison image K with a given size of m x n, i and j jointly represent the image pixel points, MAX I represents the maximum pixel value of the image, SSIM is an index for measuring the similarity of two images, x represents the original image, y represents the comparison image, μ represents the average value, σ 2 represents the variance, σ xy represents the covariance of x and y, c1=(k1L) 2 , c2=(k2L) 2is a constant used to maintain stability, L is a dynamic range of pixel values, k1=0.01, k2=0.03, the value range of the SSIM function is [-1, 1], d is the distance between x0 and x, x represents the original image, and x0 represents the contrast image; x and x0 are sent into the neural network for feature extraction, and the output of each layer is activated and normalized, and is recorded as The vector ω l is used to scale the number of activation channels, and finally the l2 distance is calculated, l represents the network level, H l , W l , and C l represent the size of the layer network, respectively representing height, width and channel number.

[0036] A three-dimensional reconstruction system based on deep learning, characterized in that it comprises:

[0037] An image data acquisition module for acquiring multi-view scene images;

[0038] An image data preprocessing module for feature extraction, feature matching and sparse reconstruction of image data, and exporting corresponding camera poses, converting camera parameters into pose constraints;

[0039] An improved neural radiance field three-dimensional reconstruction network module for reducing the interference of ambiguous feature coding on view reconstruction, and obtaining multiple reconstruction views by combining a hierarchical sampling strategy;

[0040] A joint optimization module for jointly optimizing the neural radiance field to obtain high-precision three-dimensional reconstruction views;

[0041] A reconstruction performance verification module for evaluating the quality of the reconstruction views, seeking super parameter configurations that meet the scene, and further optimizing the view reconstruction results.

[0042] As the three-dimensional reconstruction system based on deep learning described in the application, characterized in that:

[0043] The image data acquisition module sequentially arranges the image data and sends it to the data preprocessing module;

[0044] The processed parameter information is transmitted from the data preprocessing module to the improved neural radiance field three-dimensional reconstruction network module to obtain multiple reconstruction views;

[0045] The joint optimization module matches the reconstruction view generated in the last step with the pose constraint information extracted in the preprocessing stage, and jointly optimizes the neural radiance field, and transmits it to the reconstruction performance verification module;

[0046] The reconstruction performance verification module evaluates the quality of the reconstructed view by using the peak signal-to-noise ratio, the structural similarity index and the learning perception image block similarity value, and sets a reinforcement iteration strategy to seek a scene-compliant hyperparameter configuration, thereby optimizing the view reconstruction result.

[0047] The present application provides a neural radiance field three-dimensional reconstruction method and system based on improved MLP. In the coarse sampling stage, a multi-feature joint learning module is designed, which decodes spatial global features and frequency domain local features through normalized MLP bottleneck structure and discrete cosine transform, and utilizes a non-parametric attention to fuse the two features, providing the network with more rich multi-scale feature information and relieving the feature information loss caused by the conversion from position encoding to fixed-length vector. In the fine sampling stage, a standardized intermediate layer with an increased network receptive field is designed for feature extraction, relieving local confusion and improving the model's ability to fit coarse sampling features and correcting the errors in the three-dimensional features in the coarse sampling stage. In the feature reasoning process, the module learns high-order feature interaction, optimizes the gating unit to improve the network's ability to screen high-weight features in the pixel area, and overcomes the spectral bias of the neural radiance field. It can be applied to three-dimensional scene real scene rendering of mobile robots in complex environments, and only a monocular camera is needed to reconstruct a three-dimensional scene, helping mobile robots to perceive more spatial feature information. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor. Among them:

[0049] Figure 1 The overall flowchart of a three-dimensional reconstruction method based on deep learning provided for the first embodiment of the present application;

[0050] Figure 2 The NeRF network structure diagram in a three-dimensional reconstruction method based on deep learning provided for the second embodiment of the present application;

[0051] Figure 3 The network structure diagram of the neural radiance field three-dimensional reconstruction method based on improved MLP in a three-dimensional reconstruction method based on deep learning provided for the second embodiment of the present application;

[0052] Figure 4 The multi-feature joint learning module diagram in a three-dimensional reconstruction method based on deep learning provided for the second embodiment of the present application;

[0053] Figure 5A schematic diagram of a gated channel transformation multilayer perceptron module in a deep learning-based 3D reconstruction method provided in the second embodiment of the present invention;

[0054] Figure 6 A schematic diagram of a neural radiation field three-dimensional reconstruction system based on an improved MLP, which is a deep learning-based three-dimensional reconstruction method provided for the second embodiment of the present invention;

[0055] Figure 7 The second embodiment of the present invention provides a comparison of the visualization results of new view reconstruction of partial scenes using different methods on three datasets using a deep learning-based 3D reconstruction method. Detailed Implementation

[0056] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0057] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0058] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0059] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0060] Meanwhile, in the description of the present application, it should be noted that the terms "upper, lower, inner and outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first, second or third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0061] Unless otherwise expressly specified and limited, the terms "mounting, connecting, connecting" in the present application should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; It can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0062] Embodiment 1

[0063] Reference Figure 1 For an embodiment of the present application, a three-dimensional reconstruction method and system based on deep learning are provided, comprising:

[0064] S1: Obtain scene image data.

[0065] Experiments are conducted using three public data sets, namely Realistic Synthetic 360° benchmark data set, RealForward-Facing benchmark data set and DTU benchmark data set. Realistic Synthetic 360° is a real rendering 360° synthetic data set, including 8 scenes, 100 training views, 100 validation views and 200 test views, with a resolution of 800x800. Real Forward-Facing is a real scene data set taken by a handheld camera, containing 8 real scenes, each scene has 20 to 62 views with a resolution of 1008x756. The DTU data set is an indoor object data set taken by an industrial robot arm equipped with adjustable brightness lamps, containing 128 scenes.

[0066] S2: Build an improved neural radiance field three-dimensional reconstruction model.

[0067] First, improvements are made on the basis of the original neural radiance field network:

[0068] The neural radiance field structure is divided into a sine position encoding network, a feature extraction network and a volume rendering network.

[0069] It should be noted that the feature extraction network is divided into two parts, including a coarse sampling network and a fine sampling network. In the original NeRF, the two structures are the same, both containing two MLP models, one is an 8-layer fully connected sampling MLP model, and the other is a 3-layer fully connected inference MLP model. In order to reduce the parameter quantity, the improved neural radiation field changes the two MLP models to: one is a 5-layer fully connected sampling MLP model, and the other is a 3-layer fully connected inference MLP model. The embedding of the subsequent module is carried out on the basis of this simplified network.

[0070] The neural radiation field receives a set of static pictures of an object or scene under the same lighting conditions, and the network input is a continuous 5-dimensional coordinate composed of a three-dimensional position X = (x, y, z) and a two-dimensional viewing direction based on spherical coordinates The network input is a continuous 5-dimensional coordinate composed of a three-dimensional position X = (x, y, z) and a two-dimensional viewing direction based on spherical coordinates θ The color c and the volume density σ of the directional emitted light ray along the sampling point are extracted and inferred at the corresponding pixel position and the view:

[0071] The network input is a continuous 5-dimensional coordinate composed of a three-dimensional position X = (x, y, z) and a two-dimensional viewing direction based on spherical coordinates θ :(X,d)→(c,σ)

[0072] It should be noted that camera imaging can be regarded as light rays from pixel points passing through objects and calculating their color values to render the observed image. The neural radiation field integrates the camera light into the pixel, independently calculates the color of each sampling point along the light ray using the volume rendering equation, and finally reconstructs the color C(r) of the pixel in the image:

[0073]

[0074] Where: r(t) = o + td represents a light ray emitted from the camera source point o along the two-dimensional viewing direction d, t represents the sampling length, and T(t) represents the cumulative transmittance from 0 to t:

[0075]

[0076] In the formula: s represents the length of the light ray.

[0077] In order to make the MLP better fit the high-frequency information to compensate for the spectral deviation of the network, the NeRF (Neural Radiance Field) uses position encoding γ() to map the three-dimensional position X of each pixel point in the observed image and the pose d of the corresponding camera to a high-dimensional space before sampling, so that the light ray sample carrying image information is converted to high-dimensional sine and cosine signals in the Fourier space:

[0078] γ(p) = (sin(2 0 πp), cos(2 0 πp), …, sin(2 L-1 πp), cos(2 L-1 πp))

[0079] where p is the parametric representation of the 3D position X and pose d, L is the dimension of the positional encoding, L = 10 for the 3D position X and L = 4 for the camera pose d in the neural radiance field.

[0080] It should also be known that in order to improve the sampling efficiency, the neural radiance field uses a coarse-fine double-layer sampling strategy to input the high-dimensional signal into two feature extraction networks in turn. The coarse representation network uniformly samples N c sample points to quantize the light forming heuristic shape, and the output can be regarded as the probability density distribution of the effective sampling values in space; the fine optimization network resamples on this basis, uses inverse transform sampling to extract N f sample points related to the volume from the probability density distribution, and merges and reorders them with the uniformly sampled sample points, using N c +N f sample points to quantize the light to generate an implicit shape representation, and then perform a cumulative rendering step to generate the final RGB output.

[0081] As can be seen from the above description, the tasks of the two feature extraction networks are not the same, but in the neural radiance field, the structures of the two are the same, which affects the effect of new view reconstruction. Therefore, the structures of the two feature extraction networks are optimized respectively in this paper, and an improved MLP-based neural radiance field, IP-NeRF, is constructed.

[0082] In order to reduce the parameter amount of the model, in the two feature extraction networks, the original fully connected layer model with a depth of 8 and a channel size of 256 is replaced by a fully connected layer model with a depth of 5 and a channel size of 256.

[0083] In the coarse sampling network, a multi-feature joint learning module is designed, which is composed of two feature extraction branches and a parameter-free attention mechanism. The two feature extraction branches are discrete cosine transform (DCT) and normalized MLP bottleneck structure respectively.

[0084] Further, the input of the network is the X position encoding of the three-dimensional coordinates γ(X), wherein one branch first divides each feature map with a size of CxHxW into n equal parts, then uses DCT filtering and fills in the frequency components not extracted in the feature spectrum, focuses on the relatively important local information in the image, and finally obtains higher frequency domain feature aggregation by embedding more frequency component information in the network. Because the scale change position encoding of the neural radiation field is not conducive to the sampling of global features, the input of the other branch is the feature vector in the unified range after γ(X) is reweighted and adjusted after layer normalization. First, each feature map with a size of CxHxW is compressed into a scalar in the channel dimension through global average pooling to capture the global dependence at the channel level. Then, a "dimension reduction first, then dimension increase" strategy is used to construct an MLP bottleneck structure as a feature decoder, with ReLU as the activation function. In addition, based on the parameter-free attention, the local features and global features are fused in the channel dimension, and then the energy function en r , with the sum of the minimum neuron energies is taken as the optimization target to generate the corresponding neuron attention weight, and the weighted sum of the original feature map of the corresponding branch is obtained to obtain the joint learning feature γ'(X):

[0085]

[0086] In the formula, γ(X) and Norm(γ(X)) represent the original feature map, Sigmoid() represents the neuron attention weight calculation, DCT() represents the local feature filtering, and nMLP() represents the global feature sampling.

[0087] In the fine sampling network, the second layer of the fully connected layer model is replaced by a standardized intermediate layer module, which is characterized in that: first, the feature vector with an output resolution of Nx256 output by the first layer is reweighted and adjusted in a unified range after layer normalization; then, the feature vector is expanded to twice the original dimension through an intermediate layer with "dimension increase first, then dimension decrease", which includes two hidden layers and a ReLU activation function. In this process, the feature vector dimension is first expanded to twice the original dimension, with an output resolution of Nx512, then the ReLU activation function is used to increase the nonlinearity, and finally the feature vector dimension is restored, with an output resolution of Nx256. By increasing the number of neurons, the fitting ability of the network to the coarse sampling features is enhanced, the local confusion is alleviated, and the ability of the model to fit the coarse sampling features is improved, and the incorrect parts in the three-dimensional features in the coarse sampling stage are corrected.

[0088] Then, in the feature inference stage, a gated channel transformation multi-layer perception module is designed, characterized in that, first, a gated MLP module is added between the sampling layer and the inference layer of the neural radiance field to enhance the spatial screening ability of the ambiguous encoding features. To make the module more suitable for the high-dimensional features of the neural radiance field after Fourier transform, the gating unit of the gMLP is replaced by a gated channel transformation, which can effectively suppress the interference of ambiguous encoding features in the inference process, thereby improving the inference performance.

[0089] Further, the gating unit is a gating matrix composed of feature screening layers, and the passing rate of the features is controlled through matrix point multiplication, thereby controlling the parameter quantity. The GCT adopts an l2-norm normalization method to establish a competitive or cooperative relationship between channels, and selectively transmits feature information through the gating mechanism F to realize the reweighting of features in the channel dimension. When the gating weight of a channel is positively activated, the GCT will promote the "competition" between the features of this channel and the features of other channels; when the gating weight of a channel is negatively activated, the GCT will promote the "cooperation" between the features of this channel and the features of other channels, and finally output the feature screening result

[0090]

[0091] In the formula, g represents the input features of the gating unit, are three trainable parameters, the embedding weight α adjusts the encoding output by eliminating local ambiguity, and the gating weight γ and the gating bias β control the activation state of the channel features.

[0092] S3: training the improved neural radiance field three-dimensional reconstruction model. The model is deployed using the PyTorch framework, the PyCharm compiler, and the Python 3.8 language. A hierarchical sampling strategy is used for training, the number of uniformly sampled points of the coarse representation network is 64, the number of non-uniformly sampled points of the fine optimization network is 128, the number of sampled rays per batch is 1024, and the number of single sampling is 32768. The model is trained using the MSE photometric loss, with an initial value of 5x10 -4 and exponentially decaying to 5x10 -5 , a learning rate of 5x10 -7 on the Adam optimizer β1=0.9, β2=0.999, and ε=10 -7, and 200,000 iterations per scene. 1 / 8 of the image data is used as the test set, and the rest is used as the training set.

[0093] The volume rendering technique is used to reconstruct new views according to the test set images that do not participate in training. The qualitative comparison results of the reconstructed views and the original views can be used as an auxiliary evaluation standard for the performance of the model.

[0094] S4: evaluate the reconstruction results.

[0095] The reconstruction results are comprehensively evaluated by three performance indicators of peak signal-to-noise ratio, structural similarity index and learning perceptual image block similarity. The definitions are as follows:

[0096]

[0097]

[0098]

[0099] In the formula, MSE represents the mean square error, and for a given original image I and a contrast image K of size m*n, the mean square error formula is as follows:

[0100]

[0101] i, j jointly represent image pixels. As indicated in the summation symbol, for a given original image I and a contrast image K, the size of both is m*n, and the value of i, j is from 0 to m-1 (n-1), corresponding to the query of each pixel.

[0102] Further, MAX I represents the maximum pixel value of the image. Generally, for uint8 data, the maximum pixel value is 255, and for floating-point data, the maximum pixel value is 1. The unit of PSNR is dB, and the larger the PSNR value, the better the quality of the image.

[0103] Further, SSIM is an index for measuring the similarity between two images, x represents the original image, y represents the contrast image, μ represents the average value, σ 2 represents the variance, and σ xy represents the covariance of x and y. c1=(k1L) 2 , c2=(k2L) 2 are constants used to maintain stability. L is the dynamic range of pixel values, k1=0.01, k2=0.03. The value range of the SSIM function is [-1, 1], and the larger the value, the smaller the image distortion and the more similar the two images.

[0104] Further, LPIPS is used to measure the difference between two images, and the value is more consistent with human perception. Wherein, x represents the original image, x0 represents the contrast image, and d is the distance between x0 and x. x and x0 are input into a neural network (such as VGG) for feature extraction, and the output of each layer is activated and normalized, denoted as The vector ω l is used to scale the number of activation channels, and the final l2 distance is calculated. l represents the network level, corresponding to the summation at the corresponding level of the network, represents the summation of H l , Wl , C l denote the dimensions of the layer network, respectively height, width and number of channels, denote the sum over the channel dimension.

[0105] The lower the value of the neural network LPIPS, the more similar the two images are, and vice versa, the greater the difference.

[0106] It should be noted that the present application simplifies the backbone feature extraction network on the basis of neural radiance field, introduces a standardized intermediate layer module to enhance the network's fitting ability for coarse sampling features; designs a multi-feature joint learning module, uses discrete cosine transform and normalized MLP bottleneck structure to sample local and global features respectively, and aggregates the sampled features through a parameter-free attention mechanism, so that the MLP model of the neural radiance field can capture global features while obtaining more local dependencies, realizing multi-scale feature joint learning; an improved gated channel transformation multilayer perceptron module is used to enhance the model's spatial filtering ability for ambiguous encoding features, so that the MLP model of the neural radiance field focuses on learning the game state of weight information and obtains more view-dependent feature information for color and density reasoning.

[0107] The present disclosure provides an improved MLP-based neural radiance field three-dimensional reconstruction system, which comprises:

[0108] An image data acquisition module is used to acquire multi-view scene images and sequentially arrange the image data into a data preprocessing module;

[0109] An image data preprocessing module is used to extract features, match features and perform sparse reconstruction on the image data using COLMAP, and export the corresponding camera pose; at the same time, the camera parameters are converted into pose constraints using the integrated local light field fusion project pose conversion program.

[0110] An improved neural radiance field three-dimensional reconstruction network module is used to alleviate the view rendering blur and reconstruction degradation problem through a multi-feature joint learning method, reduce the interference of ambiguous feature encoding on view reconstruction through a gating filtering mechanism, and obtain multiple reconstruction views through a hierarchical sampling strategy.

[0111] A joint optimization module is used to match the reconstruction view generated in the previous step with the pose constraint information extracted in the preprocessing stage, and to jointly optimize the neural radiance field to obtain a high-precision three-dimensional reconstruction view.

[0112] A reconstruction performance verification module is used to evaluate the quality of the reconstruction view using PSNR, SSIM and LPIPS values, and to set a reinforcement iteration strategy to seek a scene-compliant super parameter configuration, and to further optimize the view reconstruction result.

[0113] Embodiment 2

[0114] Reference Figures 2-7 For an embodiment of the present application, a three-dimensional reconstruction method and system based on deep learning are provided, and in order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiments are used for scientific demonstration.

[0115] The simple linear fully connected layer sampling method of the neural radiation field will cause the loss of local information, resulting in the blurring and aliasing of the reconstructed view. Some research methods express local details by selecting appropriate network architectures, but there are still few studies on the extraction and fusion of multi-scale feature information in network training, and the internal dependence between global information and local information has not been combined, making it difficult to solve the problem of feature sparsity or loss when the frequency of local details is low. At the same time, some three-dimensional scenes contain large blank spaces or chaotic background spaces, and directly using MLPs for color and density inference will cause ambiguous encoding features to adversely affect the reconstruction results.

[0116] Effects of the improved application:

[0117] Compared with the neural radiation field NeRF and its improved algorithm neural radiation field NeRF-ID, all experiments were trained and tested on 20 scenes of the Realistic Synthetic 360° dataset, the Real Forward-Facing dataset and the DTU dataset. Under the condition of 1024x32 single sampling number, the new view reconstruction results of each algorithm are shown in Table 1, Table 2 and Table 3.

[0118] Table 1 Quantitative indicators of new view reconstruction of different methods on the Realistic Synthetic 360° dataset

[0119]

[0120] Table 2 Quantitative indicators of new view reconstruction of different methods on the Real Forward-Facing dataset

[0121]

[0122]

[0123] Table 3 Quantitative indicators of new view reconstruction of different methods on part of the scenes of the DTU dataset

[0124]

[0125] Compared with the original NeRF network, the average score values of the three evaluation indicators PSNR, SSIM and LPIPS are improved by at least 4% or more. Among them, the reconstruction performance on the RealForward-Facing dataset is more significant, with 1.32dB (4.9%), 0.065 (7.9%), and -0.009 (12.9%) improvements compared with NeRF-ID, and 1.58dB (5.9%), 0.076 (9.3%), and -0.012 (16.4%) improvements compared with NeRF. The performance indicators in all scenarios are better than those of other comparative experimental groups.

[0126] Secondly, on the Realistic Synthetic 360° dataset, the average values of PSNR, SSIM and LPIPS of the network reconstruction view are improved by 0.41dB (1.27%), 0.003 (0.3%), and -0.003 (10.3%) compared with NeRF-ID, and 1.74dB (5.6%), 0.013 (1.4%), and -0.015 (36.6%) compared with NeRF. In most scenarios, the performance is also excellent. Finally, on the DTU dataset, the average values of PSNR, SSIM and LPIPS of the network reconstruction view are improved by 0.67dB (2.6%), 0.020 (2.5%), and -0.026 (11.1%) compared with NeRF-ID, and 0.98dB (3.9%), 0.028 (3.6%), and -0.038 (15.4%) compared with NeRF. The performance indicators in all random scenarios are better than those of other comparative experimental groups, which shows that the network proposed in the patent can effectively improve the performance of new view reconstruction.

[0127] Table 4 Comprehensive performance analysis of different methods

[0128]

[0129] Table 4 lists the reconstruction quality indicators PSNR and training time of some improved neural radiation field methods, which can reflect the comprehensive performance of each method to some extent.

[0130] From the analysis of the data in the table, it can be seen that the method proposed in the patent achieves the best reconstruction quality except for NeXT. However, the high training cost of NeXT using Transformer as a feature decoder limits its application, which is also the direction for improvement of the method proposed in the patent. From the training time, NSVF uses a sparse voxel octree structure to achieve fast rendering of the reconstructed view, but it is not suitable for real scenes.

[0131] Meanwhile, the focus of the method is mainly on the joint learning and screening of multi-scale feature information in the pixel region, which can be applied to other methods to achieve a better trade-off between reconstruction quality and computational cost.

[0132] Figure 7 The qualitative results of the new view reconstruction of different methods under the Room scene of the Real Forward-Facing dataset, the Materials scene of the Realistic Synthetic 360° dataset, and the Scan109 scene of the DTU dataset are shown. The first column in the figure is the real image not participating in the training, and the second to fourth columns are the reconstruction views of NeRF, NeRF-ID and the method of the present patent, respectively. According to the real value of the image marked by the circle in the first column, the local changes of the new view reconstruction results of different methods can be observed. It can be seen from the qualitative results that the method of the present patent has excellent view reconstruction capability while avoiding rendering blur and aliasing, and the reconstruction details are more abundant, such as the symbols on the blackboard, the concave-convex texture of the sphere, and the texture of the pigeon.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A 3D reconstruction method based on deep learning, characterized in that, include: Acquire scene image data; An improved three-dimensional reconstruction model of the neural radiation field was constructed. The improved three-dimensional reconstruction model of neural radiation field was trained to obtain the reconstruction results; Finally, the reconstruction results are evaluated, and a comprehensive assessment of the reconstruction results is conducted using performance indicators. The improved neural radiation field 3D reconstruction model includes a sine and cosine position coding network, a feature extraction network, and a volume rendering network; The feature extraction network includes a coarse sampling network and a fine sampling network, both of which contain two MLP models: a 5-layer fully connected sampling MLP model and a 3-layer fully connected inference MLP model. The improved 3D reconstruction model of the neural radiation field includes: using position encoding γ() to map the 3D position X of each pixel in the observed image and the corresponding camera pose d to a high-dimensional space before sampling, so that the light samples carrying image information are converted into high-dimensional sine and cosine signals in Fourier space. γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)) Where: p is the formal parameter representation of the three-dimensional position X and pose d, L represents the dimension of the position encoding, and in the neural radiation field, the position encoding γ(X) of the three-dimensional position X is set to L=10, and the position encoding γ(d) of the camera pose d is set to L=4. In the coarse sampling network, a multi-feature joint learning module is designed; The network input is the X-position encoding γ(X) of three-dimensional coordinates. One branch first divides each feature map of size C×H×W into n equal parts, then uses DCT to filter and complete the missing frequency components in the feature spectrum, focusing on local information in the image. Finally, frequency domain feature clustering is obtained by embedding frequency component information into the network. A strategy of first reducing dimensionality and then increasing dimensionality is used to construct an MLP bottleneck structure as the feature decoder, using ReLU as the activation function. The energy function en of each neuron is calculated. r We obtain the joint learning feature γ'(X): In the formula, γ(X) represents the original feature map, Norm(γ(X)) represents the normalized original feature map, Sigmoid() represents the neuron attention weight calculation, DCT() represents local feature filtering, nMLP() represents global feature sampling, and en r * It is the sum of the energies of the smallest neurons; In the fine-sampling network, the feature vector with an output resolution of N×256 is first normalized and reweighted to be within a uniform range; the feature vector dimension is expanded to twice its original size, with an output resolution of N×512; then, the ReLU activation function is used to add non-linearity; finally, the feature vector dimension is restored, with an output resolution of N×256, correcting the erroneous parts of the three-dimensional features in the coarse-sampling stage.

2. The deep learning-based 3D reconstruction method as described in claim 1, characterized in that: The improved three-dimensional reconstruction model of neural radiation field also includes: in the feature inference stage, designing a gated channel transformation multilayer perceptron module; using the l2-norm normalization method to establish the competition or cooperation relationship between channels, selectively transmitting feature information through the gate mechanism F, and realizing the reweighting of features in the channel dimension; When the gating weight of a channel is positively activated, GCT will promote the competition between the features of this channel and the features of other channels. When the gating weight of a channel is negatively activated, GCT will promote the cooperation between the features of this channel and the features of other channels, ultimately outputting the feature selection result. Where g represents the input feature of the gated unit, These are three trainable parameters: embedding weight α adjusts the encoded output by eliminating local ambiguity, and gating weight γ and gating bias β control the activation state of the channel features.

3. The deep learning-based 3D reconstruction method as described in claim 1, characterized in that: The obtained reconstruction results include: using volumetric rendering technology to reconstruct a new view based on test set images that were not used in training, and the qualitative comparison results between the reconstructed view and the original view can be used as an auxiliary evaluation standard for model performance.

4. The deep learning-based 3D reconstruction method as described in claim 1, characterized in that: The evaluation of the reconstruction results includes a comprehensive assessment using three performance metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (d(x, x0)). Where MSE represents the mean square error of a given original image I and a contrast image K of size m×n, i and j jointly represent the number of image pixels, and MAX... I SSIM represents the maximum pixel value of an image. It is a metric for measuring the similarity between two images. x represents the original image, y represents the comparison image, μ represents the average value, and σ represents the mean value. 2 σ represents variance. xy Let c1 represent the covariance of x and y, where c1 = (k1L). 2 c2 = (k2L) 2 This is a constant used to maintain stability, L is the dynamic range of pixel values, k1 = 0.01, k2 = 0.03, the range of the SSIM function is [-1, 1], d is the distance between x0 and x, where x0 represents the contrast image; x and x0 are fed into the neural network for feature extraction, and the output of each layer is activated and normalized, denoted as . Using vector ω l To scale the number of active channels, the final L2 distance is calculated, where l represents the network layer and H... l W l C l The scale of the l-layer network is represented by height, width, and number of channels, respectively.

5. A three-dimensional reconstruction system based on deep learning, characterized in that: The method for 3D reconstruction based on deep learning as described in claim 1 includes: Image data acquisition module, used to acquire scene images from multiple perspectives; The image data preprocessing module performs feature extraction, feature matching, and sparse reconstruction on the image data, and exports the corresponding camera pose, converting the camera parameters into pose constraints. An improved neural radiation field 3D reconstruction network module was developed to reduce the interference of ambiguous feature encoding on view reconstruction. Multiple reconstructed views were obtained by training the network using a hierarchical sampling strategy. The joint optimization module performs joint optimization on the neural radiation field to obtain a high-precision three-dimensional reconstructed view; The reconstruction performance verification module evaluates the quality of the reconstructed view, seeks hyperparameter configurations that suit the scenario, and then optimizes the view reconstruction results. The improved neural radiation field 3D reconstruction model includes a sine and cosine position coding network, a feature extraction network, and a volume rendering network; The feature extraction network includes a coarse sampling network and a fine sampling network.

6. The deep learning-based 3D reconstruction system as described in claim 5, characterized in that: The image data acquisition module sequentially sends the image data into the data preprocessing module; The processed parameter information is transmitted from the data preprocessing module to the improved neural radiation field 3D reconstruction network module to obtain multiple reconstruction views; The joint optimization module performs joint optimization of the neural radiation field by matching the reconstructed view generated in the previous step with the pose constraint information extracted in the preprocessing stage, and then transmits it to the reconstruction performance verification module. The reconstruction performance verification module evaluates the quality of the reconstructed view using peak signal-to-noise ratio, structural similarity index, and learning-aware image patch similarity value, and sets up an enhancement iteration strategy to seek hyperparameter configurations that fit the scenario, thereby optimizing the view reconstruction results.

Images