Generalized neural radiation field reconstruction method based on multi-modal information fusion

Abstract

The invention discloses a generalized neural radiation field reconstruction method based on multi-modal information fusion, and the method specifically comprises the following steps: 1, constructing a luminosity feature body and a geometric feature body based on unstructured multiple views, and constructing a multi-modal neural coding body through progressive complementary fusion; step 2, converting the multi-modal neural coding body and the original RGB pixel body of the unstructured multi-view into volume density and radiation brightness; 3, sampling light rays, and fusing context features of the sampled light rays to obtain context features of the light rays; and step 4, decoding the volume density and the radiation brightness by using the light context features, rendering to generate a free view angle RGB-D image, and then guiding dense reconstruction of a low texture scene by combining photometric supervision and sparse geometric supervision. According to the method, the problem of shape radiation ambiguity can be solved, high-quality three-dimensional reconstruction and two-dimensional rendering are realized, and the surface reconstruction precision of the generalized neural radiation field is improved.

Description

Technical Field

[0001] The present invention relates to the field of visual image generation technology, and specifically to a generalizable neural radiation field reconstruction method based on multimodal information fusion. Background Art

[0002] Scene reconstruction and new view generation based on multi-view visual information are long-standing challenges in computer vision and graphics. In recent years, neural rendering methods have significantly advanced the integration and development of these two fields. Neural Radiance Fields (NeRF) based on implicit 3D reconstruction and its subsequent work can already produce realistic new view synthesis results, but they rely heavily on the use of a large number of multi-view images combined with a long single-scene perspective optimization process. A significant disadvantage of these NeRF techniques is the shape radiation ambiguity problem, that is, the inability to explicitly reconstruct the three-dimensional structure of the scene with high quality. Another is that they are not generalizable and cannot be rendered quickly across scenes.

[0003] Work based on a small number of viewpoints has great advantages in efficiency and has demonstrated the generalization potential of neural radiance fields in inferring geometry and appearance information across scenes. These methods leverage recent successes in deep multi-view stereo (MVS) by warping 2D image features from nearby input views (inferred by 2D CNNs) onto a swept plane in the view cone of a reference view, constructing a cost volume at the input reference view to implicitly reason about the scene geometry, and quickly regressing the real image of the new viewpoint. It has strong generalizability and avoids tedious scene-by-scene optimization. However, when only a small number of viewpoints are available, the existing generalizable neural radiance field surface reconstruction accuracy is low, especially for complex areas such as boundaries and reflections, and incomplete geometric perception often leads to low-quality 2D view rendering results. Summary of the invention

[0004] The purpose of the present invention is to provide a generalizable neural radiation field reconstruction method based on multimodal information fusion. The present invention can solve the shape radiation ambiguity problem while achieving high-quality three-dimensional reconstruction and two-dimensional rendering, and improve the accuracy of generalized neural radiation field surface reconstruction.

[0005] The technical solution of the present invention is a generalizable neural radiation field reconstruction method based on multimodal information fusion, which specifically includes the following steps: Step 1: construct photometric feature bodies and geometric feature bodies based on unstructured multi-views, and construct a multimodal neural encoding body by progressively and complementary fusion of the photometric feature bodies and the geometric feature bodies; Step 2: Convert the original RGB pixel volume of the multimodal neural encoding volume and the unstructured multi-view into volume density and radiance; Step 3: based on the constructed multimodal neural encoder, sample light, fuse the context features of the sampled light based on the transformer network, and obtain the light context features; Step 4: Use light context features to decode volume density and radiance, generate free-view RGB-D images based on the decoded volume density and radiance rendering, and then combine photometric supervision with sparse geometric supervision to guide the dense reconstruction of low-texture scenes.

[0006] In the aforementioned generalizable neural radiation field reconstruction method based on multimodal information fusion, in step 1, the photometric feature volume is constructed by firstly using a bidirectional fusion backbone network based on unstructured multi-views. Extract image features, use ConvNeXt to extract multi-scale semantic information that provides regional and overall surface features of the target from downsampling by 4, 8, 16, and 32 times, extract shallow local appearance features downsampled by 4 times, and then perform bidirectional feature fusion to encode unstructured multi-views into semantically enhanced photometric feature volumes : ; Where: Represents unstructured multi-views.

[0007] In the aforementioned generalizable neural radiation field reconstruction method based on multimodal information fusion, the geometric feature volume is constructed The process is as follows: First get the camera parameters ,in, is the camera intrinsic parameter matrix, is the rotation matrix of the camera relative to the world coordinate system, is the translation vector of the camera relative to the world coordinate system, and then use the homography transformation matrix The first The 2D features of the auxiliary view are transformed to the reference view to obtain the depth Photometric feature : ; in, , and For the The camera parameters of the auxiliary view, , and are the camera parameters of the reference view, is the normal direction of the reference view; ; in, is the standardized device coordinates under the reference perspective; To explicitly encode the difference between the spatial point photometric features of unstructured multi-views, the photometric variance feature volume is calculated. , the formula is as follows: ; Among them, Var is the variance function of the photometric characteristics of each spatial point under the reference viewing angle; Then use 3D CNN to analyze the photometric variance feature volume Encode and decode, and convert it into an explicit geometric feature body through sigmoid operation .

[0008] In the above-mentioned generalizable neural radiation field reconstruction method based on multimodal information fusion, in step 1, the photometric feature volume and geometric features Constructing a multimodal neural coder through progressive complementary fusion The process is as follows: Calculate the photometric peak feature : ; Among them, MVSMaxPooling calculates the maximum value of the photometric feature of each spatial point under the reference viewing angle; Using 3D convolutional layers Fusion of raw RGB pixels and photometric variance feature , using 3D convolutional layers Fusion of photometric peak features and geometric features , and finally in the trainable proportional coefficient The final multimodal neural encoding body is generated under the control of , the formula is as follows: ; ; ; in, is the original RGB pixel volume and photometric variance feature The fusion feature body, It is the peak luminosity feature and geometric features fusion features.

[0009] In the aforementioned generalizable neural radiation field reconstruction method based on multimodal information fusion, the photometric peak feature volume Contains voxels, and geometric feature volumes They are fused separately to form a gating mechanism to filter out the noise introduced by non-surface elements in the feature space.

[0010] In the above-mentioned generalizable neural radiation field reconstruction method based on multimodal information fusion, in step 2, the multimodal neural encoding body and raw RGB pixel volumes of unstructured multi-views Convert to volume density and radiance The process is to build a neural radiation field , given any 3D position , viewing direction unit vector , learning multimodal neural encoders The 3D environment geometry and appearance information encoded in the neural radiance field From multimodal neural code and raw RGB pixel volumes of unstructured multi-views The corresponding volume density is converted by continuous interpolation and radiance , which can be expressed as follows: ; in, is the standardized device coordinates under the reference perspective, is the unit vector of the reference view coordinate system, the multimodal neural encoding volume Based on coordinates Perform trilinear interpolation.

[0011] In the above-mentioned generalizable neural radiation field reconstruction method based on multimodal information fusion, in step 3, the process of fusion of light context features is to firstly transform the camera light It is expressed as: ; in, is the ray origin, is the distance along the direction of the light, is the unit vector along the direction of the light; Then, the camera rays are layered and sampled from the ray origin. The farthest boundary and the nearest boundary ,Will arrive Divided into intervals, randomly select a sample point from each interval, Sampling points The formula is: ; Then, based on the transformer residual network Multimodal neural encoders of light context and the original RGB pixel volume Fusion, get the light context feature body : .

[0012] In the aforementioned generalizable neural radiation field reconstruction method based on multimodal information fusion, in step 4, the process of decoding volume density and radiation brightness using light context features is based on a multi-layer perceptron , and Decoding volume density point by point and radiance Among them, The decoding process of sampling points is expressed as: ; ; ; The process of rendering a free-view RGB-D image is to use a differentiable ray-stepping algorithm to decode the volume density. and radiance Render the RGB image C and the depth image D; where the RGB value of the kth pixel and depth value The calculation formula is as follows: ; ; ; in, represents the volume transmittance, Represents the total number of sampling points on a single ray, Indicates The distance from the sampling point to the origin, Indicates The distance from one sampling point to the next.

[0013] In the aforementioned generalizable neural radiation field reconstruction method based on multimodal information fusion, the sparse geometric supervision is to convert the high-confidence sparse point cloud output by the screening algorithm into a voxel representation, anchor the surrounding area to limit the generation of the smooth radiation field, and use the sparse depth map output by COLMAP to construct the geometric constraints of the depth image space; the sparse geometric supervision uses voxels as anchors to expand the region to form a geometric supervision signal in the form of a heat map. ; The geometric supervision signal Applied to geometric features On top, construct sparse point cloud loss .

[0014] In the above-mentioned generalizable neural radiation field reconstruction method based on multimodal information fusion, the photometric supervision process is to Perform training and calculate RGB loss ; The process of combining photometric supervision with sparse geometric supervision is to compute a sparse depth loss , and sparse point cloud loss Perform weighted summation to obtain the neural radiation field loss : ; in, and is the weight coefficient.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention constructs photometric feature bodies and geometric feature bodies, gradually and orderly fuses multimodal information, samples light and fuses light context features, decodes volume density and radiant brightness rendering to generate free-viewpoint RGB-D images, achieves high-quality three-dimensional reconstruction and two-dimensional rendering, solves the shape radiation ambiguity problem under as few as three unstructured multi-views, and combines the advantages of explicit and implicit scene reconstruction. The present invention introduces sparse geometric supervision signals and combines them with photometric supervision, thereby enhancing the robustness and accuracy of joint appearance and geometric reconstruction. In addition, the present invention introduces a multimodal information fusion module of geometric information, photometric information and semantic information, which gradually fuses the contextual features of light, thereby enhancing the generalization ability of the multimodal neural encoder under limited viewing angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the main idea of ​​the method of the present invention; Figure 2 The overall process of the method of the present invention is as follows; Figure 3 The specific structure of the bidirectional fusion backbone network of the present invention; Figure 4 The construction process of the photometric variance feature body and the photometric peak feature body of the present invention; Figure 5 A process for constructing a geometric feature body of the present invention; Figure 6 A multimodal neural coder fusion construction process for low-texture environments of the present invention; Figure 7 The light context fusion module structure of the present invention; Figure 8 This is an example of the RGB appearance reconstruction result on the LLFF dataset of the present invention; Fig. 9 This is an example of RGB appearance reconstruction results on the DTU dataset of the present invention; Fig.10 This is an example of the depth reconstruction result on the DTU dataset of the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0018] Example A generalizable neural radiation field reconstruction method based on multimodal information fusion, the main idea is as follows Figure 1 As shown, given an unstructured multi-view and the corresponding camera parameters , the 3D scene reconstruction task is to generate arbitrary perspective The corresponding RGB image C and depth image D. Geometric feature volume explicitly constructed for unstructured multi-views and photometric features , constructing a high-information multimodal neural encoder through progressive complementary fusion The geometric feature The geometric information is integrated into the photometric feature body. The multimodal neural encoder is constructed by integrating photometric information and semantic prior information. Based on the sampling of light, the light context features are fused based on the transformer network, and the free-view RGB-D image is generated by rendering. The dense reconstruction of low-texture scenes is guided by combining photometric supervision with sparse geometric supervision. The overall process of this method is as follows Figure 2 shown.

[0019] The specific steps are as follows: Step 1: construct photometric feature bodies and geometric feature bodies based on unstructured multi-views, and construct a multimodal neural encoding body by progressively and complementary fusion of the photometric feature bodies and the geometric feature bodies; In this step, the current scene reconstruction technology relies heavily on highly recognizable feature points, but is not sensitive enough to low-texture areas. The reason is that the underlying neural network cannot use high-level semantic information to understand the region or target as a whole from top to bottom, and can only mine feature point clues from low-level texture information from bottom to top. The instability of matching a small number of viewpoint feature points increases the difficulty of reconstruction. Therefore, in order to encode low-texture scene features from images with as few as 3 viewpoints, this embodiment introduces a pre-trained deep neural network to supplement the overall semantic consistency prior, transforms the features of as few as 2 auxiliary images into the reference view cone through a plane scanning mode, and fuses the variance information and peak information of multi-view features to construct a photometric feature volume .

[0020] use Figure 3 The bidirectional fusion backbone network shown Extract image features. The bidirectional fusion backbone network in this embodiment ConvNext, ConvNeXt is a modern convolutional neural network (CNN) architecture that introduces some features of the Transformer model while maintaining the advantages of traditional CNN to achieve higher accuracy and efficiency. The ConvNeXt architecture improves the performance of the Transformer model by using larger convolution kernels, improved network block design, simplified network structure, etc. A separate downsampling layer is used in the network to achieve more flexible and effective spatial dimension reduction and feature fusion. ConvNeXt pre-trained on ImageNet22K (ImageNet22K is a large image recognition dataset containing 22,000 categories) is used to extract multi-scale semantic information from 4, 8, 16, and 32 times of downsampling, providing overall surface features of regions and targets. An additional 2D CNN is used to extract shallow local appearance features that are downsampled 4 times. Through bidirectional feature fusion, unstructured multi-view Encoded as semantically enhanced photometric feature :

[0021] ; Where: Represents unstructured multi-views.

[0022] Get camera parameters ,in, is the camera intrinsic parameter matrix, is the rotation matrix of the camera relative to the world coordinate system, is the translation vector of the camera relative to the world coordinate system, and then use the homography transformation matrix The first The 2D features of the auxiliary view are transformed to the reference view to obtain the depth Photometric feature at : ; in, , and For the The camera parameters of the auxiliary view, , and are the camera parameters of the reference view, is the normal direction of the reference view; ; in, The normalized device coordinates for the reference perspective.

[0023] To explicitly encode the difference between the spatial point photometric features of unstructured multi-views, the photometric variance feature volume is calculated. , the formula is as follows: ; Among them, Var is the variance function of the photometric characteristics of each spatial point under the reference viewing angle.

[0024] Photometric Variance Eigenbody Essentially, it is constructed by pixel matching of multiple views, which has implicitly contained some geometric information. On this basis, 3D CNN is used for encoding and decoding, and it can be converted into an explicit geometric feature volume through sigmoid operation. ,like Figure 5 shown.

[0025] The Sigmoid operation is an S-shaped function, which is common in biology and is also known as the S-shaped growth curve. In information science, due to its monotonic and inverse monotonic properties, the Sigmoid function is often used as an activation function of a neural network to map variables between 0 and 1.

[0026] Use Figure 6 The process shown in the figure gradually transforms the semantically enhanced photometric feature volume With geometric features Fusion forms a multimodal neural encoder for low-texture environments The structure of the 3D CNN used, namely a neural network module with downsampling convolutional layers, upsampling convolutional layers and skip connections, can effectively infer and propagate global information of the environment.

[0027] In the joint reconstruction of appearance and geometry, the variance cost feature volume only contains the relative differences between views and cannot provide absolute scale information. Multi-view maximum pooling operation is performed to extract the effective peak information in each view to construct the photometric peak feature volume. Used to encode the fusion information of complete semantic information and photometric information:

[0028] ; Among them, MVSMaxPooling calculates the maximum value of the photometric feature of each spatial point under the reference perspective.

[0029] Photometric Variance Eigenbody and photometric peak features The construction process is as follows Figure 4 shown.

[0030] 3D CNN (Three-Dimensional Convolutional Neural Network) is a deep learning model that applies 3D convolution kernels to image or video data to extract features.

[0031] 3D Convolutional Layer Raw RGB pixel volume used to fuse unstructured multi-views and enhanced photometric variance features , 3D convolutional layer Used to fuse enhanced photometric peak features and geometric features , and finally in the trainable proportional coefficient The final multimodal neural encoding body is generated under the control of , specifically calculated by the formula: ; ; ; Enhanced photometric peak features There are a lot of meaningless voxels in the image. When the voxel point is on the surface of the object, the peak feature can be used to supplement the photometric clues of the absolute scale, while when the element point is in other non-surface locations in space, the peak from the multi-view feature has no effective meaning. Separate fusion can form a gating mechanism to filter out the noise introduced by non-surface elements in the feature space. However, the model has not yet completed the learning of sparse point clouds in the early stage of training. There is too much self-noise. Therefore, the proportionality factor The initial value of is set to 0. In subsequent training, the two photometric peak feature volumes can be adaptively evaluated by back propagation. and geometric features value ratio.

[0032] Step 2: Convert the multimodal neural encoding volume and the original RGB pixel volume of the unstructured multi-view into volume density and radiance; Traditional scene reconstruction works explicitly represent 3D scenes as point clouds or voxels, which are not detailed enough and texture-unfriendly. Using a neural network to implicitly represent the geometry and appearance of the environment, we can render dense RGB-D images of the three-dimensional environment based on the principles of physics, but the accuracy of the depth map is extremely low. It requires a large number of multi-view images for training, ranging from dozens to hundreds of images, and can only perform 3D reconstruction of a single environment. The network obtained through long-term optimization cannot be generalized to new environments. Some research works use the universal image features encoded by convolutional networks to expand the neural radiation field. However, its implicit learning nature results in poor quality in cross-scene geometry reconstruction.

[0033] In this step, after pre-training with image data from multiple scenes, there is no need to perform a time-consuming optimization process in the new environment, and fast 3D reconstruction can be performed with as few as 3 images. The difference from previous work is that the proposed model combines the advantages of high information content of explicit geometric information, photometric information, and semantic information and the smoothness characteristics of implicit learning, which can produce more accurate representations for low-texture environments under a small number of viewpoints.

[0034] Specifically, given any 3D position , viewing direction unit vector , neural radiation field From multimodal neural code and raw RGB pixel volumes of unstructured multi-views The corresponding volume density is regressed and radiance , expressed as: ; in, is the standardized device coordinates under the reference perspective, is the unit vector of the reference view coordinate system, Based on coordinates Trilinear interpolation was performed.

[0035] In order to enhance the high-frequency details in the reconstruction results, Position encoding is applied to convert it into a high-frequency representation. In addition, in order to provide the relative position of the sampling points for the subsequent converter network, Positional encoding is applied. Multimodal neural encoder The original RGB pixel volume of the unstructured multi-view has a relatively low resolution after 4 times downsampling. Can provide more high-frequency appearance information.

[0036] Step 3: based on the constructed multimodal neural encoder, sample light, fuse the context features of the sampled light based on the transformer network, and obtain the light context features; Since the 3D CNN used has a large regional receptive field, the multimodal neural encoder The fusion process causes the reconstruction result to be too smooth. The rendering process of the RGB image and the depth map is achieved by tracking the virtual camera light, and the light is discretized and aligned with the pixels. Therefore, the independent fusion of the light context information at this stage provides a more refined feeling and improves the volume density. The accuracy of regression.

[0037] like Figure 7 The ray context fusion module shown builds a neural radiance field , learning multimodal neural encoders The 3D environment geometry and appearance information encoded in the , is converted into volume density through continuous interpolation View-dependent radiance . It is expressed as:

[0038] ; in, is the ray origin, is the distance along the direction of the light, is a unit vector along the direction of the ray; the ray origin The farthest and nearest boundaries are and .

[0039] In order to fuse the light context information, the camera light is first sampled hierarchically. arrive Divided into intervals, randomly select a sample point from each interval, Sampling points The available formula is: ; Then based on the transformer residual network The ambient feature volume in the light context and the original RGB pixel volume Fusion to obtain light context features : ; Step 4: Use light context features to decode volume density and radiance, generate free-view RGB-D images based on the decoded volume density and radiance rendering, and then combine photometric supervision with sparse geometric supervision to guide the dense reconstruction of low-texture scenes.

[0040] In this step, based on the multi-layer perceptron , and Decoding volume density point by point and radiance , No. The decoding process of sampling points can be expressed as: ; ; .

[0041] Due to the existence of shape radiation ambiguity, the radiation field that simply satisfies the RGB constraint cannot guarantee the correct geometric structure. Especially in low-texture environments, the reconstructed geometric structure often deviates seriously. A geometric constraint based on 3D voxel space is introduced to limit the generation of smooth radiation fields by anchoring the surrounding area with sparse key point clouds. Due to a series of downsampling, the resolution of the 3D voxel space is lower than that of the original image, so the geometric constraint is limited. In order to obtain accurate reconstruction results with high resolution, the sparse depth map output by COLMAP is used to further construct the geometric constraints of the depth image space.

[0042] Neural radiation field A physically based voxel rendering process is used to render the RGB image and depth image using a differentiable ray-walking algorithm. The RGB value of the kth pixel and depth value It can be calculated by the formula:

[0043] ; ; ; in, represents the volume transmittance, Represents the total number of sampling points on a single ray, Indicates The distance from the sampling point to the origin, Indicates The distance from one sampling point to the next.

[0044] The overall network model is trained in an end-to-end mode. The L2 loss function is used to calculate the RGB loss during training. , using the smooth L1 loss function to calculate the sparse depth loss , followed by the sparse point cloud loss Weighted summation is performed to obtain the final neural radiation field loss :

[0045] ; in, and is the weight coefficient.

[0046] The scene appearance reconstruction experiments are conducted on the DTU and LLFF datasets, and the depth reconstruction experiments are conducted on the DTU dataset. The DTU dataset includes some low-texture scenes, each scene has 49 viewpoints, and each viewpoint contains 7 types of illumination. The data setting is consistent with the literature. 88 scene data on the DTU dataset are used to train the proposed generalizable neural radiance field, 16 scene data are used for testing, and RGB-D images with a resolution of 512×640 are used. The LLFF dataset includes some low-texture and reflective environments, each environment has RGB images of 20 viewpoints. This dataset has a different distribution from the DTU training set, and a total of 8 environments are used for testing. For each test environment, 7 nearby views are selected, of which 3 views are used as input, and the remaining 4 views are used to evaluate the model performance.

[0047] Given a true value image and generate images In the experiment, PSNR, SSIM and LPIPS indicators are used to evaluate the model's RGB image generation performance under new perspectives.

[0048] PSNR is the peak signal-to-noise ratio, which is defined based on the mean square error (MSE). The larger the PSNR, the less image distortion. The specific calculation formula is:

[0049] ; ; SSIM is a structural similarity index that quantifies local structural similarity from three aspects: brightness, contrast, and structural similarity, following the human visual system. The larger the SSIM, the less image distortion. The specific calculation formula is:

[0050] ; in, and Respectively and The average value of and Respectively and The variance of express and The covariance of and To maintain a stable constant, avoid the situation where the denominator is 0.

[0051] LPIPS is a learning method to perceive the similarity of image blocks. It uses the L2 distance of image depth features to measure the similarity of image pairs, which better reflects human perception than PSNR and SSIM. The smaller the index, the less image distortion.

[0052] Abs err is the mean absolute error, calculated by averaging the absolute depth errors of all pixels, in meters. Acc is the threshold percentage, Acc (0.01) indicates the percentage of pixels with an absolute depth error below 0.01 meters, and Acc (0.05) indicates the percentage of pixels with an absolute depth error below 0.05 meters.

[0053] In the experiment, the number of plane scans is set to 128, the number of rays randomly sampled in a single batch is set to 1024, and the multi-task loss weight is set to and Set to 2 and 1. The model was trained on a server with a single NVIDIA Titan RTX GPU using the Adam optimizer, with an initial learning rate of , combined with the cosine annealing strategy to dynamically adjust the learning rate, the best performance was achieved after 15 cycles.

[0054] The present invention constructs photometric feature bodies and geometric feature bodies, gradually and orderly fuses multimodal information, samples light and fuses light context features, decodes volume density and radiant brightness rendering to generate free-viewpoint RGB-D images, achieves high-quality three-dimensional reconstruction and two-dimensional rendering, solves the shape radiation ambiguity problem under as few as three unstructured multi-views, and combines the advantages of explicit and implicit scene reconstruction. The present invention introduces sparse geometric supervision signals and combines them with photometric supervision to enhance the robustness and accuracy of joint appearance and geometric reconstruction. In addition, the present invention can achieve high-quality generalized environment three-dimensional reconstruction tasks, and when as few as three unstructured multi-views are captured, the depth estimation performance exceeds the advanced three-dimensional reconstruction neural network MVSNet supervised by the full depth map. In the new view generation task, the present invention significantly improves the rendering quality of complex areas such as boundaries and reflections compared to the existing advanced generalizable neural rendering work MVSNeRF, and achieves more accurate new view generation performance. The present invention introduces a multimodal information fusion module of geometric information, photometric information and semantic information, which gradually fuses the contextual features of light, thereby enhancing the generalization ability of the multimodal neural encoder under limited viewing angles.

[0055] Furthermore, the proposed method is compared with other generalizable baseline methods, including PixelNeRF, IBRNet, and MVSNeRF, for the appearance reconstruction indicators of the LLFF and DTU test datasets, and the results are shown in Tables 1 and 2. PixelNeRF discretely fuses multi-view features through average pooling operations, but cross-scene training has overfitting problems and is difficult to generalize to the LLFF environment. IBRNet greatly improves the RGB appearance reconstruction performance of the generalized environment by performing feature fusion along the light. The scope of its feature fusion is limited to 2D images and 3D light, so the perceptual similarity of the generated 3D space is low. MVSNeRF uses 3D CNN to fuse the features of the entire environment on the cost body, but the lack of regional and target-level perception introduces noise into the rendering process.

[0056] Table 1 Quantitative results of new view synthesis on the LLFF dataset

[0057] Table 2 Quantitative results of new view synthesis on the LLFF dataset

[0058] The results of quantitative comparative experiments show that the proposed method significantly outperforms the baseline method on both datasets, with average relative improvements of 1.6%, 1.6% and 4.6% on PSNR, SSIM and LPIPS, respectively, verifying that the strategy of combining multi-feature fusion and light context fusion can more reasonably fuse 3D environmental context information and generate more refined new perspective images. The performance improvement on LPIPS is the largest, indicating that the photometric feature enhanced by semantic prior extracts effective semantic perception information through bidirectional encoding, providing richer semantic knowledge for environmental generalization of neural radiation fields. In particular, this method supplements the unfair comparison with the original NeRF, which is a non-generalizable offline tuning method. It uses the full amount of test scene images for training, and after 10.2 hours of optimization, about 200 thousand iterations. It is worth noting that without the use of test scene image tuning, the method of the present invention surpasses the NeRF method in all indicators, verifying the feasibility of high-quality generalized NeRF method.

[0059] The geometric reconstruction indicators of the reference view and the new view are divided into two categories. For the DTU test data set, the method proposed in the present invention is compared with other generalizable baseline methods, including the neural radiation field methods PixelNeRF, IBRNet and MVSNeRF, and the classic multi-view deep neural network method MVSNet. The results are shown in Table 3. Like other neural radiation field methods, the method of the present invention only has RGB images as input data, and new viewpoints can be freely selected for geometric reconstruction. MVSNet is trained using real depth maps, but can only reconstruct the depth of the reference view.

[0060] Table 3 Quantitative depth reconstruction results on the DTU dataset

[0061] The results of quantitative comparative experiments show that the method of the present invention significantly surpasses all baseline methods, regardless of the input perspective or the new perspective. Using only three input images, the reconstructed Abs err is 1 cm in the reference perspective. Compared with the existing generalizable neural radiation field model, it is reduced by 56.5% and Acc(0.01) is improved by 13.5%. In addition, the reconstruction performance of the new perspective is even greater, with Abs err reduced by 62.9% and Acc(0.01) increased by 18.4%.

[0062] It is worth noting that unlike MVSNet, which uses real annotated dense depth maps, the neural radiance field method proposed in this paper only uses sparse key points reconstructed by SFM, and can still achieve high-quality depth reconstruction using geometric supervision at both voxel and ray levels. This is the first time that a neural radiance field type model has achieved comprehensive performance superior to multi-view stereo networks in generalized neural networks, reducing the reconstructed Abs err by 44.4% in the reference view.

[0063] Figure 8 An example of RGB appearance reconstruction results on the LLFF dataset is shown, and the proposed method is qualitatively compared with the state-of-the-art baseline method MVSNeRF. The results show that flickering and artifacts can be observed in the generated views of MVSNeRF, while the proposed method can better adapt to environments with different distributions, proving that the proposed technology has good generalizability. In reflective low-texture areas, such as TV screens and lacquered desktops, the highlight effect generated by the proposed method changes significantly with the viewing angle, and is more realistic in brightness and shape, proving that the proposed feature volume fusion and light fusion strategies can effectively help the radiation field model understand the anisotropy of reflective surfaces.

[0064] Fig. 9An example of RGB appearance reconstruction results on the DTU dataset is shown, and the proposed method is qualitatively compared with the state-of-the-art baseline method MVSNeRF. The method of the present invention has a strong advantage in the results of free view generation in low-texture environments. Inside the low-texture area, both methods can generate reasonable new view results. However, at the edge of the low-texture area, the generation result of MVSNeRF has obvious artifacts, while the generation result of the method of the present invention has clear and sharp boundaries. This proves that the neural radiation field model that integrates semantic priors and sparse key point cloud information has a certain overall target perception ability, and can effectively divide the boundary area in the three-dimensional space even with a small number of input perspectives, thereby improving the rendering quality of new views.

[0065] Fig.10 An example of deep reconstruction results on the DTU dataset is shown, and a qualitative comparison is made between the proposed method and the most advanced generalizable neural radiation field method MVSNeRF. The visualization results show that MVSNeRF fails to solve the shape radiation ambiguity problem of the neural radiation field, and its reconstruction results contain a lot of background noise, the edges of objects are blurred, and the overall surface reconstruction accuracy is affected by the resolution of the feature body and tends to be smooth. The method of the present invention extracts effective features from low-texture surfaces, emphasizes the edge information at the boundaries of objects by anchoring robust key points, and solves the shape radiation ambiguity problem of the neural radiation field.

[0066] In summary, the present invention can solve the shape radiation ambiguity problem while achieving high-quality three-dimensional reconstruction and two-dimensional rendering, and improve the accuracy of generalized neural radiation field surface reconstruction.

Claims

1. A generalizable neural radiation field reconstruction method based on multimodal information fusion, characterized in that: The specific steps include: Step 1: construct photometric feature bodies and geometric feature bodies based on unstructured multi-views, and construct a multimodal neural encoding body by progressively and complementary fusion of the photometric feature bodies and the geometric feature bodies; Step 2: Convert the original RGB pixel volume of the multimodal neural encoding volume and the unstructured multi-view into volume density and radiance; Step 3: based on the constructed multimodal neural encoder, sample light, fuse the context features of the sampled light based on the transformer network, and obtain the light context features; Step 4: Use light context features to decode volume density and radiance, generate free-view RGB-D images based on the decoded volume density and radiance rendering, and then combine photometric supervision with sparse geometric supervision to guide the dense reconstruction of low-texture scenes.

2. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 1 is characterized in that: In step 1, the photometric feature volume is constructed by first using a bidirectional fusion backbone network based on unstructured multi-views. Extract image features, use ConvNeXt to extract multi-scale semantic information that provides regional and overall surface features of the target from downsampling by 4, 8, 16, and 32 times, extract shallow local appearance features downsampled by 4 times, and then perform bidirectional feature fusion to encode unstructured multi-views into semantically enhanced photometric feature volumes : ； Where: Represents unstructured multi-views.

3. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 2 is characterized in that: The geometric feature body is constructed The process is as follows: First get the camera parameters ,in, is the camera intrinsic parameter matrix, is the rotation matrix of the camera relative to the world coordinate system, is the translation vector of the camera relative to the world coordinate system, and then use the homography transformation matrix The first The 2D features of the auxiliary view are transformed to the reference view to obtain the depth Photometric feature : ； in, , and For the The camera parameters of the auxiliary view, , and are the camera parameters of the reference view, is the normal direction of the reference view; ； in, , , 1 is the standardized device coordinate under the reference view angle; To explicitly encode the difference between the spatial point photometric features of unstructured multi-views, the photometric variance feature volume is calculated. , the formula is as follows: ； Among them, Var is the variance function of the photometric characteristics of each spatial point under the reference viewing angle; Then use 3D CNN to analyze the photometric variance feature volume Encode and decode, and convert it into an explicit geometric feature body through sigmoid operation .

4. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 3 is characterized by: In step 1, the photometric feature and geometric features Constructing a multimodal neural coder through progressive complementary fusion The process is as follows: Calculate the photometric peak feature : ； Among them, MVSMaxPooling calculates the maximum value of the photometric feature of each spatial point under the reference viewing angle; Using 3D convolutional layers Fusion of raw RGB pixels and photometric variance feature , using 3D convolutional layers Fusion of photometric peak features and geometric features , and finally in the trainable proportional coefficient The final multimodal neural encoding body is generated under the control of , the formula is as follows: ； ； ； in, is the original RGB pixel volume and photometric variance feature The fusion feature body, It is the peak luminosity feature and geometric features fusion features.

5. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 4 is characterized in that: The photometric peak feature Contains voxels, and geometric feature volumes They are fused separately to form a gating mechanism to filter out the noise introduced by non-surface elements in the feature space.

6. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 4 is characterized in that: In step 2, the multimodal neural encoder and raw RGB pixel volumes of unstructured multi-views Convert to volume density and radiance The process is to build a neural radiation field , given any 3D position , viewing direction unit vector , learning multimodal neural encoders The 3D environment geometry and appearance information encoded in the neural radiance field From multimodal neural code and raw RGB pixel volumes of unstructured multi-views The corresponding volume density is converted by continuous interpolation and radiance , which can be expressed as follows: ； in, is the standardized device coordinates under the reference perspective, is the unit vector of the reference view coordinate system, the multimodal neural encoding volume Based on coordinates Perform trilinear interpolation.

7. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 6, characterized in that: In step 3, the process of fusion of light context features is to firstly transform the camera light It is expressed as: ； in, is the ray origin, is the distance along the direction of the light, is a unit vector along the direction of the light; Then, the camera rays are layered and sampled from the ray origin. The farthest boundary and the nearest boundary ,Will arrive Divided into intervals, randomly select a sample point from each interval, Sampling points The formula is: ； Then, based on the transformer residual network Multimodal neural encoders of light context and the original RGB pixel volume Fusion, get the light context feature body : 。 8. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 7, characterized in that: In step 4, the process of decoding volume density and radiance using light context features is based on a multi-layer perceptron , and Decoding volume density point by point and radiance Among them, The decoding process of sampling points is expressed as: ； ； ； The process of rendering a free-view RGB-D image is to use a differentiable ray-stepping algorithm to decode the volume density. and radiance Render the RGB image C and the depth image D; where the RGB value of the kth pixel and depth value The calculation formula is as follows: ； ； ； in, represents the volume transmittance, Represents the total number of sampling points on a single ray, Indicates The distance from the sampling point to the origin, Indicates The distance from one sampling point to the next.

9. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 5, characterized in that: The sparse geometric supervision is to convert the high-confidence sparse point cloud output by the screening algorithm into voxel representation, anchor the surrounding area to limit the generation of the smooth radiation field, and use the sparse depth map output by COLMAP to construct the geometric constraints of the depth image space; the sparse geometric supervision uses voxels as anchor points to expand the region and form a geometric supervision signal in the form of a heat map. ; The geometric supervision signal Applied to geometric features On top, construct sparse point cloud loss .

10. The generalizable neural radiation field reconstruction method based on multimodal information fusion according to claim 9, characterized in that: The photometric supervision process is to Perform training and calculate RGB loss ; The process of combining photometric supervision with sparse geometric supervision is to compute a sparse depth loss , and sparse point cloud loss Perform weighted summation to obtain the neural radiation field loss : ； in, and is the weight coefficient.

Images