Generalized method for joint optimization of structured light and neural radiation field

Abstract

The invention belongs to the technical field of new view angle synthesis, and discloses a generalized method for joint optimization of structured light and a neural radiation field. Superposing a plurality of complex components with independent phases to generate structured light; the sampling factor is applied to the structured light, bicubic interpolation is used for sampling the structured light, and the process is irrelevant to depth information; simulating a light transmission process from the scene to be measured to the stereo camera by adopting geometrical optics; a view angle conversion operation is completed according to the depth image and the shielding image of the camera view angle, the structured light is distorted to a binocular view angle, and an active stereo image is obtained; the active stereo image and the RGB stereo image generate stereo image features and a three-dimensional feature body together through a feature extractor, and a renderer based on a neural radiation field is adopted to perform new view angle synthesis and geometric prediction; the whole process from the structured light to the nerve radiation field can be microprocessed, and joint optimization is carried out.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of novel view synthesis, and in particular to a generalizable method of joint optimization of structured light and neural radiance field. BACKGROUND

[0002] Novel view synthesis is an important problem in computer vision and graphics that has existed for a long time. The introduction of neural radiance field (NeRF) has revolutionized this field. NeRF uses a multi-layer perceptron to implicitly represent a scene, and through volume rendering in a five-dimensional radiance field, it achieves a realistic synthesis effect. However, NeRF needs to be optimized for each scene, and its generalizability is poor, limiting its practical application scenarios.

[0003] To solve the problem of insufficient generalizability of NeRF, depth information guided NeRF optimization has become a recognized effective approach: by introducing additional geometric constraints, the model's dependence on single scene data can be significantly reduced, improving cross-scene generalization ability. However, traditional depth-guided schemes rely heavily on monocular depth estimation techniques, and such methods still have shortcomings in terms of geometric prediction accuracy and synthesis results.

[0004] Active stereo depth estimation methods based on structured light provide a low-cost solution to the above problems. The active stereo system is equipped with two cameras and a projection module. The projection module generates a pre-designed structured light pattern to artificially add a layer of texture to the measured scene, and the cameras are used to calculate the parallax to obtain the scene depth information. This process takes advantage of the spatial characteristics of structured light, enhancing the uniqueness of matching points and achieving reliable depth estimation, providing a key technical path for guiding NeRF optimization.

[0005] Using active stereo depth estimation methods to guide NeRF optimization still faces some problems. The structured light patterns projected by active stereo technology are all pre-designed, which means that the model cannot dynamically optimize the structured light pattern based on reconstruction feedback during NeRF training, resulting in the inability to dynamically improve the accuracy and reliability of depth information during the NeRF reconstruction process. Instead, it becomes a "static bottleneck" that restricts the upper limit of NeRF optimization, ultimately making it difficult for NeRF to fully utilize the active sensing advantages of structured light. SUMMARY

[0006] The purpose of the present application is to propose a method of joint optimization of structured light and NeRF, which improves the depth perception ability by optimizing the structured light, thereby further enhancing the generalizability of NeRF.

[0007] The technical solution of the present application is as follows: a generalizable method of joint optimization of structured light and neural radiance field, comprising the following steps:

[0008] Step 1, active stereo image synthesis; according to Fourier optics theory, a plurality of complex components with independent phase are superimposed to generate structured light;

[0009] The sampling factor is applied to the structured light, and bicubic interpolation is used to complete the sampling of the structured light , bicubic interpolation is represented, structured light generated according to Fourier optics theory is represented, the sampling factor of the structured light in the projection process is represented, which is irrelevant to the depth information;

[0010] The geometric optics is used to simulate the light transmission process from the scene to be measured to the stereo camera; according to the depth map of the camera view angle and the occlusion map the view angle conversion operation is completed, the structured light is warped to the binocular view angle, and the Lambert model is used to obtain the active stereo image;

[0011] Step 2, the active stereo image and the RGB stereo image are jointly subjected to a feature extractor to generate two-dimensional stereo image features and three-dimensional feature bodies ; the left image features are represented, the right image features are represented, the three-dimensional feature bodies of the left image are represented, the three-dimensional feature bodies of the right image are represented, the source view angle image is represented;

[0012] Step 3, the three-dimensional feature bodies and the two-dimensional stereo image features are subjected to a neural radiance field-based renderer for new view angle synthesis and geometric prediction;

[0013] Step 4, the structured light, the feature extractor and the neural radiance field-based renderer are trained by using depth true values and image true values; based on the full-process differentiable processing of the structured light to the neural radiance field, joint optimization is performed.

[0014] The structured light is parameterized and represented as density, size and gray scale; the density is controlled by controlling the phase matrix , the size of the speckle point is controlled by increasing a square function constructed based on a Sigmoid function , the brightness of the speckle point is controlled by increasing a gray scale matrix , and after Fourier transform , the structured light is represented as:

[0015] ​

[0016] in, The phase matrix is ​​processed using an automatic differentiation mechanism. With grayscale matrix ;and The definition is as follows:

[0017]

[0018] and Indicates the width and height of the image. Set to 30; Indicates the size of the speckle; , Represents the pixel coordinates in the image.

[0019] The sampling factor of structured light during projection is the ratio of the camera pixel size to the illumination pattern pixel size:

[0020]

[0021] in, Indicates the pixel pitch of the camera. Indicates the pixel pitch of the projector. Indicates the camera's focal length. This indicates the focal length of the projector.

[0022] The feature extractor uses a stereo feature extractor as the main path, integrates information from the depth estimation network branch, generates stereo image features and stereo depth information, and generates a three-dimensional feature volume through a multi-view network.

[0023] The stereo feature extractor includes an encoder, a stereo attention module, and a decoder;

[0024] Depth information generated by the depth estimation network branch from the active stereo image and the RGB stereo image is stitched together with the RGB stereo image and used as input to the stereo feature extractor. A weighted encoder maps the stitched image to the feature space. Three stacked stereo attention modules effectively fuse the features in the feature space with the stereo-related features generated by the depth estimation network branch from the active stereo image and the RGB stereo image to obtain geometric perception features. These geometric perception features are then used by the weighted decoder to generate two-dimensional stereo image features. .

[0025] The depth estimation network uses UniMatch, a generalizable depth perception method based on Transformer, as the baseline. In the encoder part, when the feature resolution is 1 / 8, the feature information of the DepthAnythingV2 base model is incorporated by feature addition to construct a dual-branch network, including an RGB stereo image branch and an active stereo image branch, which fuses RGB texture information and structured light information.

[0026] Depth estimation networks convert RGB stereo images With active stereo imaging As input to the dual-branch network; for the RGB stereo image branch, the encoder, DepthAnythingV2, and Transformer are frozen; for the active stereo image branch, DepthAnythingV2 is frozen, and the encoder and Transformer are retrained; the left and right feature maps corresponding to the two branches are fused through the feature fusion module, and the fused stereo-related features are obtained. Additional information is input into the stereo attention module of the stereo feature extractor; stereo-related features The stereo depth information is obtained after the matching and propagation module:

[0027]

[0028] In obtaining features of two-dimensional stereo images and 3D depth information Subsequently, a multi-view network was used to generate 3D feature volumes. and depth map and using stereo depth information To guide the construction of 3D feature volumes

[0029] The neural radiation field-based renderer specifically works as follows: for a new viewpoint, it emits a ray of light for each pixel and samples the light along the ray points in three-dimensional space to obtain... Each sampling point; sequentially passed through a multi-head attention layer, an autoencoder network, and a multilayer perceptron. Aggregation is performed to obtain the volume density of each sampling point. The estimation; projecting each sampling point onto the source viewpoint, from as well as Mid-sample corresponding features and use a multilayer perceptron to predict color. Traditional volume rendering techniques are used to achieve new perspective compositing and geometric estimation.

[0030]

[0031]

[0032] in, Let K be the volume density at point k in space. For point Relative to the depth value of the new pose, For color prediction values, This is a geometric prediction value.

[0033] The training loss function Including depth loss and color loss ;

[0034]

[0035] in, Defined as rendering color The mean square error between the actual color and the true color; Divided into and Two parts:

[0036]

[0037] in, The depth loss function is the depth estimate output by the depth estimation network. Depth estimate output by multi-view network And geometric predictions obtained by a renderer based on neural radiation fields. Compare; Defined as , , The distance function between the depth and the true depth is defined as follows:

[0038]

[0039] in, For the mask, To predict depth values, For the true depth value; for the norm Using smoothing Norm; True depth value If the depth falls within the effective depth range defined by the near and far depth thresholds provided by COLMAP, then... Set to 1; otherwise, The value is assigned to 0.

[0040] The joint optimization is defined as:

[0041]

[0042] in, Defined as structured light parameters, Defined as the network parameters of the feature extractor and the neural radiation field-based renderer.

[0043] The beneficial effects of this invention are:

[0044] (1) This invention proposes a generalizable method for joint optimization of structured light and neural radiation field. By jointly optimizing structured light and neural radiation field, the structured light pattern can be dynamically and adaptively adjusted according to the synthesis effect and geometric prediction results, thereby effectively improving the depth estimation accuracy, enhancing the accuracy and stability of neural radiation field reconstruction, and improving the generalization performance of the method for unknown scenes.

[0045] (2) This invention integrates active stereo vision into a generalizable viewpoint synthesis task and proposes a feature extractor to effectively extract stereo image features and obtain high-quality depth prior information. High-quality depth prior information helps to improve the generalization performance of the neural radiation field. Under conditions such as complex geometric structures and low texture, it can provide stable and reliable geometric constraints for the neural radiation field, thereby improving the accuracy and consistency of the viewpoint synthesis results.

[0046] (3) An experimental setup was built for this invention, and experimental verification was conducted in both publicly available datasets and real-world scenes collected by the experimental setup. The experimental results show that the method of this invention can still obtain stable and accurate rendering and geometric reconstruction results under real imaging and projection conditions, indicating that the method does not depend on specific datasets or ideal acquisition conditions and has strong practical application capabilities. Attached Figure Description

[0047] Figure 1 The flowchart is as follows: (a) is the process of training and testing on a public dataset; (b) is the process of testing in a real-world scenario.

[0048] Figure 2 This is a diagram of the feature extractor architecture.

[0049] Figure 3 This is a schematic diagram of the fusion module;

[0050] Figure 4 Diagram of the device;

[0051] Figure 5 (a) is an active stereo image from a public dataset, and (b) is an active stereo image from a real-world scene.

[0052] Figure 6The graphs are: (a) scene graph, (b) IBRNet processing result, (c) GeoNeRF processing result, (d) GNT processing result, (e) StereoNeRF processing result, and (f) processing result of our method.

[0053] Figure 7 The graphs are: (a) scene graph, (b) IBRNet processing result, (c) GeoNeRF processing result, (d) GNT processing result, (e) StereoNeRF processing result, and (f) processing result of this method.

[0054] Figure 8 The graphs are geometric prediction graphs; (a) is the result of IBRNet processing, (b) is the result of GeoNeRF processing, (c) is the result of GNT processing, (d) is the result of StereoNeRF processing, and (e) is the result of our method. Detailed Implementation

[0055] Figure 1 This is a flowchart of the method of the technical solution of the present invention, where (a) is the process during the training phase and testing on a public dataset, and (b) is the process during testing in a real-world scenario. The generalizable method for joint optimization of structured light and neural radiation fields proposed in this invention comprises three parts: active stereo image synthesis, a feature extractor, and a renderer based on neural radiation fields. First, the method uses Fourier transform to generate structured light, achieving parameterized representation of the size, grayscale, and density of the structured light, enabling adaptive optimization of the structured light during the training process. Active stereo images are then synthesized using geometric optics and a Lambertian model. Next, the stereo image and the active stereo image are used as input to the feature extractor to extract stereo image features and high-precision depth information. Finally, a neural renderer is used to achieve the synthesis of new perspectives.

[0056] The specific implementation process includes the following steps:

[0057] (1) Active stereo image synthesis. Based on Fourier optics theory, structured light is generated by superimposing multiple complex components with independent phases. First, the structured light is parameterized, representing its density, size, and grayscale. This is achieved by controlling the phase matrix. To control the density, a square function based on the Sigmoid function is added. To control the speckle size, the grayscale matrix is ​​increased. To control the brightness of the speckle, a Fourier transform is performed. Structured light can be represented as:

[0058]

[0059] in Regarding the differentiability of structured light parameters, the phase matrix is ​​handled using the automatic differentiation mechanism in PyTorch. With grayscale matrix ,and The definition is as follows:

[0060]

[0061] and Indicates the width and height of the image. Set it to 30. The differentiability of structured light is guaranteed by the continuity and differentiability of the sigmoid function. By ensuring the differentiability of these components, a differentiable representation of structured light generation is achieved. This allows the optimization process of structured light to be seamlessly embedded into the algorithm flow, thereby supporting end-to-end training and optimization.

[0062] According to geometric optics, when using a conventional projector, the sampling factor of the structured light pattern during projection is the ratio of the camera pixel size to the illumination pattern pixel size:

[0063]

[0064] in and These represent the pixel pitch of the camera and the projector, respectively. and These represent the focal lengths of the camera and projector, respectively. A sampling factor is applied to the structured light pattern, and bicubic interpolation is used to sample the structured light. Since this process is unrelated to depth information, It can be applied in any scenario.

[0065] After simulating the projection process, it is necessary to simulate the light transmission process from the scene under test to the stereo camera. This process is implemented using geometric optics. This is achieved using a depth map from the camera's perspective. and occlusion map Complete the viewpoint transformation operation and apply structured light. Distorted to a binocular perspective:

[0066]

[0067] in, This represents element-wise multiplication, and warp represents the warp operator; after viewpoint transformation, an active stereo image is obtained using the Lambertian model:

[0068]

[0069] in, A scalar value describing exposure and sensor spectral quantum efficiency is set to 0.9. This represents ambient light, set to 0.3. This indicates the projector's power; set it to 0.8. It is Gaussian noise. This represents a reflectance map.

[0070] (2) Feature Extractor. The network architecture of the feature extractor is as follows: Figure 2 As shown, the feature extractor uses a stereo feature extractor as the main path, fusing information from the depth estimation network branch to generate stereo image features and stereo depth, which are then processed by a multi-view network to generate a 3D feature volume. The stereo feature extractor consists of an encoder, a stereo attention module, and a decoder.

[0071] Regarding the stereo feature extractor, firstly, the depth information generated by the depth estimation network branch is concatenated with the RGB stereo image as input to the feature extractor; a weighted encoder is used to map the concatenated image to the feature space; then, three stacked stereo attention modules are used to effectively fuse the features in the feature space and the stereo-related features from the depth estimation network branch to obtain geometric perception features; finally, a weighted decoder is used to generate stereo image features. The reason for using RGB stereo images as input is that the purpose of constructing 3D feature volumes later is to encode the appearance and geometric information of the scene. If a simple active stereo image or a mixture of the two images is used as input, it will affect the encoding of the scene appearance.

[0072] Regarding the depth estimation network, the UniMatch architecture is used as the baseline, and feature information from the DepthAnythingV2 deep learning model is incorporated to improve generalization. A dual-branch network is constructed to fully integrate RGB texture information and structured light information. The introduction of structured light information enhances the reliability of feature matching. The depth estimation network uses RGB stereo images... With active stereo imaging As input to the dual-branch network, firstly, for the RGB stereo image branch, the encoder, DepthAnythingV2, and Transformer are frozen; then, for the active stereo image branch, considering that UniMatch is trained on RGB images and cannot effectively extract features from active stereo images, the encoder and Transformer for this branch are retrained, while DepthAnythingV2 remains frozen; subsequently, a simple feature fusion module is constructed, such as... Figure 3As shown, the left and right feature maps corresponding to the two branches are fused, and the fused stereo related features are then combined. Additional information is input into the attention module of the stereo feature extractor to transfer geometric knowledge. Finally, the depth information is obtained through the matching and propagation module.

[0073]

[0074] In obtaining features of two-dimensional stereo images and 3D depth Subsequently, a multi-view network architecture was used to generate the 3D feature volume. and depth map and using depth information To guide the construction of three-dimensional feature volumes, where express For the source view image.

[0075] (3) A renderer based on neural radiation fields. This renderer obtains the 3D feature volume from the source viewpoint. and two-dimensional stereo image features Next, a neural renderer is used for new viewpoint synthesis and geometric prediction. First, for a new viewpoint, a ray is emitted for each pixel, and samples are taken along the ray points in 3D space to obtain... Each sampling point; then, sequentially through a multi-head attention layer, an autoencoder network, and a multilayer perceptron... Aggregation is performed to obtain the volume density of each sampling point. The estimation; then, each sampling point is projected onto the source viewpoint, from as well as Mid-sample corresponding features and use a multilayer perceptron to predict color. Finally, traditional volume rendering techniques are used to achieve new perspective compositing and geometric estimation:

[0076]

[0077]

[0078] in, Let K be the volume density at point k in space. For point The depth value relative to the new pose.

[0079] (4) Training process. The model is trained using ground truth depth data and ground truth image data. The loss function mainly consists of two parts: depth loss and image loss. and color loss :

[0080]

[0081] in, Defined as rendering color The mean square error between the actual color and the true color. For Divided into and Two parts:

[0082]

[0083] in, The depth loss function is the depth estimate output by the depth estimation network. Depth estimate output by multi-view network And geometric predictions obtained by a renderer based on neural radiation fields. Compare them. It is then defined as , , The distance function between the depth and the true depth. This distance function is defined as follows:

[0084]

[0085] in, For the mask, To predict depth values, This represents the true depth value. For the norm... Using smoothing Norm. Specifically, if the true depth value If the depth falls within the effective depth range defined by the near and far depth thresholds provided by COLMAP, then... Set to 1; otherwise, The value is assigned to 0.

[0086] (5) Joint optimization: Joint learning can be achieved based on the above methods. Specifically, firstly, all parameters controlling the structured light have been differentiable to ensure the differentiability of structured light generation; then, during the projector projection process, since the camera and projector parameters are fixed, This can be considered a constant, only controlling the size of the structured light pattern and not affecting differentiability. Furthermore, the camera acquisition process involves fundamental differentiable operations, ensuring the differentiability of this process. Finally, the feature extractor and the neural radiation field-based renderer are naturally differentiable. Thus, the entire process from structured light to NeRF is differentiable. Therefore, based on the above definition of the loss function, the joint optimization is defined as:

[0087]

[0088] in, Defined as structured light parameters, Defined as network parameters. Therefore, the joint optimization problem can be formulated as: minimizing the loss function by optimizing the structured light parameters and network parameters. Since the entire process is fully differentiable, the problem can be solved using stochastic gradient descent.

[0089] The device of the present invention, such as Figure 4 As shown. The camera used is a binocular stereo camera with a resolution of 1280×720; the projector has a resolution of 1920×1080. The binocular stereo camera was calibrated to perform epipolar correction of the image. Simultaneously, to ensure accurate geometric correspondence between projection and imaging, the binocular stereo camera and projector were jointly calibrated. The core objective of this calibration was to establish the precise geometric relationship between the binocular stereo camera and the projector. Figure 5 This demonstrates active stereo images synthesized from public datasets and employing... Figure 4 Active stereo images of the actual scene captured by the device.

[0090] Compared with existing technologies, this invention enhances the generalization of NeRF and significantly improves the quality of new perspective synthesis and the accuracy of geometric prediction.

[0091] Qualitative evaluation results on public datasets, such as Figure 6 As shown, (a) is the scene image to be synthesized, (b) is the qualitative evaluation image of IBRNet with enhanced generalization using the Transformer architecture, (c) is the qualitative evaluation image of GeoNeRF with enhanced generalization through multi-view feature aggregation, (d) is the qualitative evaluation image of GNT with enhanced generalization using the Transformer architecture, (e) is the qualitative evaluation image of StereoNeRF with enhanced generalization using a stereo setup, and (f) is the qualitative evaluation image of the present invention. Compared with other methods, the method proposed in this invention significantly improves the quality of synthesized new perspective images, not only reducing artifacts and better preserving scene details, but also demonstrating excellent performance in thin structure restoration.

[0092] Qualitative evaluation results in real-world scenarios, such as Figure 7 As shown, (a) is the scene image to be synthesized, (b) is the qualitative evaluation image of IBRNet with enhanced generalization using the Transformer architecture, (c) is the qualitative evaluation image of GeoNeRF with enhanced generalization through multi-view feature aggregation, (d) is the qualitative evaluation image of GNT with enhanced generalization using the Transformer architecture, (e) is the qualitative evaluation image of StereoNeRF with enhanced generalization using a stereo setup, and (f) is the qualitative evaluation image of the present invention. Compared with other methods, the method of the present invention exhibits stronger robustness, smoother synthesis results, and fewer artifacts.

[0093] New perspective synthetic images and corresponding geometric prediction maps, such as Figure 8 As shown, (a) is the synthesized image and geometric prediction map of the new perspective using IBRNet with enhanced generalization via Transformer architecture; (b) is the synthesized image and geometric prediction map of the new perspective using GeoNeRF with enhanced generalization via multi-view feature aggregation; (c) is the synthesized image and geometric prediction map of the new perspective using GNT with enhanced generalization via Transformer architecture; (d) is the synthesized image and geometric prediction map of the new perspective using StereoNeRF with enhanced generalization via stereo setup; and (e) is the synthesized image and geometric prediction map of the new perspective of this invention. It can be seen that where other methods exhibit errors in the synthesized new perspective images, corresponding inaccuracies or distortions are also observed in the geometric prediction, indicating that geometric estimation is closely related to the synthesis of the new perspective.

[0094] The quantitative evaluation results are shown in Table 1. PSNR, SSIM, and LPIPS were used to evaluate the synthesis quality of the new perspective, while AbsRel, Sq Rel, and... <1.25 is used to evaluate the quality of geometric predictions. As can be seen from Table 1, the method of this invention achieves optimal performance across all metrics.

[0095] Table 1 Quantitative Assessment Results .

Claims

1. A generalizable method of structured light and neural radiance field joint optimization, characterized in that, The method comprises the following steps: Step 1, active stereo image synthesis; according to Fourier optical theory, a plurality of complex components with independent phases are superimposed to generate structured light; The sampling factor is applied in structured light and the bicubic interpolation is used to complete the sampling of structured light , represents bicubic interpolation, represents structured light generated according to Fourier optics theory, represents the sampling factor of structured light in the projection process, which is irrelevant to the depth information; The light transmission process from a scene to be measured to a stereo camera is simulated using geometric optics; a depth map according to a camera perspective and an occlusion map A perspective conversion operation is performed to warp structured light to a binocular perspective, and an active stereo image is obtained using a Lambertian model; Step 2, the active stereo image and the RGB stereo image are jointly fed into a feature extractor to generate two-dimensional stereo image features and three-dimensional feature volumes ; representing left image features, representing right image features, representing left image three-dimensional feature volumes, representing right image three-dimensional feature volumes, representing to the source view image; Step 3, three-dimensional feature body and two-dimensional stereoscopic image features New view synthesis and geometry prediction with neural radiance fields based renderers; Step 4, training the structured light, the feature extractor and the neural radiance field-based renderer by using the depth true value and the image true value; based on the full-process differentiable processing of the structured light to the neural radiance field, joint optimization is performed.

2. The method of claim 1, wherein, Parameterize the structured light, represent the structured light as density, size and gray scale; control the phase matrix Control the density by increasing the square function based on the Sigmoid function Control the size of the speckle points by increasing the gray scale matrix Control the brightness of the speckle points by Fourier transform The structured light is represented as: wherein ; using an automatic differentiation mechanism to handle the phase matrix and the gray scale matrix ; and are defined as follows: and denotes the width and height of the image, is set to 30; denotes the size of the speckle points; , denotes the pixel coordinates in the image.

3. The method of claim 1, wherein, The sampling factor of the structured light in the projection process is the ratio of the camera pixel size to the illumination pattern pixel size: wherein, denotes the pixel pitch of the camera, denotes the pixel pitch of the projector, denotes the focal length of the camera, denotes the focal length of the projector.

4. The method of claim 1, wherein, The feature extractor mainly uses a stereo feature extractor, fuses the information of a depth estimation network branch, generates stereo image features and stereo depth information, and generates a three-dimensional feature body through a multi-view network.

5. The method of claim 4, wherein, The stereo feature extractor comprises an encoder, a stereo attention module and a decoder; The depth information generated by the depth estimation network branch of the active stereo image and the RGB stereo image is spliced with the RGB stereo image as the input of the stereo feature extractor; An encoder with shared weights is used to map the spliced image to a feature space; The three stacked three-dimensional attention modules are used to effectively fuse features in the feature space and stereo-related features generated by the active stereo image and the RGB stereo image by a depth estimation network branch to obtain geometric perception features; the geometric perception features are generated into two-dimensional stereo image features by a shared weight decoder .

6. The method of claim 4, wherein, The depth estimation network uses a generalizable depth perception method UniMatch based on Transformer as a baseline, at a feature resolution of 1 / 8 in the encoder part, the feature information of a depth base model DepthAnythingV2 is fused through feature addition to construct a double-branch network, including an RGB stereo image branch and an active stereo image branch, and RGB texture information and structured light information are fused; Depth estimation networks convert RGB stereo images With active stereo imaging As input to the dual-branch network; for the RGB stereo image branch, the encoder, DepthAnythingV2, and Transformer are frozen; for the active stereo image branch, DepthAnythingV2 is frozen, and the encoder and Transformer are retrained; the left and right feature maps corresponding to the two branches are fused through the feature fusion module, and the fused stereo-related features are obtained. Additional information is input into the stereo attention module of the stereo feature extractor; stereo-related features The stereo depth information is obtained after the matching and propagation module: After obtaining the two-dimensional stereo image features and stereo depth information , a multi-view network is used to generate a three-dimensional feature volume and a depth map , and the stereo depth information is used to guide the construction of the three-dimensional feature volume.

7. The method of claim 6, wherein, The neural radiance field-based renderer specifically comprises: for a new view angle, emitting a light ray for each pixel, and sampling points in a three-dimensional space along the light ray to obtain a sampling point; The multi-head attention layer, the auto-encoder structure network and the multi-layer perception are sequentially passed through to aggregate, so as to complete the estimation of the density of each sampling point ; each sampling point is projected to a source view angle, corresponding features are sampled from and , and a multi-layer perception is used to predict color ; a traditional volume rendering technique is used to realize new view synthesis and geometry estimation: wherein, is the volume density of the spatial midpoint k, is the point is the depth value relative to the new pose, is the color prediction value, is the geometry prediction value.

8. The method of claim 7, wherein, The loss function of the training includes a depth loss and a color loss ; wherein, defined as the rendered color the mean squared error between the true color; divided into and two parts: wherein, is a self-supervised depth loss function comparing the depth estimate value output by the depth estimation network , the depth estimate value output by the multi-view network , and the geometry prediction value obtained by the neural radiance field-based renderer ; is defined as , , is a distance function between the true depth and the predicted depth, the distance function being defined as wherein, is a mask, is a predicted depth value, is a true depth value; for the norm , a smoothing norm is applied; the true depth value falls within the valid depth interval defined by the near and far depth thresholds provided by COLMAP, then is set to 1; otherwise, is assigned a value of 0.

9. The method of claim 1, wherein, The joint optimization is defined as: wherein, defined as a structured light parameter, defined as a network parameter of the feature extractor and the neural radiance field based renderer.

Images