Robust single-frame structured light three-dimensional imaging method and system based on neural feature decoding

Abstract

This invention discloses a robust single-frame structured light 3D imaging method and system based on neural feature decoding, comprising: a data generation module, a neural feature matching module, and a depth optimization module; constructing a large-scale synthetic structured light dataset through physical simulation of structured light; building a neural feature matching module to obtain an initial depth map through neural feature matching; and designing a depth optimization module to inject the initial depth map as geometric cues into the visual base model, further optimizing the initial depth map to generate the final depth map, thus achieving single-frame structured light 3D imaging. This invention overcomes the shortcomings of low accuracy and poor robustness of 3D imaging methods in complex real-world scenes such as strong ambient light, high reflectivity, and semi-transparency, and solves the problem of a severe lack of large-scale, high-quality training data when applying deep learning methods to the field of structured light, thereby improving the accuracy and robustness of single-frame structured light 3D imaging in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, and provides a 3D imaging method, specifically involving a robust single-frame structured light 3D imaging method and system based on deep learning neural feature decoding. Background Technology

[0002] Structured light is an active 3D scanning technology that actively projects specially designed light patterns (such as stripes, grids, or speckles) onto an object. One or more cameras then observe the deformation of these light patterns on the object's surface, and finally, calculations are used to deduce the object's three-dimensional shape and depth information. Active structured light (SL) 3D imaging is a key technology for acquiring three-dimensional information about objects. Among these technologies, single-frame structured light technology, which only requires projecting a single statically encoded pattern to complete 3D reconstruction, has the advantages of high efficiency and high dynamic adaptability, and is widely used in commercial 3D imaging equipment.

[0003] Existing single-frame structured light technologies can be divided into traditional pixel-domain matching-based decoding and deep learning-based decoding. The former remains the mainstream method and is widely used in commercial devices, while the latter is still in the early exploratory stage. Traditional pixel-domain matching-based decoding is the mainstream solution for commercial systems (such as Intel RealSense D435 and Microsoft Kinect V1). Its core idea is to calculate depth by matching the pixel intensity information of the projected pattern and local image patches in the infrared image captured by the camera. However, this type of method heavily relies on low-level, localized pixel information, which is easily disturbed in complex scenes. Therefore, when faced with missing surface textures, object occlusion, or complex non-Lambertian materials such as reflections or transparency, pixel-level matching becomes highly unstable, resulting in poor decoding robustness and a significant decrease in 3D reconstruction quality. Deep learning-based decoding methods are still in their early stages of application in the structured light field, mainly limited by data and methodology. In terms of data, structured light technology lacks large-scale, high-quality public datasets. Existing work mostly relies on small synthetic data or real data without ground truth for training, resulting in poor model generalization ability. In terms of methodology, existing works (such as Yinda Zhang et al., Activestereonet: End-to-end self-supervised learning for active stereo systems. ECCV 2018) ignore the rich spatial coding information contained in the known projection pattern when decoding, and treat it merely as an additional texture. They fail to fully utilize the inherent advantages of structured light, and the training process is unstable, with poor generalization, making it difficult to apply in practice.

[0004] In summary, existing traditional single-frame structured light 3D imaging technology suffers from poor robustness and low accuracy in complex scenes, while 3D imaging methods using deep learning have weak model generalization ability due to data scarcity and model design limitations. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention provides a robust single-frame structured light 3D imaging method and system based on neural feature decoding, and provides a matching model training data generation scheme to overcome the defects of low accuracy and poor robustness of traditional 3D imaging methods in complex real-world scenes (such as weak texture, reflection, occlusion, etc.), and to solve the bottleneck problem of a severe lack of large-scale, high-quality training data when deep learning methods are applied to the field of structured light.

[0006] This invention upgrades the fragile pixel-domain matching in traditional structured light decoding to a robust neural feature space by designing a complete framework that includes large-scale synthetic data generation and neural decoding. The method supports hardware configurations for both monocular cameras (single infrared camera + projector) and binocular cameras (two infrared cameras + projector). The core of this invention includes: 1. Using the open-source software Blender to perform physical simulation of structured light, rendering a large-scale, high-quality synthetic structured light dataset containing diverse 3D scenes, lighting conditions, physical materials, and various structured light projection patterns in a physically and programmatically manner. 2. In structured light decoding, a neural network is used to encode the structured light image and structured light projection pattern into a high-dimensional neural feature space, obtaining the neural features of the structured light image and structured light projection pattern. Then, a recurrent neural network based on convolutionally gated recurrent units is used to perform neural feature matching of the structured light image features and structured light projection pattern features, obtaining high-precision, robust disparity estimation. Finally, triangulation is used to obtain high-precision, robust depth estimation. 3. In terms of decoding, based on neural feature matching, this invention also utilizes the DepthAnythingV2 (Lihe Yang et al., Depth Anything V2. NeurIPS 2024) visual foundation model for depth estimation to extract visual prior information from structured light images. The depth map obtained from neural feature matching is injected into the DepthAnythingV2 visual foundation model for depth estimation. The powerful perception capability of the visual foundation model is used to further optimize the depth results, thereby improving the accuracy and robustness of single-frame structured light 3D imaging in complex scenes such as strong ambient light, high reflectivity, and semi-transparency.

[0007] The method includes the following steps:

[0008] a) Construct a simulated structured light environment, programmatically synthesize a simulated structured light environment containing diverse scene layouts, multiple objects, multiple lighting, and various object surface materials, perform physical simulation of structured light in the environment, and construct a large-scale synthetic structured light dataset.

[0009] Using a physically based simulation rendering engine (such as Cycles, which comes with the open-source 3D software Blender), a large-scale synthetic structured light dataset containing diverse 3D scenes, lighting conditions, object materials, and various structured light projection patterns is rendered physically and procedurally to support model training. The structured light dataset includes: a large number of synthetic scene models, diverse lighting conditions, materials, and various structured light projection patterns, as well as a large number of structured light infrared images and corresponding true depth and true parallax.

[0010] b) Construct a neural feature matching module, which includes: a feature encoder, a context encoder, and a recurrent neural network based on convolutional gated recurrent units;

[0011] First, a feature encoder encodes the input single-frame structured light projection pattern and the structured light infrared image of the scene captured by at least one camera into a neural feature space, obtaining neural feature maps for the structured light projection pattern and the structured light infrared image, respectively. Simultaneously, a context encoder encodes contextual features from the structured light infrared image captured by the left infrared camera. The neural feature maps and contextual features are combined into multi-source neural features, and iterative neural feature matching is performed through a recurrent neural network based on convolutional gated recurrent units to obtain a disparity map. Finally, an initial depth map is calculated using triangulation principles. This includes:

[0012] b1) Use a convolutional neural network to encode the structured light projection pattern and the scene structured light infrared image into the neural feature space, and extract multi-source neural features respectively;

[0013] Extracting multi-source neural features: In practice, based on the input single-frame projection pattern image, left infrared image, and (in a binocular configuration) right infrared image, the feature encoder in the neural feature matching module extracts high-dimensional neural feature maps from the left infrared image and projection pattern image, respectively. Multi-source neural features are further subdivided into local neural features and contextual neural features. For local neural features, in a monocular configuration, the same feature encoder is used to extract local neural features from the left infrared image and structured light projection pattern; in a binocular configuration, while using the same feature encoder to extract local neural features from the left infrared image and structured light pattern as in the monocular configuration, a second set of neural feature maps is extracted from the left and right infrared images using an additional feature encoder with the same structure but different parameters. For contextual neural features, regardless of whether it is a monocular or binocular configuration, the same context encoder is used to extract multi-scale contextual neural features from the left infrared image for subsequent iterative optimization.

[0014] b2) Construct one (in monocular configuration) or two (in binocular configuration) multi-scale structured light cost volumes based on local neural features.

[0015] Specifically, a three-dimensional cost volume is constructed by calculating the correlation between local neural feature maps of two different images. The three-dimensional cost volume encodes the dense matching relationship between the projection pattern and the captured image, with each element in the cost volume representing the similarity between the two images. Multi-level average pooling is then applied to the three-dimensional cost volume to construct a multi-level cost volume pyramid, capturing multi-scale information. In a monocular configuration, a cost volume is constructed using the local neural features of the left infrared image and the structured light projection pattern; in a binocular configuration, a second cost volume is constructed using the local neural features of the left and right infrared images, resulting in two cost volumes.

[0016] b3) A recurrent neural network based on a convolutional gated recurrent unit (GRU) is used to iteratively predict and optimize disparity based on the cost volume and contextual neural features to obtain a disparity map. Finally, the initial depth map is calculated using the triangulation principle. The recurrent neural network specifically uses a gated recurrent unit (GRU).

[0017] In practice, the predicted disparity map is initialized as a disparity map with all zeros. In each iteration, based on the current disparity estimate, relevant features are sampled from the cost volume pyramid and combined with contextual neural features. These features are then input into a convolutional gated recurrent unit (CLU) to update the hidden state, resulting in a new hidden state. The updated hidden state outputs the disparity difference value through the CLU's output header, which is then added to the current disparity estimate to obtain the updated disparity estimate. After N iterations, the final predicted disparity map is output, and the disparity map is converted into an initial depth map using triangulation principles.

[0018] c) Design a depth optimization module that uses the initial depth map as a geometric cue and injects it into the open-source vision base model DepthAnythingV2. This module further optimizes the initial depth map to generate the final depth map. The depth optimization module uses only the left infrared image and is compatible with both monocular and binocular configurations.

[0019] The depth optimization module includes: an open-source, pre-trained depth estimation visual foundation model, DepthAnythingV2, and a prompt net consisting of three convolutional layers. The DepthAnythingV2 visual foundation model comprises a ViT encoder (Vision Transformer) and a DPT decoder (DensePrediction Transformer). The ViT encoder encodes high-level global information from the image, and the DPT decoder infers the image depth from the encoded high-level global information. The depth optimization module uses the ViT encoder to extract high-level global information from the left infrared image, encodes the initial depth map into geometric cue features using the prompt net, and injects these features into the DPT decoder. The DPT decoder, based on the high-level global information extracted by the ViT encoder and the geometric cue features of the initial depth, infers a more accurate final depth map, achieving single-frame structured light 3D imaging.

[0020] The design of the deep optimization module includes the following steps:

[0021] c1, an extended depth estimation visual base model, DepthAnythingV2.

[0022] A depth estimation visual foundation model, DepthAnythingV2, pre-trained on massive images and fine-tuned on structured light data, is used as the backbone network of the depth optimization module. DepthAnythingV2 comprises a Transformer-based image ViT encoder and a DPT (Dense PredictionTransformer) decoder. The ViT encoder extracts high-level global information such as semantics from the image; it is extended at the decoder, receiving both the intermediate layer encoding from the ViT encoder and an initial depth map as a prompt to repair low-quality regions in the initial depth map.

[0023] c2, Input the left infrared image for encoding and inject the initial depth map; including: inputting the left infrared image into the ViT encoder, extracting the output features of the intermediate layer of the ViT encoder, and inputting them into the DPT decoder;

[0024] The initial depth map is used as a strong geometric cue, which is then encoded into geometric cue features by a prompt network.

[0025] The intermediate layer features extracted from the intermediate layer of the ViT encoder and the cue features obtained from the initial depth map encoding are input into the DPT decoder.

[0026] c3, the DPT decoder outputs the final depth map.

[0027] The DPT decoder fuses monocular visual features from the infrared image obtained from the ViT encoder with geometric cue features from the initial depth map to optimize the initial depth map and output the final depth map.

[0028] In specific implementation, the present invention utilizes the above method to realize a single-frame structured light three-dimensional imaging system, including:

[0029] a) A data generation module for generating large-scale synthetic structured light datasets through physical simulation, the datasets containing diverse scenes, lighting, materials and projection patterns, and providing corresponding true depth and true parallax;

[0030] b) A neural feature matching module for matching the input single-frame projection pattern with the scene infrared image captured by at least one camera in the feature space, and generating an initial depth map through iterative optimization;

[0031] c) A depth optimization module is used to receive the initial depth map as a geometric cue, combine it with the high-level visual features of the scene infrared image, and optimize it using a pre-trained and structured light-tuned depth estimation visual base model DepthAnythingV2 to generate the final depth map.

[0032] The technical solution provided by this invention has the following significant beneficial effects:

[0033] This invention provides a robust single-frame structured light 3D imaging method and system based on neural feature decoding, which has the following technical advantages:

[0034] (i) Solving the data bottleneck: The large-scale, diverse and physically realistic synthetic dataset proposed in this invention successfully solves the data shortage problem faced by deep learning methods in the field of single-frame structured light, laying the foundation for the effective training and powerful generalization ability of subsequent neural networks.

[0035] (II) High Precision and Robustness: This invention elevates the decoding process from pixel space to feature space, significantly improving the matching robustness in challenging regions such as weak texture, reflection, and occlusion. Experiments show that the accuracy metrics of this method (such as mean absolute error (MAE) and root mean square error (RMSE)) are significantly better than traditional template matching methods and existing learning-based single-frame structured light algorithms such as ActiveStereoNet.

[0036] (iii) Powerful generalization ability from synthesis to reality: Thanks to high-quality synthetic data and robust feature matching strategy, the method of this invention only needs to be trained on synthetic datasets and can be directly applied to real structured light hardware devices without any fine-tuning of real-world data, and achieves excellent performance.

[0037] (iv) Universality to Diverse Projection Patterns: This invention uses a mixture of multiple patterns during training, enabling the trained model to directly handle various projection patterns without further training. It also exhibits good generalization ability for new patterns not previously seen in the training set, with no significant performance degradation. This invention does not rely on the design of specific patterns, possessing strong universality and flexibility.

[0038] (v) Improved reconstruction integrity and detail: By introducing a depth optimization module and utilizing the prior knowledge of large-scale pre-trained models, boundary artifacts and missing details in the initial depth map are effectively repaired. For areas where the structured light projection pattern features are damaged due to special materials (such as semi-transparent, highly reflective) or poor lighting (such as strong ambient light), complete and high-quality depth results can also be obtained, making the final output depth map and 3D point cloud more structurally complete and sharper in detail. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the process of constructing a dataset according to the present invention.

[0040] Figure 2 This is a flowchart of the method of the present invention.

[0041] Figure 3 This is a structural block diagram of the neural feature matching module in the system of the present invention.

[0042] Figure 4 This is a structural block diagram of the deep optimization module in the system of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] This invention provides a robust single-frame structured light 3D imaging method based on neural feature decoding. Its core includes: 1. In terms of training data acquisition, unlike existing methods that only use consumer-grade structured light camera products to collect a small amount of low-quality real data, or only superimpose structured light projection patterns on existing non-structured light color images to obtain synthetic data, this invention uses the open-source software Blender to perform physical simulation of structured light, and physically and programmatically renders a large-scale, high-quality synthetic structured light dataset containing diverse 3D scenes, lighting conditions, physical materials and various structured light projection patterns. 2. In structured light decoding, unlike existing methods that directly use the pixel values ​​of the original image for matching and decoding, and unlike existing methods that ignore the structured light projection pattern and rely on two structured light images obtained from two cameras during matching and decoding, this invention utilizes a neural network to encode the structured light image and structured light projection pattern into a high-dimensional neural feature space, obtaining the neural features of the structured light image and structured light projection pattern. Then, a recurrent neural network based on convolutional gated recurrent units is used to perform neural feature matching of the structured light image features and structured light projection pattern features, obtaining high-precision and robust disparity estimation. Finally, triangulation is used to obtain high-precision and robust depth estimation. Thanks to the full utilization of the structured light projection pattern, this invention supports both monocular and binocular camera hardware configurations. 3. Unlike existing methods that only utilize feature matching for decoding, this invention, based on neural feature matching, also utilizes the DepthAnythingV2 (Lihe Yang et al., Depth Anything V2. NeurIPS 2024) visual foundation model for depth estimation to extract visual prior information from structured light images. The depth map obtained from neural feature matching is injected into the DepthAnythingV2 visual foundation model for depth estimation. By leveraging the powerful perception capabilities of the visual foundation model, the depth results are further optimized, improving the accuracy and robustness of this invention in complex scenes such as strong ambient light, high reflectivity, and semi-transparency.

[0045] like Figure 2 As shown, the robust single-frame structured light 3D imaging method based on neural feature decoding provided by this invention includes the following stages and steps:

[0046] Phase 1: Constructing a large-scale physical simulation dataset for structured light. Figure 1 The diagram illustrates the process of constructing a dataset according to this invention, including:

[0047] Step A: Construct a simulated structured light environment, design a method for large-scale synthetic data production, and generate a large-scale, diverse structured light dataset.

[0048] This invention first proposes and constructs a physical simulation platform for structured light imaging. This platform uses Blender's Cycles path tracing engine for simulation rendering to ensure the physical realism of the rendering.

[0049] The platform procedurally generates a large number (e.g., 2860) of indoor scenes with different layouts and fills them with thousands (e.g., over 5000) of 3D objects with diverse materials such as diffuse reflection, specular reflection, and transparency. At the same time, it uses multiple (e.g., 100) high dynamic range (HDR) environment maps for lighting to simulate complex lighting conditions in the real world.

[0050] The camera pose, object placement, and projection patterns of various types (such as dot matrix, stripe, and random binary textures) are randomized in the scene to generate a large number (e.g., 953,000 sets) of high-fidelity synthetic samples. Each sample contains a binocular infrared image, a projection pattern, and pixel-aligned ground truth and disparity maps. The infrared image, projection pattern, and ground truth map are obtained through physically based simulation rendering, and the ground truth disparity map is calculated from the ground truth map based on the principle of triangulation.

[0051] Phase 2: Construct a neural feature matching module to obtain an initial depth map through neural feature matching;

[0052] The neural feature matching module specifically adopts a module from passive binocular depth estimation. It extracts neural features from structured light infrared images and projection patterns using a neural network, thereby performing matching within the neural feature space. Existing structured light methods either directly match raw pixel values ​​or only extract neural features from the infrared image, ignoring the structured light projection pattern. The structure of the neural feature matching module constructed in this invention is as follows: Figure 3 As shown, it includes:

[0053] Step B1: Receive the externally input single-frame structured light projection pattern and the structured light infrared image captured by the infrared camera, and perform multi-source neural feature extraction.

[0054] It receives a single-frame structured light projection pattern from external input, a structured light left infrared image captured by the left infrared camera, and a structured light right infrared image captured by the right infrared camera (in a binocular configuration).

[0055] A feature encoder with a convolutional neural network (CNN) structure is used to extract high-dimensional local neural feature maps and contextual features from the left infrared image and the projected pattern image, respectively. In a stereo configuration, an additional feature encoder with the same structure but different parameters is used to extract a second set of local neural feature maps from the left and right infrared images, respectively.

[0056] A separate convolutional neural network-based context encoder is used to extract multi-scale contextual neural features from the left infrared image for subsequent iterative optimization. The operation of extracting contextual neural features is the same in both monocular and binocular configurations.

[0057] Step B2: Construct a multi-scale structured light cost body.

[0058] Unlike existing methods that require two camera images to construct the cost volume, this invention proposes constructing a structured light cost volume simultaneously between the left infrared camera image and the structured light projection pattern (in both monocular and binocular configurations), and between the left infrared camera image and the right infrared camera image (only in binocular configurations), fully utilizing the information contained in the structured light projection pattern. Specifically, a three-dimensional structured light cost volume is constructed by calculating the correlation between the local neural feature maps of the left infrared image and the structured light projection pattern image (e.g., using dot product operations). This cost volume encodes the dense matching relationship between the projection pattern and the captured image, and the elements in the cost volume represent the similarity of the local neural features of the left infrared image and the structured light projection pattern. In a binocular structured light configuration, a second three-dimensional structured light cost volume is constructed using the correlation between the local neural feature maps of the left and right infrared images. Each element of this cost volume represents the similarity of the local neural features of the left and right infrared images.

[0059] Multi-level average pooling is performed on the 3D cost volume formed by the local neural feature maps of the left infrared image and the structured light projection pattern. Multiple (e.g., 4) average pooling operations are performed along the last dimension of the 3D cost volume. Each average pooling operation reduces the dimension of the last dimension to half its original value. The 3D cost volume before and after each pooling operation is saved, constructing a multi-level (e.g., 4-level) cost volume pyramid to capture multi-scale information. In the case of binocular structured light, the same operation is performed on the second 3D cost volume formed by the local neural feature maps of the left and right infrared images.

[0060] Step B3: Iteratively optimize to generate an initial depth map.

[0061] A recurrent neural network based on convolutionally gated recurrent units (GRU) is used as the backbone network for iterative optimization. This network iteratively predicts the disparity map, and the resulting disparity map is converted into a depth map using triangulation principles. This depth map serves as a low-precision initial depth output, which will be further optimized in the next step.

[0062] In the first iteration, the predicted disparity map is initialized as a disparity map with all zeros. This disparity map is the target of iterative optimization. Each iteration has a disparity map estimate before the current iteration (i.e., the current disparity estimate). After this iteration, this disparity estimate is updated and becomes the "current disparity estimate" for the next iteration. As the iteration progresses, this current disparity estimate is continuously optimized, and finally, after multiple iterations (e.g., 22 iterations), the final predicted disparity map is formed. In each iteration, based on the current disparity estimate, the similarity of local neural features of the left infrared image and the structured light projection pattern is sampled from the cost volume pyramid (if it is a binocular configuration, the similarity of local neural features of the left infrared image and the right infrared image is additionally sampled from the second cost volume pyramid, and the two similarities are concatenated). Combined with the contextual neural features obtained in step B, the data is input into the convolutional gated recurrent unit to update the hidden state, resulting in a new hidden state. The updated hidden state outputs the disparity difference through an output header. This difference is added to the current disparity estimate to obtain the updated disparity estimate. This new disparity estimate then becomes the "current disparity estimate" for the next iteration. After multiple iterations (e.g., 22 iterations), the final disparity map is output, and it is converted into an initial depth map using triangulation principles.

[0063] The third stage: Generating the final depth map through a depth optimization module (depth optimization module). This invention is the first to propose introducing a depth estimation visual foundation model into structured light decoding and designing a depth optimization module to further optimize the output initial depth map. This depth optimization module includes an open-source pre-trained depth estimation visual foundation model, DepthAnythingV2, and a prompt network consisting of three convolutional layers. The visual foundation model, DepthAnythingV2, consists of a ViT encoder (Vision Transformer) and a DPT decoder (DensePrediction Transformer). The ViT encoder encodes high-level global information from the image, and the DPT decoder infers the image depth from the encoded high-level global information. The depth optimization module uses the ViT encoder to extract high-level global information from the left infrared image, uses the prompt network to encode the initial depth map into geometric prompt features, and injects them into the DPT decoder. The DPT decoder, based on the high-level global information extracted by the ViT encoder and the geometric prompts of the initial depth, infers a more accurate final depth map, achieving single-frame structured light 3D imaging. The structure of the depth optimization module is as follows: Figure 4 As shown.

[0064] Step C: Extend the depth estimation visual base model DepthAnythingV2.

[0065] A depth estimation visual foundation model, DepthAnythingV2, pre-trained on massive images and fine-tuned on structured light data, is used as the backbone network for the depth optimization module. The DepthAnythingV2 model comprises a Transformer-based image ViT (Vision Transformer) encoder and a DPT (Dense Prediction Transformer) decoder. The ViT encoder takes the left infrared image as input and extracts high-level global information, such as semantics, from it. This information is then expanded at the decoder, which simultaneously receives the intermediate layer encoding from the ViT encoder and the initial depth map as a prompt, thus repairing the initial depth from the previous stage. There may be low-quality areas in the image due to missing or damaged binocular information.

[0066] Step D: Input the left infrared image for encoding and inject the initial depth map.

[0067] The left infrared image is input into the ViT encoder to extract its high-level global information, and the intermediate layer encoding of the ViT encoder is extracted as input to the DPT decoder.

[0068] The initial depth map generated in the second stage is used as a strong geometric cue, which is then encoded into geometric cue features through a prompt network.

[0069] The geometric cue features obtained from the intermediate layer encoding of the ViT encoder and the initial depth map encoding are input into the DPT decoder.

[0070] Step G: The DPT decoder outputs the final depth map.

[0071] The DPT decoder combines high-level global information from the left infrared image of the ViT encoder with geometric cue features from the initial depth. Leveraging the powerful scene structure understanding capabilities of the DepthAnythingV2 depth estimation visual foundation model, it optimizes the initial depth map, restoring fine structures, correcting mismatches, and filling holes. The final output is a metrically accurate, structurally complete, and detail-rich final depth map. .

[0072] In practice, to effectively train the neural network model in this invention, we first constructed a large-scale, high-fidelity synthetic structured light dataset. This process used Blender software and its Cycles ray tracing rendering engine. We procedurally generated 2860 indoor scenes, collecting over 5000 3D objects, 103 materials, and 100 high dynamic range environment maps. These materials cover a variety of categories, including plastic, metal, leather, ceramic, and glass, and possess diverse and complex properties such as diffuse reflection, specular reflection, and transparency. During rendering, we randomized the scene layout, object poses, object materials, camera parameters, lighting conditions, and projection patterns. The projection pattern library contains eight different types of patterns (such as dot matrix and stripes). Using this method, approximately 953,000 training samples were generated. Each sample set includes a binocular RGB image, a binocular infrared image, a projection pattern, a true depth map, and a true disparity map calculated from the true depth map based on triangulation principles.

[0073] The implementation of the neural feature matching module is based on the RAFT-Stereo architecture (Lahav Lipson et al. RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching. 3DV2021), including the structured light cost body construction method proposed in this invention, feature encoder, context encoder, and recurrent neural network based on convolutional gated recurrent unit (GRU module).

[0074] The neural feature matching module includes a feature encoder, which is a convolutional neural network that downsamples the input image to 1 / 4 resolution and outputs a 256-dimensional feature map. The context encoder is also a convolutional neural network, used to extract context features at 1 / 4, 1 / 8, and 1 / 16 resolutions, for a total of three context features, which are used by the convolutionally gated recurrent units (RNUs) in the recurrent neural network. The RNU consists of three sub-modules, each receiving the 1 / 4, 1 / 8, and 1 / 16 resolution context features from the context encoder, respectively. In each iteration of the RNU, the sub-module receiving the 1 / 4 resolution context features uses the output of the sub-module receiving the 1 / 8 resolution context features from the previous iteration as additional input; the sub-module receiving the 1 / 8 resolution context features uses the outputs of all other two sub-modules from the previous iteration as additional input; and the sub-module receiving the 1 / 16 resolution context features does not receive any additional input.

[0075] Model Training: The neural feature matching module was trained independently, with all training settings consistent with those used in Raft-Stereo (Lahav Lipson et al. RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching. 3DV2021). The training iterations totaled 200,000 steps. The AdamW optimizer was used with a learning rate of 1×10⁻⁴ and a batch size of 8. The recurrent neural network's iteration update steps were set to 32 during training. The loss function was the L1 norm distance between the disparity map predicted by the neural feature matching module and the true disparity map.

[0076] 3. Implementation of the deep optimization module

[0077] This invention is the first to propose using the initial depth map obtained after structured light feature matching (i.e., the output of the neural matching module mentioned above) as a geometric cue, which is then injected into the visual foundation model for depth estimation to further optimize the quality of depth estimation. This module is based on the DepthAnythingV2 (Lihe Yang et al. Depth Anything V2. NeurIPS 2024) architecture.

[0078] The backbone network of the depth optimization module consists of a Transformer-based visual encoder (ViT) and a convolutional neural network-based DPT decoder. The ViT encoder uses a ViT-Base configuration, containing 12 Transformer layers with an embedded feature dimension of 768. The intermediate features output from the 3rd, 6th, 9th, and 12th transformer layers of this encoder are input into the DPT decoder. The initial depth map output from the neural feature matching module is encoded by a simple CNN cue network containing only three convolutional layers, transforming it into cue features, which are then injected into the DPT decoder as geometric cues to guide the final depth inference, resulting in the final depth map.

[0079] Fine-tuning Training: After the neural feature matching module is trained, its parameters (Phase 1) are frozen, and fine-tuning is performed only on this module for a total of 100,000 iterations. The deep optimization module is trained using the PromptDA (Haotong Lin et al., Prompting Depth Anything for 4K Resolution Accurate MetricDepth Estimation, CVPR 2025) training settings. The deep optimization module is initialized with publicly available DepthAnythingV2 pre-trained weights and fine-tuned based on these weights. During fine-tuning, the ViT encoder uses a smaller learning rate (5×10−6), while the DPT decoder and cue network use a larger learning rate (5×10−5). The loss function is the same as defined in PromptDA.

[0080] Through the above embodiments, the present invention can efficiently and accurately reconstruct high-quality three-dimensional depth information from a single frame structured light image, thereby realizing single-frame structured light three-dimensional imaging.

[0081] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A single-frame structured light 3D imaging method based on neural feature decoding, characterized in that, Includes the following steps: A) Construct a simulated structured light environment, perform physical simulation of structured light, and generate a large-scale synthetic structured light dataset; B) Construct a neural feature matching module to obtain an initial depth map through neural feature matching; including: B1: Receives a single-frame structured light projection pattern and a structured light infrared image captured by an infrared camera from external input, and performs multi-source neural feature extraction; the infrared camera can be configured monocularly or binocularly; the multi-source neural features include local neural features and contextual neural features; B2: By calculating the correlation between local neural feature maps of two different images, and simultaneously constructing multi-scale structured light cost volumes between the left infrared camera image and the structured light projection pattern, and / or between the left infrared camera image and the right infrared camera image, a multi-level cost volume pyramid is formed. B3: Iterative optimization generates the initial depth map; Based on a recurrent neural network, disparity is iteratively predicted and optimized according to the cost volume and contextual neural features to obtain a disparity map, and an initial depth map is calculated. C) Design a depth optimization module that uses the initial depth map as a geometric cue and injects it into the visual base model. Based on the initial depth map, further optimize it to generate the final depth map. The depth optimization module only uses the left infrared image. The depth optimization module includes: a pre-trained depth estimation visual base model and a cueing network consisting of multiple convolutional layers; The visual foundation model includes a ViT encoder and a DPT decoder. The ViT encoder encodes high-level global information from the image, and the DPT decoder infers the image depth from the encoded high-level global information. A cueing network is used to encode the initial depth map into geometric cue features and inject them into the DPT decoder. The DPT decoder infers a high-precision final depth map based on the high-level global information extracted by the ViT encoder and the geometric cue features of the initial depth, realizing single-frame structured light 3D imaging. The visual base model uses DepthAnythingV2; the design of the depth optimization module includes the following steps: c1 extends the depth estimation visual base model DepthAnythingV2; the depth estimation visual base model DepthAnythingV2, which is pre-trained on massive images and fine-tuned on structured light data, is used as the backbone network of the depth optimization module. The DepthAnythingV2 visual base model for depth estimation includes a Transformer-based image ViT encoder and a DPT decoder. The ViT encoder is used to extract semantic high-level global information from the image. The decoder is extended by receiving the intermediate layer encoding of the ViT encoder and the initial depth map as a cue, and repairing low-quality regions in the initial depth map. c2, Input the left infrared image for encoding and inject the initial depth map; including: inputting the left infrared image into the ViT encoder, extracting the output features of the intermediate layer of the ViT encoder, and inputting them into the DPT decoder; The initial depth map is used as a strong geometric cue and encoded into geometric cue features through a cue network. The intermediate layer features extracted from the intermediate layer of the ViT encoder and the cue features obtained from the initial depth map encoding are input into the DPT decoder; c3, the DPT decoder outputs the final depth map; The DPT decoder fuses monocular visual features from the infrared image obtained from the ViT encoder with geometric cue features from the initial depth map to optimize the initial depth map and output the final depth map.

2. The single-frame structured light 3D imaging method based on neural feature decoding as described in claim 1, characterized in that, Step B1 extracts multi-source neural features. Specifically, based on the input single-frame projection pattern image, left infrared image and / or right infrared image in binocular configuration, the feature encoder in the neural feature matching module extracts high-dimensional neural feature maps from the left infrared image and projection pattern image respectively. For local neural features, in monocular configuration, the same feature encoder is used to extract local neural features from the left infrared image and the structured light projection pattern; in binocular configuration, while using the same feature encoder as in monocular configuration to extract local neural features from the left infrared image and the structured light pattern, another additional feature encoder with the same structure but different parameters is used to extract a second set of neural feature maps from the left and right infrared images respectively. For contextual neural features, regardless of whether it is a monocular or binocular configuration, the same context encoder is used to extract multi-scale contextual neural features from the left infrared image for subsequent iterative optimization.

3. The single-frame structured light 3D imaging method based on neural feature decoding as described in claim 1, characterized in that, In the multi-scale structured light cost volume constructed in step B2, the structured light cost volume encodes the dense matching relationship between the projection pattern and the captured image. The elements in the structured light cost volume represent the similarity of local neural features between the left infrared image and the structured light projection pattern, or the similarity of local neural features between the left infrared image and the right infrared image.

4. The single-frame structured light 3D imaging method based on neural feature decoding as described in claim 3, characterized in that, Specifically, multi-level average pooling is performed on the three-dimensional cost volume formed by the local neural feature map of the left infrared image and the local neural feature map of the structured light projection pattern. The average pooling is performed four times along the last dimension of the three-dimensional cost volume. Each average pooling operation reduces the dimension of the last dimension to half of the value before pooling. The three-dimensional cost volume before and after each pooling is saved to construct a four-level cost volume pyramid to capture multi-scale information. For binocular structured light, the same operation is performed on the second three-dimensional cost volume formed by the local neural feature map of the left infrared image and the local neural feature map of the right infrared image.

5. The single-frame structured light 3D imaging method based on neural feature decoding as described in claim 1, characterized in that, In step B3, the initial depth map is generated through iterative optimization. A recurrent neural network based on convolutional gated recurrent units is used to iteratively predict and optimize the disparity to obtain a disparity map. The initial depth map is then calculated using the principle of triangulation.

6. The single-frame structured light three-dimensional imaging method based on neural feature decoding as described in claim 5, characterized in that, Specifically, the predicted disparity map is initialized as a disparity map with all zeros. In each iteration, based on the current disparity estimate, relevant features are sampled from the cost volume pyramid and combined with contextual neural features. These features are then input into the convolutional gated recurrent unit to update the hidden state, resulting in a new hidden state. The updated hidden state outputs the disparity difference through the output head of the convolutional gated recurrent unit, and this disparity difference is added to the current disparity estimate to obtain the updated disparity estimate.

7. A single-frame structured light three-dimensional imaging system based on neural feature decoding, implemented using the method of claim 1, characterized in that, include: The module consists of a data generation module, a neural feature matching module, and a depth optimization module; among which: The data generation module is used to generate large-scale synthetic structured light datasets through physical simulation; The neural feature matching module is used to match the input single-frame projection pattern with the scene infrared image captured by the camera in the feature space, and generate an initial depth map through iterative optimization. The depth optimization module receives the initial depth map as a geometric cue, combines it with high-level visual features from the scene's infrared image, and optimizes it using a pre-trained depth estimation visual base model fine-tuned with structured light data to generate the final depth map.

8. The single-frame structured light 3D imaging system based on neural feature decoding as described in claim 7, characterized in that, The dataset contains diverse scenes, lighting, materials, and projection patterns, and provides corresponding true depth; the neural feature matching module is used to match the input single-frame projection pattern with the scene infrared image captured by at least one camera within the feature space.

Images