Hyperspectral reconstruction feature enhanced binocular stereo matching depth estimation method and system

Abstract

The application discloses a kind of binocular stereo matching depth estimation method and system of hyperspectral reconstruction feature enhancement.Method includes the following steps:S1: receiving left, right binocular RGB image;S2: using multi-level spectral transformation module is reconstructed to high-dimensional spectral feature to the left, right binocular RGB image, through learning the spectral reflectivity prior of material, output multi-channel hyperspectral feature map covering visible light band;S3: through learnable spectral dimension reduction layer, adaptive channel selection and feature reorganization are carried out to multi-channel hyperspectral feature map, and left, right pseudo-color feature map that the physical property difference of material is strengthened is generated;S4: using stereo matching module, based on left, right pseudo-color feature map carries out geometric feature extraction and disparity iteration optimization, and outputs the final dense disparity map.The application learns optical physics prior through model training, so that low-cost RGB hardware can reproduce high-fidelity spectral features, effectively solve the depth perception problem under complex material and extreme light.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of machine vision and artificial intelligence, and relates to a binocular stereo matching method and system based on latent spectral feature enhancement. This invention does not require a hyperspectral camera during the inference stage, but rather utilizes spectral priors through a "reconstruction-perception-dimensionality reduction" mechanism. Background Technology

[0002] Binocular stereo vision technology, as a core means of machine vision for perceiving the three-dimensional world, is based on the principle of mimicking human eyes. It utilizes triangulation to recover the depth information of a scene by calculating the disparity between corresponding pixels in the left and right camera views. With the rapid development of deep learning technology, end-to-end stereo matching methods based on convolutional neural networks (CNNs) and Transformer architectures (such as GC-Net, RAFT-Stereo, and IGEV-Stereo) have gradually replaced traditional hand-crafted feature methods, becoming the mainstream technology. These methods have achieved significant accuracy breakthroughs on standard datasets such as Middlebury and KITTI by constructing cost volumes containing rich contextual information and performing cost aggregation and regression.

[0003] Although existing deep learning stereo matching models perform well in scenes with clear geometric structures, they still suffer from an inherent flaw determined by the physical characteristics of RGB sensors in real-world open-world applications, causing them to frequently fail in certain scenarios:

[0004] Matching ambiguities caused by metamerism: Most existing stereo matching networks rely solely on RGB three-channel color information as feature input. However, standard RGB cameras can only capture a wide range of red, green, and blue information, losing continuous spectral features in the scene. This causes stereo matching algorithms to be unable to distinguish material differences based on color when faced with metameristic targets (such as simulated vegetation and real vegetation). Especially under strong reflections or specular reflections, pseudo-textures produced by simulated materials (such as plastic) severely interfere with parallax calculations, resulting in holes in the depth map. Current technology lacks a means to extract the "physical fingerprint" of objects under passive imaging conditions and use it to correct geometric ambiguities.

[0005] To overcome the limitations of RGB information, hyperspectral imaging (HSI) is widely recognized as an ideal means of solving the material differentiation problem because it can acquire fine spectral reflectance of objects in tens to hundreds of wavebands. Although a few cutting-edge studies have attempted to use physical hyperspectral cameras to build stereo vision systems to resolve matching ambiguities, the high hardware cost, large size, and low spatial resolution of physical spectroscopic devices (such as prisms and LCTFs) mean that such hyperspectral stereo matching systems are currently only in the laboratory exploration stage and cannot meet the stringent requirements of cost, portability, and real-time performance in large-scale applications such as autonomous driving and consumer electronics.

[0006] In recent years, spectral reconstruction (SR) technology in computational optics has made groundbreaking progress, providing a new solution to the aforementioned contradictions. This technology demonstrates that by learning nonlinear mappings through deep neural networks (such as MST++ and HSCNN-R), high-dimensional spectral information can be reconstructed with high fidelity from ordinary RGB images. Successful applications in related fields fully confirm that RGB images contain key information at the level of physical properties.

[0007] 1) In the biomedical field: Research shows that by reconstructing the spectrum from RGB images of ordinary endoscopes or fundus cameras, the algorithm can accurately distinguish between early cancerous tissues (such as melanoma and gastrointestinal lesions) that are not visible to the naked eye and healthy tissues. The essence of this is to utilize the high sensitivity of the reconstructed spectrum to biological components such as hemoglobin and melanin.

[0008] 2) In the field of remote sensing and security: For camouflaged targets that are difficult to distinguish in RGB images, the algorithm based on spectral reconstruction has successfully achieved pixel-level classification of artificial camouflage nets and natural vegetation, proving the unique advantages of spectral features in solving the problem of "metachromatic".

[0009] 3) In the field of food safety: Low-cost RGB sensors combined with spectral reconstruction algorithms have been successfully used to detect the freshness of meat (such as histamine and glutamic acid content) and the internal quality of fruit, achieving effective identification of targets that are similar in appearance but different in biochemical state.

[0010] These cross-disciplinary prior knowledge provide important insights: ordinary RGB images actually contain subtle spectral differences that can be recovered through computation, and these differences have great potential value in resolving disparity ambiguity in macroscopic scenes.

[0011] Although spectral reconstruction techniques have proven effective in classification and segmentation tasks, a mature method for effectively incorporating spectral reconstruction priors into geometric depth calculations is still lacking in the field of binocular stereo matching. Currently, two main technical obstacles exist:

[0012] The lack of an effective transfer and fusion mechanism for cross-domain prior knowledge: Existing stereo matching network architectures mainly focus on extracting geometric structural features from the RGB three channels, without considering the input of high-dimensional spectral features. Although spectral reconstruction technology has shown initial success in semantic-level tasks such as material classification and target recognition, how to effectively transfer this "spectral prior" based on material properties to the geometric depth calculation of high-precision pixel-level matching, and design an adaptive feature fusion mechanism that can both enhance the identification of "metachromatic" regions and avoid the curse of dimensionality in high-dimensional data, is a pressing problem that needs to be solved.

[0013] The lack of high-quality multimodal datasets: Training an end-to-end network for joint RGB-spectral-depth perception requires a large amount of pixel-aligned training data. Existing mainstream public datasets (such as SceneFlow and KITTI) only provide RGB and disparity ground truth values, lacking corresponding real hyperspectral data; while existing spectral datasets often lack high-precision depth ground truth values. This data scarcity directly limits research and algorithm validation in this direction.

[0014] In summary, the industry urgently needs a method that can leverage prior knowledge from spectral reconstruction to enhance the ability to distinguish the materials of macroscopic objects, achieve high-precision depth estimation based on low-cost RGB hardware, and provide a corresponding high-precision training data construction scheme. Summary of the Invention

[0015] To overcome the shortcomings of existing technologies, this invention provides a binocular stereo matching depth estimation method and system with enhanced hyperspectral reconstruction features. This invention learns photophysical priors through model training, enabling low-cost RGB hardware to reproduce high-fidelity spectral features, effectively solving the depth perception challenge under complex materials and extreme lighting conditions.

[0016] The technical solution for achieving the technical objective of this invention is as follows:

[0017] A stereo matching depth estimation method with hyperspectral reconstruction feature enhancement includes the following steps:

[0018] S1: Receive left and right binocular RGB images;

[0019] S2: Use a multi-level spectral transformation module to reconstruct high-dimensional spectral features of the left and right binocular RGB images, and output a multi-channel hyperspectral feature map covering the visible light band by learning the spectral reflectance prior of the material.

[0020] S3: Adaptive channel selection and feature recombination are performed on the multi-channel hyperspectral feature map through a learnable spectral dimensionality reduction layer to generate left and right pseudo-color feature maps that enhance the differences in material physical properties;

[0021] S4: Using a stereo matching network, stereo matching is performed based on the left and right pseudo-color feature maps to output the final dense disparity map.

[0022] The multi-level spectral transformation network is an MST++ network, and the stereo matching network is an IGEV-Stereo network.

[0023] The learnable spectral dimensionality reduction layer uses the Gumbel-Softmax mechanism to achieve differentiable channel selection, employing soft sampling during the training phase and hard selection during the inference phase.

[0024] The weight calculation formula for the Gumbel-Softmax mechanism is as follows: , Where, π i (k) For learnable nonnormalized probability parameters, g i For noise samples that follow a Gumbel(0,1) distribution, τ is a temperature coefficient used to control the smoothness of the distribution.

[0025] The initial weights of the learnable spectral dimensionality reduction layer are set based on the Gaussian spectral response model, wherein the center wavelengths of the Gaussian responses of the red, green, and blue channels are 660nm, 570nm, and 490nm, respectively, and the bandwidths are 40nm, 50nm, and 40nm, respectively.

[0026] The method described above uses sparse depth point clouds obtained based on physical common optical paths as hard constraints during the training phase, and combines them with a pre-trained basic stereo matching model for fine-tuning to generate dense depth pseudo-real values.

[0027] The method described above, the method for constructing a dense depth dataset, includes:

[0028] a1: Obtain a sparse depth map through physical scanning as a hard constraint;

[0029] a2: Using a pre-trained basic stereo matching large model, scene-adaptive fine-tuning is performed using the sparse depth map and RGB image;

[0030] a3: Outputs the dense depth pseudo-values ​​and pairs them with the corresponding RGB and spectral images to form a multimodal dataset.

[0031] The pre-trained basic stereo matching large model is a basic stereo matching large model network pre-trained on a large-scale stereo matching dataset.

[0032] A multimodal data acquisition system includes: a central main camera and a monochrome industrial camera, which achieve common aperture imaging through a front beam splitter prism; a liquid crystal tunable filter (LCTF) and a spectral imaging lens for acquiring multi-band spectral images; a second industrial camera and the central main camera forming a binocular stereo vision baseline; a line laser projector and a Sham imaging camera, wherein the imaging plane, depth receiving imaging lens plane, and laser plane of the Sham imaging camera satisfy Sham's law and are equipped with a narrowband filter; and a precision electric rotary stage for driving the system to perform rotary scanning.

[0033] The system described above can achieve depth estimation using only a standard RGB industrial camera during the actual inference phase, without relying on hyperspectral imaging equipment or laser hardware.

[0034] The beneficial effects of this invention are:

[0035] A high-quality multimodal data generation system based on the "sparse physical truth fine-tuning basic model" was constructed (data and hardware advantages). This invention proposes a dense truth construction scheme of "sparse supervision of physical measurement + fine-tuning of basic model".

[0036] 1) At the acquisition end: The strategy of "physical common optical path + algorithm fine-tuning" is adopted. The main parallax is eliminated by using a beam splitter prism and LCTF, and the pixel-level strict alignment of spectrum and texture is achieved by combining image registration algorithm. At the same time, the Sham structured light system and narrowband filtering technology are used to obtain high-precision semi-compact laser point cloud while ensuring high signal-to-noise ratio.

[0037] 2) On the generation side: To address the occlusion sparsity inherent in physical scanning, the aforementioned point cloud was innovatively used as a sparse hard constraint to fine-tune the pre-trained Foundation Stereo Model. The generalization ability of the large model was utilized to complete the hole regions in a physically consistent manner, ultimately generating structurally complete, high-fidelity Dense Depth Pseudo-GT, providing a high-quality data foundation for network training.

[0038] This invention improves stereo matching performance in metamerism and weakly textured scenes by proposing an end-to-end joint sensing framework of "spectral prior reconstruction - adaptive dimensionality reduction - geometric matching".

[0039] Spectral prior injection: The MST++ network is used to recover potential high-dimensional spectral features from RGB images. These features contain subtle band information that is lost in traditional RGB images and is sensitive to material properties. This effectively alleviates matching ambiguities in metamerism and color confusion regions and improves the accuracy of feature matching.

[0040] Adaptive Feature Fusion: A learnable spectral reduction layer was designed to adaptively select spectral channel combinations (Pseudo-RGB) that contribute significantly to the geometric matching task. This mechanism enhances the discriminative power of spectral textures while avoiding the curse of dimensionality in high-dimensional data. Experiments demonstrate that this algorithm improves the depth recovery integrity of macroscopic objects in weakly textured regions and enhances the sharpness of edge structures. Attached Figure Description

[0041] Figure 1 This is a top view of the multimodal data acquisition system of the present invention, which mainly shows the overall layout and optical path of the spectral-visual common aperture imaging unit, the binocular RGB imaging branch and the active line scanning depth measurement module.

[0042] Figure 2 This is a side view structural diagram of the multimodal data acquisition system of the present invention, which mainly shows the assembly relationship of the spectral acquisition branch, the RGB texture acquisition branch and the bottom precision electric rotary stage composed of a liquid crystal tunable filter (LCTF).

[0043] Figure 3 This is a schematic diagram of the end-to-end depth estimation network architecture proposed in this invention, illustrating the process from binocular RGB input to spectral prior reconstruction, adaptive dimensionality reduction, and disparity optimization.

[0044] Figure 4 This is a comparison chart of the depth estimation effects of the method of the present invention and the comparative scheme on a small boat model, used to demonstrate the gain of the present invention in the reconstruction of complex edge structures;

[0045] Figure 5 This is a detail image of depth recovery in the shadowed area of ​​the leaf using the method of the present invention, used to demonstrate the robustness of the present invention in areas with weak texture.

[0046] Figure 6 This is a comparison chart of spectral fingerprint recognition and reflectance curves for real and fake leaf materials, demonstrating the supporting role of reconstructed spectral features in material identification.

[0047] In the figure, the following components are included: depth receiving camera 1, central main camera 2, second industrial camera 3, depth receiving imaging lens 4, narrowband filter 5, included angle 6, central imaging lens 7, second imaging lens 8, beam splitter 9, depth scanning system baseline 10, binocular stereo vision baseline 11, line laser projector 12, liquid crystal tunable filter (LCTF) 13, spectral imaging lens 14, monochrome industrial camera 15, optical breadboard 16, and precision electric rotary stage 17. Detailed Implementation

[0048] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] Example 1: Integrated Multimodal "Spectral-Geometric" Data Acquisition System

[0050] To address the lack of multimodal paired datasets in existing technologies that simultaneously contain depth information, spectral information, and RGB texture information with strict spatiotemporal alignment, this embodiment constructs an integrated acquisition system based on front-mounted beam splitting co-aperture imaging and rotating line scanning.

[0051] like Figure 1 and Figure 2 As shown, the system is integrated and mounted on an optical breadboard 16, the bottom of which is rigidly connected to a precision motorized rotary stage 17. The precision motorized rotary stage 17 is configured to drive the entire acquisition system to rotate around a vertical axis, thereby performing a rotational scan of a stationary target scene to obtain depth and spectral information across the entire field of view.

[0052] 1. Common aperture spectral-visual imaging unit

[0053] To ensure the field-of-view consistency between spectral and texture data at the physical level, this system adopts a front-mounted beam splitting common aperture design:

[0054] Central main imaging optical path: A first industrial camera, namely the central main camera 2, and its matching central imaging lens 7 are provided. A beam splitter cube 9 is rigidly fixed in the optical path in front of the central imaging lens. After the scene incident light enters the beam splitter, part of the light energy (e.g., 50%) is transmitted and converged to the RGB sensor of the central main camera 2 via the central imaging lens (component 7) (450nm-700nm light transmission) to acquire a high-resolution RGB texture image.

[0055] Spectral imaging optical path: The beam splitter 9 reflects another portion of the incident light energy (e.g., 50%) to the vertical direction (90-degree turn). The reflected light path passes sequentially through the liquid crystal tunable filter (LCTF) 13, the spectral imaging lens 14, and the monochrome industrial camera 15.

[0056] Spectral modulation parameters: The LCTF (model: LCTF-V10) is configured to perform stepped wavelength scanning in the 450nm-700nm visible light band, with a step size set to 10nm.

[0057] Alignment effect: This optical path structure allows the monochrome camera 15 and the central main camera 2 to image the scene through the same physical aperture (the entrance of the beam splitter prism), thereby essentially eliminating parallax between different sensors at the physical level. This provides a reliable physical benchmark for subsequent sub-pixel-level fine-tuning using image registration algorithms, ultimately achieving strict spatial registration between the spectral image sequence and the RGB texture image.

[0058] 2. A second industrial camera 3 and its second imaging lens 8 are arranged parallel to one side of the central main camera in the stereo vision baseline unit. The optical axis of this camera is parallel to that of the central main camera 2, and the two together form a standard binocular stereo vision baseline 11, which is used to provide binocular parallax information required for passive stereo matching.

[0059] 3. Sham structured light depth measurement unit

[0060] To acquire high-precision depth information of the scene while collecting spectral and texture data, this system integrates an active line scanning module:

[0061] Line laser projector 12: Located on one side below the optical axis of the system, it is used to emit a vertical line laser plane.

[0062] Field of view coincidence constraint: The spatial position of the laser projection plane is configured to coincide with the vertical field of view center line of the binocular left camera 2. This design aims to ensure that the depth information acquired by the laser scan is strictly within the field of view of the binocular left camera, avoiding invalid sampling areas that "have only depth data but no corresponding texture data", and ensuring the one-to-one correspondence of multimodal data.

[0063] Scheimpflug Camera: Includes a depth receiving camera 1, a depth receiving lens 4, and a line-scan laser narrowband filter 5.

[0064] Scheimpflug Path Layout: In order to solve the defocusing blur problem when the line laser scans within a large depth of field, the imaging plane of the depth receiving camera 1, the plane of the depth receiving imaging lens 4, and the laser projection plane (emission surface of the line laser projector 12) are configured to satisfy the geometric constraints of Scheimpflug Principle (angle 6, depth scanning system baseline 10), thereby ensuring that the laser stripes can be clearly imaged throughout the entire measurement depth of field.

[0065] Narrowband filtering: The narrowband filter 5 is installed in front of the depth receiving imaging lens 4, and its center wavelength is strictly matched with the line laser wavelength. It is used to filter out ambient stray light and improve the signal-to-noise ratio of laser stripe extraction.

[0066] Example 2: Method for constructing a multimodal dense depth dataset

[0067] Based on the integrated hardware system described in Embodiment 1, this embodiment proposes a data production method of "rotational scanning-sparse reconstruction-large model fine-tuning," aiming to solve the problem of data voids in areas with low reflectivity and occlusion in single physical scanning, and to construct high-quality dense depth ground truth. The specific steps are as follows:

[0068] Step S1:

[0069] Time-series multimodal data acquisition: The target object is placed at the designated observation position of the acquisition system, ensuring that the object is centered in the field of view of the main camera. The control system performs acquisition according to the following timing sequence to ensure that the data for each modality are based on the same static scene reference:

[0070] Static texture acquisition: Keep the turntable and the object under test relatively still, and simultaneously expose the main camera and the second industrial camera at the control center to acquire a set of binocular RGB images from the current viewpoint.

[0071] Static staring spectral acquisition: Keep the rotating stage and the object under test relatively still, and enter the LCTF staring imaging mode: control the liquid crystal tunable filter (LCTF) to perform step wavelength modulation in the 450nm-700nm band, with a step size set to 10nm; at each specific center wavelength, trigger the monochrome industrial camera to capture a full-field single-band image; repeat the above process until a complete 26-channel spectral image sequence is obtained.

[0072] Dynamic Depth Scan Acquisition: After completing spectral and texture acquisition, the line laser projector is activated. A precision motorized rotary stage is controlled to rotate the acquisition system within the field of view of the central main camera. During the scanning process, the SAM imaging camera continuously acquires a sequence of laser stripe images projected onto the object's surface.

[0073] Step S2: Physical Reconstruction of Sparse Point Cloud Based on Joint Calibration

[0074] The laser stripe image sequence acquired in step S1 is processed using pre-obtained joint calibration parameters of the structured light system and camera.

[0075] Stripe center extraction: Perform subpixel-level processing on each frame of the image to extract the center coordinates of the laser stripes;

[0076] 3D spatial back projection: Based on the principle of triangulation, combined with the calibrated line laser plane equation and SAM camera intrinsic parameters, the two-dimensional fringe coordinates are back projected into three-dimensional space to calculate the three-dimensional point cloud of the current scanning section;

[0077] Coordinate unification and projection: Based on pre-determined joint calibration parameters and the kinematic parameters of the rotary table, the point clouds of all scanned sections are uniformly transformed to the world coordinate system of the central main camera (binocular left camera). The joint calibration parameters are obtained by capturing images of a checkerboard calibration board using the binocular left camera and simultaneously acquiring the physical point cloud data of the corresponding checkerboard grid using a line-scan laser system, thereby determining the spatial pose transformation relationship between the laser system and the camera. After completing coordinate system one, the 3D point cloud is reprojected onto the image plane of the central main camera to generate a sparse depth map that strictly matches the field of view of the RGB image.

[0078] Note: This sparse depth map has physical measurement accuracy in high reflectivity regions accessible by lasers, but data is missing in self-occluding areas, low reflectivity regions, or optical blind spots.

[0079] Step S3: Deep densification based on fine-tuning of the base model

[0080] To fill in the data gaps created by the physical scan and generate a complete true value for compact depth:

[0081] Constraint Construction: The sparse depth map generated in step S2 is used as sparse hard constraints.

[0082] Model construction: The Foundation Stereo Model, pre-trained on a large-scale dataset, is selected as the base network.

[0083] Scene-adaptive fine-tuning: The large model is fine-tuned using RGB images and sparse depth map data acquired by this system. During training, sparse physical measurements are used to supervise the geometric output of the model at corresponding pixel positions. At the same time, the semantic understanding and generalization capabilities obtained from the pre-training of the base model are used to perform physically consistent depth completion for occluded and low-reflectivity areas not covered by laser scanning.

[0084] Ground truth generation: The depth map output by the fine-tuned model is used as the final dense depth pseudo-ground truth (Dense Depth Pseudo-GT). This ground truth, together with the RGB and spectral images acquired in step S1, constitutes a multimodal pairing dataset.

[0085] Example 3: Stereo Matching Method Based on Spectral Prior Enhancement

[0086] This embodiment proposes an end-to-end depth estimation network architecture, as shown in Figure 3. The core logic of this method lies in using high-fidelity spectral data acquired by a physical acquisition system as a "strong supervision signal" to complete the initial training of the network, enabling it to recover the physical properties of materials from RGB images. Once training is complete, in the actual inference stage, the system only requires a standard RGB industrial camera to achieve material-aware depth estimation, without relying on expensive hyperspectral imaging equipment. The specific processing flow is as follows:

[0087] Network model architecture module A: Spectral prior reconstruction module (MST++)

[0088] Input: Receive the left view I from the stereo RGB image. L With right view I R .

[0089] Processing: A multi-level spectral transformation network (MST++) is used as the backbone to map the 3-channel RGB image to a high-dimensional spectral feature space.

[0090] Output: 26-channel hyperspectral feature maps covering the 450nm-700nm wavelength range. Experimental verification shows that the module achieved a peak signal-to-noise ratio (PSNR) of 39.37 dB on the test set, demonstrating the system's ability to faithfully reproduce subtle spectral differences in objects. This high-precision spectral recovery capability enables the system to identify the unique 'red-edge effect' of biological materials (such as true leaves) and the flat spectral features of synthetic materials (such as false leaves), thus providing strong physical semantic support for subsequent stereo matching. Figure 6 As shown.

[0091] Module B: Learnable Spectral Dimensionality Reduction Layer Function

[0092] To select the features that contribute most to geometric matching from the 26 spectral channels, this embodiment designs a differentiable channel selection mechanism based on Gumbel-Softmax. Processing logic: Define the selection weight w of the output channel k (k∈{R,G,B}) on the input spectral channel i (i∈{1,…,26}). i (k)During the training phase, the Gumbel-Softmax technique is used to achieve differentiable "soft sampling," as shown in the following formula:

[0093] , Where, π i (k) For learnable nonnormalized probability parameters, g i For noise samples that follow a Gumbel(0,1) distribution, τ is the temperature coefficient used to control the smoothness of the distribution.

[0094] Feature Reconstruction: Output Pseudo-color Feature Map F out The calculation is as follows:

[0095] .

[0096] Inference phase optimization: During the inference phase, the argmax operation is used directly to perform "hard selection" to select the channel combination with the highest probability, thus avoiding computational redundancy.

[0097] Module C: Stereo Matching Module (IGEV-Stereo)

[0098] Input: Receive the left and right pseudo-color feature maps processed by module B. .

[0099] Processing: An IGEV-Stereo network is used to extract geometric features by constructing a Combined GeometryEncoding Volume, and GRU units are used for iterative disparity optimization.

[0100] Output: Output the final dense parallax map D pred .

[0101] Physical initialization strategy for dimensionality reduction layer weights:

[0102] To address the slow convergence issue in the early stages of training caused by random initialization and to give the dimensionality reduction layer a more explicit physical meaning, this embodiment preferably adopts a weight setting method based on the Gaussian Spectral Response Model during the network initialization phase.

[0103] Principle: Simulating the light-sensing characteristics of a standard RGB camera, the spectral response curves of the red (R), green (G), and blue (B) channels are modeled as Gaussian distributions.

[0104] , Parameter settings:

[0105] .

[0106] Implementation method:

[0107] The theoretical response values ​​for 26 discrete wavelengths (450-700nm) are calculated using the above formula and directly assigned to the weights of the 1×1 convolutional layer in module B as the initial state. This strategy enables the network to possess spectral compression capabilities similar to those of a physical camera from the initial training stage, thereby accelerating the subsequent fine-tuning and convergence process.

[0108] Example 4: Algorithm Performance Verification and Comparative Analysis

[0109] To verify the beneficial effects of the stereo matching method based on latent spectral feature enhancement proposed in this invention, a high-precision multimodal dataset collected and constructed using the aforementioned physical system was used for training. This embodiment selected a small boat model with rich surface curvature and fine edge features, and a vegetation scene containing a comparison of real and fake materials for depth verification. The classic stereo matching network without a spectral prior module (i.e., the original IGEV-Stereo with only RGB images as input) was used as a baseline method for qualitative and quantitative comparison.

[0110] The beneficial effects of deep recovery are analyzed in Table 1 below.

[0111] Global average error (EPE) 0.4329 0.4192 Decrease by 3.2% Large parallax area error (EPE > 120px) 0.9656 0.9134 Decrease of 5.4%

[0112] The method of this invention shows certain improvement effects under different evaluation indicators:

[0113] Global accuracy optimization: In terms of global average error, the method of this invention achieves an error reduction of approximately 3.2% compared to existing technologies. This indicates that by introducing latent spectral features, the system improves the accuracy of reconstructing the overall geometric structure of the scene.

[0114] Optimization for close-range and complex material regions: In the high parallax region (EPE > 120px) that reflects the ability to detect close-range targets, the method of this invention reduces the error from 0.9656 to 0.9134, an improvement of approximately 5.4%.

[0115] Qualitative visualization analysis: combined with appendix Figure 4 Appendix Figure 5 Appendix Figure 6 The specific analysis is as follows:

[0116] 1. Fine reconstruction of complex geometric structures (small boat models): For small boat models with slender components, existing technologies are prone to parallax adhesion or edge breakage when processing small structures such as masts due to insufficient discrimination power of RGB features at narrow edges.

[0117] Edge sharpness: such as Figure 4 As shown, in the depth map generated by this invention, the edges of the ship's mast and its auxiliary components are clear, sharp, and structurally complete, which is superior to the benchmark model.

[0118] Hyperspectral reconstruction performance and physical property analysis: For the spectral reconstruction module A (MST++), the average PSNR measured on the test set reached 39.37 dB. This high-fidelity spectral reconstruction capability provides the system with an accurate material physical fingerprint.

[0119] Material fingerprint recognition: As shown in Figure 6, relying on a reconstruction accuracy of 39.37dB, the system can accurately capture the "red-edge effect" unique to real blades in the 700nm-740nm band. In contrast, the reflection curve of simulated blades (fake leaves) in this band shows a smooth or decreasing characteristic.

[0120] 2. Dark Detail Restoration with Material Fingerprint Assistance: For biological leaf scenes, the experiment focused on examining the model's performance in complex lighting and backlit dark areas.

[0121] Effect comparison: Even in shadow areas where visible light information is weak, such as Figure 5 The present invention can still accurately identify the material boundaries of the blade and restore a more continuous, smooth and detailed blade geometry.

[0122] In summary, this embodiment demonstrates that the present invention, through the technical means of "reconstruction-perception-dimensionality reduction," not only achieves a breakthrough in quantitative accuracy in global and large parallax regions, but also exhibits superior effects compared to existing technologies in terms of edge preservation of fine structures (such as masts), restoration of dark details, and correction of geometric distortion. The above descriptions are merely preferred embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A binocular stereo matching depth estimation method with hyperspectral reconstruction feature enhancement, characterized in that, Includes the following steps: S1: Receive left and right binocular RGB images; S2: High-dimensional spectral feature reconstruction is performed on the left and right binocular RGB images using a multi-level spectral transformation network. By learning the spectral reflectance prior of the material, a multi-channel hyperspectral feature map covering the visible light band is output. S3: Adaptive channel selection and feature recombination are performed on the multi-channel hyperspectral feature map through a learnable spectral dimensionality reduction layer to generate left and right pseudo-color feature maps that enhance the differences in material physical properties; S4: Using a stereo matching network, geometric feature extraction and disparity iterative optimization are performed based on the left and right pseudo-color feature maps to output the final dense disparity map; The multi-level spectral transformation network is an MST++ network, and the stereo matching network is an IGEV-Stereo network; The learnable spectral dimensionality reduction layer uses the Gumbel-Softmax mechanism to achieve differentiable channel selection, employing soft sampling during the training phase and hard selection during the inference phase. The method described above uses sparse depth point clouds obtained based on physical common optical paths as hard constraints during the training phase. It is then fine-tuned in combination with a pre-trained basic stereo matching model and utilizes the semantic generalization ability of the large model to perform depth completion on the missing regions of the physical scan in order to generate dense depth pseudo-real values.

2. The method according to claim 1, characterized in that, The weight calculation formula for the Gumbel-Softmax mechanism is as follows: ， in, π i (k) For learnable nonnormalized probability parameters, g i For noise samples that follow a Gumbel(0,1) distribution, τ This is a temperature coefficient used to control the smoothness of the distribution.

3. The method according to claim 1, characterized in that, The initial weights of the learnable spectral dimensionality reduction layer are set based on the Gaussian spectral response model, wherein the center wavelengths of the Gaussian responses of the red, green, and blue channels are 660nm, 570nm, and 490nm, respectively, and the bandwidths are 40nm, 50nm, and 40nm, respectively.

4. The method according to claim 1, characterized in that, Methods for constructing dense depth datasets include: a1: obtaining sparse depth maps that are pixel-level aligned with RGB images through physical scanning; a2: Using a pre-trained basic stereo matching large model, scene-adaptive fine-tuning is performed using the sparse depth map and RGB image; a3: Outputs the dense depth pseudo-values ​​and pairs them with the corresponding RGB and spectral images to form a multimodal dataset.

5. The method according to claim 4, characterized in that, The pre-trained basic stereo matching large model is a basic stereo matching large model network pre-trained on a large-scale stereo matching dataset.

6. A multimodal data acquisition system, characterized in that, The method based on claim 1 includes: a central main camera and a monochrome industrial camera, achieving common aperture imaging through a front beam splitter prism; a liquid crystal tunable filter (LCTF) and a spectral imaging lens for acquiring multi-band spectral images; a second industrial camera and the central main camera forming a binocular stereo vision baseline; a line laser projector and a Sham imaging camera, wherein the imaging plane, depth receiving imaging lens plane, and laser plane of the Sham imaging camera satisfy Sham's law, and are equipped with a narrowband filter; and a precision electric rotary stage for driving the system to perform rotary scanning.

7. The system according to claim 6, characterized in that, In the actual inference phase, only a standard RGB binocular camera is needed to achieve the depth estimation with enhanced material perception, without the need to call up hyperspectral imaging equipment or laser hardware.

Images