A training-free snapshot photometric stereo vision method based on spectral compressive imaging

Abstract

The application discloses a kind of training-free snapshot photometric stereo vision methods based on spectral compression imaging, comprising: (1) setting multiple different wavelength monochromatic light sources to simultaneously irradiate object from different directions, form high-dimensional spectral-angle data cube;(2) the spectral-angle data cube is modulated in space by coded aperture, along the spectrum dimension is sheared translation by using optical dispersion element, finally integrates on single-channel image sensor, generates single two-dimensional gray compressed measurement image;(3) inverse problem model is constructed, and multiple illumination image sequence is decoupled and recovered from single two-dimensional gray compressed measurement image using iterative optimization algorithm based on physical drive;(4) the surface normal map of the measured object is predicted by inputting multiple illumination image sequence into pre-trained photometric stereo vision network module.The application can realize low-cost, high data efficiency and high physical fidelity three-dimensional reconstruction by single snapshot.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, and in particular to a training-free snapshot photometric stereo vision method based on spectral compression imaging. Background Technology

[0002] Photometric Stereo (PS) is a classic technique for estimating surface normals by analyzing the intensity changes of an object under different lighting conditions.

[0003] Traditional photometric stereo (CPS) typically uses a fixed camera and a dense array of sequentially activated light sources.

[0004] For example, Chinese patent document CN107403449A discloses a visual system based on photometric stereo vision, including: a spot generator capable of projecting dense spots; at least three LEDs, each of the at least three LEDs forming a point light source; two cameras constituting a pair of stereo cameras; a control circuit that is communicatively connected to the spot generator, the at least three LEDs, and the two cameras respectively to control the spot generator to project dense spots; at least three LEDs being lit sequentially, with only one LED lit at a time; the two cameras being simultaneously triggered to acquire images of the object surface illuminated by the spot generator; and one of the two cameras being triggered to acquire images of the object surface illuminated by the LED.

[0005] While this method can reconstruct high-fidelity and high-precision normal maps, it requires capturing multiple images from different lighting directions, and the acquisition process is sequential, resulting in a long time consumption and a large amount of data. Therefore, CPS is severely limited in real-time scenarios or storage-constrained applications, and time redundancy becomes a key bottleneck.

[0006] To address these limitations, the research community has turned to the Single-image Stereo (SIS) paradigm, which aims to recover dense three-dimensional geometry from a single snapshot.

[0007] As a representative physics-based single-image stereo vision method, multispectral photometric stereo (MPS) achieves this by simultaneously illuminating the image with monochromatic light of different wavelengths and separating these channels using a multispectral camera in a single shot. Although MPS successfully compresses the temporal dimension into a single exposure, it introduces significant hardware costs (requiring expensive multispectral cameras) and low data efficiency (generating high-dimensional spectral data cubes).

[0008] On the other hand, monocular 3D reconstruction methods attempt to estimate geometry from a single standard RGB image, pursuing ultimate simplicity. These methods typically utilize depth generation priors to "illusory" geometry based on semantic context. However, they differ fundamentally from photometric stereo vision based on physical measurements of light transmission, often relying on semantic estimation rather than physically inverse rendering, resulting in an inability to recover high-frequency surface details faithful to the real physical object.

[0009] Existing technologies struggle to achieve the optimal balance between physical fidelity, data acquisition efficiency, and hardware cost. Therefore, there is an urgent need for a 3D sensing method that can maintain the physical accuracy of photometric stereo vision, achieve single-snapshot acquisition, and has low hardware cost and requires no task-specific training. Summary of the Invention

[0010] This invention provides a training-free snapshot photometric stereo vision method based on spectral compression imaging. By introducing spectral-angle encoding and compressed acquisition at the optical front end, and employing a strategy combining physical optimization and deep learning at the computational back end, it achieves efficient 3D perception.

[0011] A training-free snapshot photometric stereo vision method based on spectral compression imaging includes the following steps: (1) Set up multiple monochromatic light sources of different wavelengths to simultaneously illuminate the object from different directions. Each wavelength spectral slice corresponds to a lighting image in one direction, thereby encoding the angular lighting information of the object into the spectral dimension of the scene emissivity to form a high-dimensional spectral-angle data cube. (2) Spatial modulation of the spectral-angle data cube is performed by encoding aperture, and then shearing and translation along the spectral dimension is performed by optical dispersive element. Finally, integration is performed on a single-channel image sensor to generate a single two-dimensional grayscale compressed measurement image. (3) Based on the vectorized form of the compressed measurement image, an inverse problem model is constructed. Using a physics-driven iterative optimization algorithm, the multi-light image sequence corresponding to different lighting directions is decoupled and recovered from a single two-dimensional grayscale compressed measurement image without the need for external training data. (4) Input the multi-light image sequence recovered in step (3) into the pre-trained photometric stereo vision network module, and predict the surface normal map of the object under test by extracting the lighting features and spatial features.

[0012] In step (1), the monochromatic light source is a narrow-band monochromatic LED light source, and the center wavelength of the LED light source covers the range of 460 nanometers to 665 nanometers; the LED light source is distributed in a hemispherical or circular shape around the object being measured to ensure that light of different wavelengths is incident from different preset directions.

[0013] In step (2), the single two-dimensional grayscale compressed measurement image generated by the single-channel image sensor is the weighted integral of all spectral slices that have been spatially modulated and sheared.

[0014] In step (2), the optical dispersive element is a double Amish prism or a grating. The single-channel image sensor is a CMOS image sensor.

[0015] In step (3), constructing the inverse problem involves building a mathematical model for an underdetermined inverse problem based on total variational regularization: ; in, This represents a sequence of multi-illuminated images to be reconstructed. This represents the input single-frame two-dimensional grayscale compressed measurement image. Represents the sensing matrix, For the introduced auxiliary variables, For regularization parameters, This represents the total variation algorithm. Indicates the first The next iteration.

[0016] The physics-driven iterative optimization algorithm employs the generalized alternating projection algorithm, and the iterative optimization process is as follows: Set the number of iterations In the In this iteration, the auxiliary variables from the previous stage are utilized. Update the currently estimated multi-illumination image sequence Then, using the currently estimated multi-illumination image sequence Update current auxiliary variable .

[0017] In step (4), the pre-trained photometric stereo vision network module is built based on the Transformer architecture, including a spatial-lighting feature encoder, an illumination Transformer, and a spatial Transformer. The spatial-lighting feature encoder uses ConvNeXt as the backbone network to extract multi-scale image-level features, and uses the Transformer module to model the pixel-level optical axis interaction. Finally, the multi-scale features are fused through a feature pyramid network.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. Single-shot acquisition and high data efficiency: Only a single exposure is needed to acquire multi-light information that would otherwise require multiple acquisitions, and the high-dimensional spectral-angle data cube is compressed into a single two-dimensional grayscale image, significantly improving acquisition efficiency and reducing storage and transmission costs.

[0019] 2. Low hardware cost: It uses narrowband LEDs and ordinary grayscale cameras, eliminating the need for expensive multispectral cameras and significantly reducing hardware costs.

[0020] 3. Training-free reconstruction and plug-and-play pre-trained networks: The physics-driven iterative optimization algorithm (GAP) does not require training on specific data, and the decoupled images have the same properties as regular multi-illumination images. General pre-trained weights can be directly loaded without hardware fine-tuning.

[0021] 4. High physical fidelity: Real surface details are restored through physical decoupling, avoiding semantic "illusion" and maintaining the precision advantage of photometric stereo vision. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This diagram shows a comparison of the optical path structures of the training-free snapshot photometric stereo vision method based on spectral compression imaging of the present invention with those of two existing photometric stereo vision methods.

[0024] Figure 2 This is a flowchart illustrating the generation of a single two-dimensional grayscale compression measurement image in an embodiment of the present invention.

[0025] Figure 3 This is a flowchart of the training-free reconstruction algorithm in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0028] A training-free snapshot photometric stereo vision method based on spectral compression imaging includes the following steps: S1. Construct a spectral-angle encoding mapping mechanism: Obtain the input light field of the scene under test. For example... Figure 1 As shown, unlike the traditional stereophotometric method in (a) which uses a white light source to illuminate and capture images sequentially from different angles, this invention employs the snapshot-style stereophotometric method corresponding to (c). This method utilizes the physical coupling characteristics of illumination and wavelength, setting up a set of monochromatic light sources of different wavelengths to simultaneously illuminate the object from different directions. In this case, each wavelength spectral slice in the scene actually corresponds to an illumination image in a specific direction. This encoding method "folds" high-dimensional angular information into the spectral dimension, providing a physical basis for subsequent compression.

[0029] S2, Compressed Imaging Acquisition and Measurement Generation: (and) Figure 1 Unlike the multispectral photometric stereo method corresponding to (b), which uses expensive multispectral cameras to acquire spectral image data, this invention utilizes the optical path of a spectral compression imaging (SCI) system to perform dimensionality reduction acquisition of the encoded light field. For example... Figure 2 As shown, the optical path contains a fixed binary coded aperture. (Right now Figure 2 The scene contains a coded mask and an optical dispersive element (i.e., a dispersive prism). The scene's spectral-angle data cube. First, spatial modulation of the coded aperture yields coded scene data. Then, spectral shifting using a dispersive element results in an image captured by a single-channel image sensor. It is a superposition of all modulated and shifted spectral slices: ; This step compresses the data at the speed of light, compressing information from multiple illumination images into a single grayscale image.

[0030] S3. Construct a physics-driven inverse solution model: compress the measurement image Spectral-Angle Data Cube Binary encoding aperture Vectorize them respectively , , This invention constructs an inverse problem model. Instead of directly regressing normals using an end-to-end black-box network, it first performs physical-level decoupling. The following optimization model based on total variational (TV) regularization is constructed: ; The model aims to find a solution that satisfies both the compressibility of measurements and the smoothness of natural images prior.

[0031] S4. Perform illumination decoupling based on Generalized Alternating Projection (GAP): To avoid the need for training on specific SCI data, this invention uses the GAP algorithm to iteratively solve the model in S3. This process is decomposed into two closed-form subproblems: Measurement consistency correction: via linear manifold projection: This ensures that the recovered image matches the actual observation after optical compression.

[0032] Prior constraint recovery: The projected result is processed using a TV denoiser to remove artifacts and noise introduced by compressed sampling. In each iteration, the implicit multi-illumination image sequence is physically decoupled.

[0033] S5. The multi-illumination image sequence decoupled from S4 is input into the photometric stereo network based on the normal estimation of the pre-trained transformer. The network architecture adopts the following approach: 1) Spatial-Illumination Feature Encoder: Image-level features are extracted using the ConvNeXt backbone, pixel-level optical axis interactions are modeled using the Transformer network, and multi-scale representations are fused through the Feature Pyramid Network (FPN).

[0034] 2) Feature Transformation and Prediction: The encoded features are sequentially processed by the Light Transformer and the Spatial Transformer, and finally the surface normals are estimated by the Normal Prediction MLP.

[0035] Since the input decoupled image is physically equivalent to a regular multi-illuminated image, this invention can directly load model weights pre-trained on a general RGB dataset without needing to fine-tune them for specific hardware.

[0036] The following is combined Figure 1 , Figure 2 and Figure 3 The present invention will be further described below.

[0037] 1. System setup and coding (corresponding steps S1-S2).

[0038] This invention proposes a snapshot-type photometric stereo system based on spectral compression imaging. For example... Figure 1 As shown, the system mainly consists of an illumination module and an imaging module.

[0039] 1) Spectrum-Angle Encoded Illumination Module: This module consists of 12 narrowband monochromatic LEDs. These LEDs are arranged around the sample under test, each corresponding to a different spatial illumination angle. The center wavelength of the LEDs covers the visible light range of 460 nm to 665 nm. Through this configuration, incident light from different directions is given a specific spectral fingerprint, achieving physical locking of the "wavelength-angle".

[0040] 2) Compressed Imaging Acquisition Module (SCI Camera): The system employs a customized multispectral SCI camera. This camera integrates the following core components in its optical path: Coded Aperture: Placed at the front or middle of the imaging optical path, it is used for high-frequency spatial intensity modulation of the incident scene light field. Its transmittance function is... ; Optical Dispersion Element: Typically a dispersive prism or grating, used to generate a lateral displacement corresponding to the wavelength of light signals of different wavelengths (i.e., different illumination angles) on the sensor target surface; Lens group: Used for focusing and imaging in the optical path; CMOS image sensor: As a single-channel detector, it is used to integrate and capture the full-spectrum light energy after modulation and dispersion to generate a single two-dimensional compressed measurement image.

[0041] When all 12 LEDs are lit simultaneously to illuminate the sample, the reflected light from the object's surface carries angular information (marked by wavelength). The light enters the SCI camera, undergoes spatial modulation of the coded aperture and spectral shifting by the dispersive element, and is finally superimposed onto the CMOS sensor to form an image.

[0042] 2. Training-free decoupling algorithm (corresponding to steps S3-S4).

[0043] like Figure 3 As shown, the processor receives a single compressed measurement image. and the formation of a coding mask by coding aperture Initialize auxiliary variables and estimated value It is zero.

[0044] Next, total variation constraints are applied to the data, and the GAP iteration loop is entered (the... (nth iteration) (1) Linear manifold projection (updated) Based on the measurement consistency constraint, the auxiliary variables from the previous round are... Project onto the linear manifold to update the current estimate. .

[0045] (2) Variational denoising (update) ): The projected result Treating the image as noisy, the total variation (TV) denoising algorithm is applied to solve for the new auxiliary variables. This section employs an iterative clipping algorithm, which limits the gradient... Norms can remove noise while preserving image edge information. go through After each iteration, the final decoupled multi-illumination image sequence is output. .

[0046] 3. Normal estimation network (corresponding to step S5).

[0047] like Figure 3 As shown, multi-illumination image sequence It is fed into the neural network.

[0048] (1) Feature extraction: The decoupled measurements are first processed by a spatial-light feature encoder. This encoder uses the ConvNeXt backbone to extract multi-scale image-level features, uses a Transformer network to model pixel-level optical axis interactions, and fuses the multi-scale features mentioned above through a feature pyramid network implemented in the MMSegmentation toolbox.

[0049] (2) Global transformation: Use Light Transformer to gather information from all lighting directions and generate a global lighting context vector; use Spatial Transformer to handle non-local dependencies in space and resolve ambiguity of normals in flat regions.

[0050] (3) Regression: Finally, the features are mapped to a 3-channel unit normal vector map through the MLP layer.

[0051] This invention, through the combination of the aforementioned physical device and algorithm decoding, enables high-precision photometric stereoscopic reconstruction using only a single compressed snapshot without the need for targeted training data.

[0052] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training-free snapshot-based photometric stereo vision method based on spectral compression imaging, characterized in that, Includes the following steps: (1) Set up multiple monochromatic light sources of different wavelengths to simultaneously illuminate the object from different directions. Each wavelength spectral slice corresponds to a lighting image in one direction, thereby encoding the angular lighting information of the object into the spectral dimension of the scene emissivity to form a high-dimensional spectral-angle data cube. (2) Spatial modulation of the spectral-angle data cube is performed by encoding aperture, and then shearing and translation along the spectral dimension is performed by optical dispersive element. Finally, integration is performed on a single-channel image sensor to generate a single two-dimensional grayscale compressed measurement image. (3) Based on the vectorized form of the compressed measurement image, an inverse problem model is constructed. Using a physics-driven iterative optimization algorithm, the multi-light image sequence corresponding to different lighting directions is decoupled and recovered from a single two-dimensional grayscale compressed measurement image without the need for external training data. (4) Input the multi-light image sequence recovered in step (3) into the pre-trained photometric stereo vision network module, and predict the surface normal map of the object under test by extracting the lighting features and spatial features.

2. The training-free snapshot photometric stereo vision method based on spectral compression imaging according to claim 1, characterized in that, In step (1), the monochromatic light source is a narrow-band monochromatic LED light source, and the center wavelength of the LED light source covers the range of 460 nanometers to 665 nanometers; the LED light source is distributed in a hemispherical or circular shape around the object being measured to ensure that light of different wavelengths is incident from different preset directions.

3. The training-free snapshot photometric stereo vision method based on spectral compression imaging according to claim 1, characterized in that, In step (2), the single two-dimensional grayscale compressed measurement image generated by the single-channel image sensor is the weighted integral of all spectral slices that have been spatially modulated and sheared.

4. The training-free snapshot photometric stereo vision method based on spectral compression imaging according to claim 1, characterized in that, In step (2), the optical dispersive element is a double Amish prism or a grating.

5. The training-free snapshot photometric stereo vision method based on spectral compression imaging according to claim 1, characterized in that, In step (2), the single-channel image sensor is a CMOS image sensor.

6. The training-free snapshot photometric stereo vision method based on spectral compression imaging according to claim 1, characterized in that, In step (3), constructing the inverse problem involves building a mathematical model for an underdetermined inverse problem based on total variational regularization: ； in, This represents a sequence of multi-illuminated images to be reconstructed. This represents the input single-frame two-dimensional grayscale compressed measurement image. Represents the sensing matrix, For the introduced auxiliary variables, For regularization parameters, This represents the total variation algorithm. Indicates the first The next iteration.

7. The training-free snapshot photometric stereo vision method based on spectral compression imaging according to claim 6, characterized in that, In step (3), the physics-driven iterative optimization algorithm adopts the generalized alternating projection algorithm, and the iterative optimization process is as follows: Set the number of iterations In the In this iteration, the auxiliary variables from the previous stage are utilized. Update the currently estimated multi-illumination image sequence Then, using the currently estimated multi-illumination image sequence Update current auxiliary variable .

8. The training-free snapshot photometric stereo vision method based on spectral compression imaging according to claim 1, characterized in that, In step (4), the pre-trained photometric stereo vision network module is built based on the Transformer architecture, including a spatial-lighting feature encoder, an illumination Transformer, and a spatial Transformer. The spatial-lighting feature encoder uses ConvNeXt as the backbone network to extract multi-scale image-level features, and uses the Transformer module to model the pixel-level optical axis interaction. Finally, the multi-scale features are fused through a feature pyramid network.

Images