Structured light three-dimensional reconstruction method based on frequency domain-spatial domain fusion

Abstract

The invention discloses a structured light three-dimensional reconstruction method and system based on frequency domain-spatial domain fusion, and belongs to the technical field of deep learning and three-dimensional measurement. The method comprises the following steps: acquiring and preprocessing a stripe image; the method comprises the following steps: respectively extracting spatial domain features and frequency domain features through a double-branch encoder, wherein a frequency domain encoder explicitly extracts phase, frequency and modulation degree information by adopting 2D-FFT (Fast Fourier Transform) and a complex convolutional network; bidirectional attention fusion is carried out on the spatial domain and frequency domain features on four scales through a cross-domain attention fusion module, and the fusion weight is adaptively adjusted by using a gating mechanism; guiding an up-sampling process by using frequency domain features through a frequency domain guiding decoder, and recovering spatial resolution in combination with attention residual jump connection; and outputting the depth map and carrying out post-processing optimization. According to the method, the frequency domain physical priori of the structured light stripes is fully utilized, and high-precision and high-efficiency single-frame stripe image three-dimensional reconstruction is realized.

Description

Technical Field

[0001] This invention relates to the fields of 3D measurement and deep learning technology, specifically to a structured light 3D reconstruction method based on frequency-spatial dual-domain fusion. Background Technology

[0002] Structured light 3D measurement is widely used in industrial inspection, robot navigation, cultural relic restoration, and medical measurement due to its simplicity, speed, and high accuracy. A typical system consists of a projector and a camera. It projects periodic stripes onto the surface of an object and acquires images of the deformed stripes. Phase encoding and decoding are used to recover the correspondence between the projected coordinates and the object's surface, and finally, triangulation is used to obtain the object's 3D shape information. Traditional structured light 3D reconstruction methods are mainly divided into two categories: multi-frame methods (such as phase-shifting methods and Gray code methods) and single-frame methods (such as Fourier transform profilometry (FTP)). The advantage of multi-frame methods is high accuracy, reaching sub-pixel levels. The disadvantages are that they require projecting multiple images (usually 3-12), have long acquisition and processing times, are unsuitable for dynamic scenes or moving objects, and cannot meet real-time measurement requirements. The advantage of single-frame methods is high speed, requiring only a single frame image. Its disadvantages are lower accuracy, susceptibility to noise, significant loss of detail information, and difficulty in meeting high-precision measurement requirements.

[0003] In recent years, deep learning technology has made groundbreaking progress in the field of computer vision, and researchers have begun to apply it to structured light 3D reconstruction, attempting to achieve high-precision single-frame reconstruction. Existing deep learning methods mainly adopt an end-to-end approach, directly predicting the depth map from the stripe image. Although existing deep learning methods have improved the accuracy and speed of single-frame reconstruction to some extent, the following problems still exist: First, most existing methods only perform feature extraction and processing in the spatial domain (image domain), ignoring the physical nature of structured light stripe images. In fact, stripe images are essentially modulations of sinusoidal signals, and their frequency domain features (frequency, phase, modulation) are directly related to the depth information of the object. Traditional methods such as Fourier Transform Profilometry (FTP) achieve 3D reconstruction based on frequency domain analysis, but existing deep learning methods have failed to effectively utilize this method. Second, during the downsampling process of the encoder, existing deep learning methods easily lose high-frequency detail information (such as edges and textures). Simultaneously, during the upsampling process, they mainly rely on spatial domain features, resulting in limited recovery capabilities and leading to blurred edges and unclear details in the reconstruction results. Finally, in regions of abrupt changes in depth (object edges, occlusion boundaries), existing methods generally suffer from decreased accuracy and blurred contours, affecting reconstruction quality. These shortcomings severely impact the application of deep learning technology in the field of 3D reconstruction. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a structured light 3D reconstruction method based on frequency-spatial dual-domain fusion, comprising the following steps:

[0005] S1, perform system calibration on the structured light 3D measurement system, and obtain the spatial transformation relationship between the camera, projector and world coordinate system;

[0006] S2, perform stripe image acquisition and preprocessing to obtain a normalized single-frame deformed stripe image;

[0007] S3, based on the dual-branch encoder structure, performs feature extraction on the stripe image to obtain spatial domain features and frequency domain features respectively;

[0008] S4, adaptively fuses features at different scales through a variable scale fusion module;

[0009] S5 employs a decoder structure combined with attention residual skip connections to reconstruct the fused features step by step.

[0010] S6 outputs an absolute phase map and combines it with system calibration parameters to complete the phase-to-depth conversion, thereby realizing three-dimensional topography reconstruction.

[0011] The present invention is further configured such that: in step S1, the intrinsic and extrinsic parameters of the camera and the projector are calibrated by a calibration plate, and the mapping relationship between the camera coordinate system, the projector coordinate system and the world coordinate system is established, thereby providing geometric constraints for the subsequent conversion of absolute phase to real space coordinates.

[0012] The present invention is further configured such that: in step S2, a sinusoidal stripe pattern is projected onto the surface of the object being measured by a projector, a camera simultaneously acquires a single frame of deformed stripe image, and the stripe image is converted into a grayscale image and then normalized. At the same time, the stripe image is denoised according to the actual application scenario to reduce the influence of environmental noise and imaging noise.

[0013] The present invention is further configured such that: in step S3, the encoder adopts a dual-branch structure, wherein:

[0014] One branch is used to extract multi-scale spatial features from the stripe image;

[0015] Another branch is used to extract multi-scale frequency domain features from stripe images;

[0016] The dual-branch parallel output of feature maps at different scales provides a foundation for subsequent feature fusion.

[0017] The present invention is further configured such that: in step S4, the feature maps output by the spatial domain branch and the frequency domain branch are input into the variable scale fusion module, and the fusion weights are adaptively allocated according to the importance of features at different scales, so as to make full use of multi-scale stripe information.

[0018] The present invention is further configured such that: in step S5, the decoder recovers the feature resolution by upsampling step by step, and introduces an attention residual skip connection mechanism during the upsampling process, so as to suppress background noise while retaining the high-resolution feature information of the encoder, thereby improving the accuracy of absolute phase prediction.

[0019] The present invention is further configured such that: in step S6, the decoder finally outputs a single-channel absolute phase map, and converts the absolute phase map into a depth map according to the system calibration parameters to obtain the three-dimensional shape information of the object under test.

[0020] Compared with known public technologies, the technical solution provided by this invention has the following beneficial effects:

[0021] 1. Frequency Domain Feature Extractor: Frequency domain analysis is introduced into the deep learning reconstruction of structured light. The frequency, phase and modulation information of the stripes are extracted by 2D-FFT. A complex convolutional neural network is designed to process the real and imaginary parts at the same time and preserve the phase information.

[0022] 2. Cross-domain attention fusion module: A spatial-frequency bidirectional attention mechanism is proposed to achieve mutual enhancement of features from the two domains; a gating fusion strategy is designed to adaptively learn the fusion weights of the dual-domain features; cross-domain fusion is performed at multiple levels of the encoder to make full use of multi-scale information;

[0023] 3. Frequency Domain Guided Decoder: Utilizes frequency domain features as the upsampling guide signal to guide the recovery of spatial details; frequency domain features contain rich edge and high-frequency information, which can effectively improve edge blurring problems; a hierarchical frequency domain guiding strategy is designed to guide at different scales; Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a structured light 3D reconstruction method based on frequency-spatial dual-domain fusion according to the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments.

[0026] Example 1

[0027] like Figure 1 As shown, this invention provides a technical solution: a structured light 3D reconstruction method based on frequency-spatial dual-domain fusion, comprising the following steps:

[0028] S1. Perform system calibration on the camera and projector to obtain the transformation relationship between the camera, projector and world coordinate system;

[0029] S2. Stripe image acquisition and preprocessing;

[0030] S3, Feature extraction of dual-branch encoder (spatial domain encoder and frequency domain encoder).

[0031] S4, Variable-scale fusion of cross-domain attention fusion module;

[0032] S5, Frequency Domain Guided Decoder and Attention Residual Jump Connection;

[0033] S6. Depth map prediction and post-processing optimization.

[0034] In step S1, the system calibration of the camera and projector is performed to obtain the transformation relationship between the camera, projector, and world coordinate system:

[0035] S1.1 Camera Calibration: Use Zhang Zhengyou's calibration method to calibrate the industrial camera, acquire at least 15 images of the checkerboard calibration board from different angles, and calculate the camera intrinsic parameter matrix K. c and distortion coefficient D c :

[0036] K c =

[0037] Among them, f x and f y c is the focal length. x and c y Principal point coordinates;

[0038] S1.2 Projector Calibration: Treat the projector as a reverse camera, projecting fringe patterns of different phases onto the calibration board. The camera acquires the distorted fringe images. By unwrapping the phases, the correspondence between projector pixels and camera pixels is established, and the projector intrinsic parameter matrix K is calculated. p and distortion coefficient D p ;

[0039] S1.3, Stereo Calibration: Using images of the calibration plate at different positions, calculate the rotation matrix R and translation vector T between the camera coordinate system and the projector coordinate system.

[0040]

[0041] Among them, P c P is a point in the camera coordinate system. p The point is in the projector coordinate system;

[0042] S1.4 Establishing the World Coordinate System: Select a position on the calibration plate as the origin of the world coordinate system, and calculate the extrinsic parameter matrices of the camera and projector relative to the world coordinate system [R]. c |T c ] and [Rp |T p ];

[0043] S1.5 Depth Mapping Relationship Establishment: Based on the system calibration parameters, establish the mapping relationship between fringe phase and depth;

[0044] S1.6 Calibration Parameter Storage: Save all calibration parameters as a configuration file for use in the subsequent reconstruction process.

[0045] In step S2, stripe image acquisition and preprocessing:

[0046] S2.1 Stripe Projection: A sinusoidal stripe pattern is projected onto the surface of the object to be measured using an LED projector. The stripe direction is horizontal, and the stripe frequency is set to 0.1-0.3 cycles / pixel. The grayscale distribution of the stripe pattern is as follows:

[0047]

[0048] Where A is the background light intensity, B is the modulation index, and f0 is the fringe frequency. This is the initial phase;

[0049] S2.2 Image Acquisition: Use an industrial camera to acquire images of the deformed stripes on the surface of the object. The image resolution is 512×512 pixels, and the exposure time is adaptively adjusted according to the ambient light.

[0050] S2.3 Image Normalization: The acquired stripe image is normalized, mapping the pixel value range from [0, 255] to [0, 1].

[0051]

[0052] Where I is the original image, I norm The image is after normalization;

[0053] S2.4 Noise Removal: A bilateral filter is applied to denoise the image, preserving edge information while removing noise. The filter kernel size is 5×5, and the spatial domain standard deviation σ is... s = 2.0, standard deviation of grayscale σ r = 0.1:

[0054] ;

[0055] S2.5 Contrast Enhancement: The CLAHE (Contrast Limit Adaptive Histogram Equalization) algorithm is used to enhance stripe contrast, with a block size of 8×8 and a contrast limit threshold of 2.0, improving the clarity of stripe edges;

[0056] In step S3, feature extraction from the dual-branch encoder (spatial domain encoder and frequency domain encoder):

[0057] The neural network employs a dual-branch encoder structure to extract spatial and frequency domain features, respectively.

[0058] S3.1 Spatial Encoder Feature Extraction

[0059] S3.1.1, First layer encoding: Input stripe image After the first layer of convolutional encoding, shallow spatial features are extracted:

[0060] ;

[0061] Output feature dimensions: The kernel size is 3×3, the stride is 1, and the padding is 1.

[0062] S3.1.2, Second-level encoding: For Perform downsampling to extract mid-level spatial features:

[0063] ;

[0064] Output feature dimensions: A stride of 2 achieves a 2x downsampling.

[0065] S3.1.3, Third Layer Encoding: Continue downsampling to extract high-level spatial features:

[0066] ;

[0067] Output feature dimensions: ;

[0068] S3.1.4, Fourth Layer Encoding: Extracting the Deepest Spatial Features:

[0069] ;

[0070] Output feature dimensions: ;

[0071] S3.1.5, Construction of the Spatial Feature Pyramid: Organizing the four levels of spatial features into a feature pyramid:

[0072] ;

[0073] Used for subsequent cross-domain fusion and jump connections.

[0074] S3.2 Feature Extraction from Frequency Domain Encoder

[0075] S3.2.1 Two-dimensional Fast Fourier Transform: Perform a 2D-FFT transform on each channel of the input stripe image to convert the spatial domain signal to the frequency domain.

[0076] ;

[0077] Where (x,y) are spatial coordinates, (u,v) are frequency coordinates, W=512, H=512;

[0078] S3.2.2 Complex Spectrum Separation: Separate the complex spectrum into two channels: the real part and the imaginary part.

[0079] ;

[0080] ;

[0081] Output dimensions: ;

[0082] S3.2.3, Spectral Feature Extraction: Extracting physical prior information from the complex spectrum:

[0083] Amplitude spectrum: , representing the intensity of each frequency component;

[0084] Phase spectrum: It contains stripe phase information;

[0085] Power spectrum: , representing energy distribution;

[0086] S3.2.4 Complex Convolutional Layer 1: Perform complex convolutions on the real and imaginary parts respectively, preserving phase information:

[0087] ;

[0088] ;

[0089] ;

[0090] Output feature dimensions: (complex field);

[0091] S3.2.5, Complex Convolutional Layer 2: Continue to extract frequency domain features and downsample:

[0092] ;

[0093] Output feature dimensions: ;

[0094] S3.2.6, Complex Convolutional Layer 3:

[0095] ;

[0096] Output feature dimensions: ;

[0097] S3.2.7, Complex Convolutional Layer 4:

[0098] ;

[0099] Output feature dimensions: ;

[0100] S3.2.8, Converting Complex Number Features to Real Number Features: Converting complex number features to real number features facilitates subsequent processing.

[0101] ;

[0102] Each layer of frequency domain features contains real and imaginary part information;

[0103] S3.2.9, Construction of Frequency Domain Feature Pyramid: Organize the frequency domain features of the four levels into a feature pyramid:

[0104] ;

[0105] For subsequent cross-domain integration.

[0106] In step S4, the variable-scale fusion of the cross-domain attention fusion module:

[0107] The cross-domain attention fusion module fuses spatial and frequency domain features at four different scales to achieve complementarity between the two domains.

[0108] S4.1 Feature Dimension Alignment: For the i-th layer (i=1,2,3,4), ensure that the spatial domain feature F si and frequency domain features F fi For features with the same spatial dimensions and number of channels, if the frequency domain characteristics are in complex form, then the real and imaginary parts are concatenated to convert them into real features:

[0109] ;

[0110] S4.2 Attention Calculation from Spatial Domain to Frequency Domain: Generate a Query using spatial domain features, generate a Key and Value using frequency domain features, calculate attention weights, and enhance the frequency domain features:

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] Where, d k The dimension for the Key is used to scale the attention score;

[0117] S4.3 Attention Calculation from Frequency Domain to Spatial Domain: Generate a Query using frequency domain features and a Key and Value using spatial domain features. Calculate attention weights and enhance spatial domain features.

[0118] ;

[0119] ;

[0120] ;

[0121] ;

[0122] ;

[0123] S4.4, Gating Weight Generation: The enhanced spatial and frequency domain features are concatenated, and gating weights G are generated through convolutional layers and a sigmoid activation function to adaptively adjust the contributions of the two domain features.

[0124] ;

[0125] ;

[0126] The gate weights G ∈ [0,1], when G is close to 1, they depend more on spatial domain features, and when G is close to 0, they depend more on frequency domain features.

[0127] S4.5 Adaptive Feature Fusion: Weighted fusion of features from two domains based on gating weights.

[0128] ;

[0129] Where ⊙ represents element-wise multiplication;

[0130] S4.6 Multi-scale fusion output: Perform steps S4.1-S4.5 on each of the four levels to obtain fused features at the four scales:

[0131] ;

[0132] These correspond to resolutions of 512×512, 256×256, 128×128, and 64×64, respectively, and are used for skip connections in subsequent decoders.

[0133] In step S5, the frequency domain guided decoder and attention residual skip connection:

[0134] The decoder employs a 4-layer upsampling structure, starting from the deepest feature layer. The spatial resolution is restored layer by layer. During the upsampling process of each layer, frequency domain features are used for guidance, and the encoder features are fused through attention residual jump connections.

[0135] S5.1 Frequency Domain Guided Weight Generation: For the i-th layer decoding (i=4,3,2,1), the frequency domain features F of the corresponding layer are used. fi Generate guided weight graph :

[0136] ;

[0137] Guiding weight This is used to indicate which areas need enhanced detail restoration;

[0138] S5.2 Low-resolution feature upsampling: Upsampling the input features of the current layer Perform bilinear interpolation upsampling to double the spatial resolution:

[0139] ;

[0140] For example, during layer 4 decoding, The size was upsampled from 64×64 to 128×128;

[0141] S5.3 Frequency Domain Guided Feature Enhancement: The upsampled features are enhanced using the guided weights after upsampling to highlight high-frequency details.

[0142] ;

[0143] ;

[0144] Where α = 0.5 is the enhancement coefficient, and ⊙ represents element-wise multiplication;

[0145] S5.4 Attention Residual Skip Connection: Obtaining fused features from the corresponding level of the encoder As a skip connection, residual weights are calculated using an attention mechanism:

[0146] ;

[0147] ;

[0148] Attention weight Adaptively select useful information from skip connections;

[0149] S5.5 Feature Concatenation and Fusion: The enhanced upsampled features and weighted skip connection features are concatenated along the channel dimension.

[0150] ;

[0151] The number of channels is doubled after splicing, for example, when decoding the 4th layer, the number of channels changes from 256 to 512.

[0152] S5.6 Feature Refinement Convolution: Two 3×3 convolution layers are used to refine the concatenated features, reducing the number of channels and extracting fused features.

[0153] ;

[0154] ;

[0155] Output features As input to the next layer of decoder;

[0156] S5.7 Decode and output layer by layer: Repeat steps S5.1-S5.6 to complete the decoding of 4 layers:

[0157] Level 4: ;

[0158] Level 3: ;

[0159] Level 2: ;

[0160] Level 1: ;

[0161] Finally, a feature map F with the same resolution as the input image is obtained. final .

[0162] In step S6, depth map prediction and post-processing optimization:

[0163] S6.1 Preliminary depth map prediction: The decoder's final output feature F final Mapped to a single-channel depth map using a 1×1 convolutional layer:

[0164] ;

[0165] Output dimensions: , representing the original predicted depth value;

[0166] S6.2 Depth Value Normalization and Denormalization: During the training phase, the ground truth depth map is normalized to the range [0,1]. During the prediction phase, it needs to be denormalized to the actual depth range [Z]. min Z max ]:

[0167] ;

[0168] Z min and Z max These are the minimum and maximum depth values ​​of the scene, obtained through calibration;

[0169] S6.3 Median Filtering for Noise Reduction: Apply a 3×3 median filter to the predicted depth map to remove isolated noise points.

[0170] ;

[0171] Median filtering removes salt-and-pepper noise while maintaining edge sharpness;

[0172] S6.4 Edge-Preserving Bilateral Filtering: Apply a bilateral filter to further smooth the depth map while maintaining sharp edges.

[0173] ;

[0174] in, Controlling the smoothness of the spatial domain, Control the smoothness of the depth range;

[0175] S6.5 Depth Discontinuity Detection and Repair: Detect discontinuous regions (such as occlusion boundaries) in the depth map and repair them through morphological operations.

[0176] ;

[0177] ;

[0178] The threshold is set adaptively based on the scene depth range.

[0179] S6.6 Confidence-Weighted Optimization: Calculate the confidence level of depth prediction using the modulation intensities of the stripe image, and perform interpolation optimization for low-confidence regions.

[0180] ;

[0181] ;

[0182] Among them, D interpolated Obtained by interpolation through surrounding high-confidence regions;

[0183] S6.7 Final Depth Map Output: After the above post-processing steps, the final optimized depth map D is obtained. final Used for 3D point cloud reconstruction:

[0184] ;

[0185] Among them, Kc Let P(x,y) be the camera intrinsic parameter matrix, and let P(x,y) be the coordinates of a point in three-dimensional space.

[0186] This embodiment combines bi-branch feature extraction, variable-scale fusion, and attention residual skip connections to effectively improve the accuracy of local detail reconstruction while ensuring global phase consistency. It is suitable for complex surfaces and high-precision single-frame structured light 3D reconstruction scenarios. Equivalent substitutions or modifications made by those skilled in the art to the above steps without departing from the spirit of this invention should fall within the protection scope of this invention.

Claims

1. A structured light 3D reconstruction method based on frequency-spatial dual-domain fusion, characterized in that, Includes the following steps: S1. Perform system calibration on the camera and projector to obtain the transformation relationship between the camera, projector and world coordinate system; S2. Stripe Image Acquisition and Preprocessing: A sinusoidal stripe pattern is projected onto the surface of the object to be measured using a projector, and the deformed stripe image is acquired by a camera and normalized. S3. Extract features using a dual-branch encoder: 1) Multi-scale spatial features are extracted by performing multi-layer convolutional coding on the stripe image using a spatial encoder; 2) The stripe image is transformed in the frequency domain by a frequency domain encoder to extract multi-scale frequency domain features; S4. Feature fusion is performed through the cross-domain attention fusion module: a first attention weight is generated based on the spatial domain features to adjust the frequency domain features, and a second attention weight is generated based on the frequency domain features to adjust the spatial domain features. The contribution ratio of the two domain features is adaptively adjusted and fused through a gating mechanism. S5. Upsampling is performed by guiding the decoder in the frequency domain: guiding weights are generated using frequency domain features, the fused features are upsampled and enhanced by the guiding weights, and the spatial resolution is restored layer by layer. S6. Output depth map and perform post-processing optimization.

2. The structured light 3D reconstruction method based on frequency-spatial dual-domain fusion according to claim 1, characterized in that, In step S3, the frequency domain encoder extracts frequency domain features, including phase information, frequency information, and modulation information, by performing frequency domain transformation on the stripe image.

3. The structured light 3D reconstruction method based on frequency-spatial dual-domain fusion according to claim 1, characterized in that, The cross-domain attention fusion module in step S4 includes: 1) Generate a query based on spatial domain features, generate a key and value based on frequency domain features, calculate the first attention weight, and enhance the frequency domain features; 2) Generate a query based on frequency domain features, generate a key and value based on spatial domain features, calculate the second attention weight, and enhance the spatial domain features; 3) Generate fusion weights through a gating mechanism, and perform adaptive weighted fusion of the enhanced spatial and frequency domain features.

4. The structured light 3D reconstruction method based on frequency-spatial dual-domain fusion according to claim 1, characterized in that, The frequency domain-guided decoder upsampling process in step S5 includes: 1) Generate guiding weights using frequency domain features; 2) Upsample the fused features; 3) Enhance upsampled features using guided weights; 4) Fuse with encoder features and restore spatial resolution layer by layer.

Images