A color interference resistant single-frame three-dimensional reconstruction method, device, equipment and medium

Abstract

The application discloses an anti-color-interference single-frame three-dimensional reconstruction method, device, equipment and medium, and belongs to the field of optical three-dimensional sensing, and aims at the problems of low measurement efficiency and inability to accurately measure color objects in the prior art traditional multi-frame phase shift method.The anti-color-interference single-frame three-dimensional reconstruction method constructs a double-context scale perception network, and only needs to collect a single Hal-like composite fringe image to restore a complete multi-frequency multi-step phase shift fringe sequence, so that 11 times of image collection are reduced compared with the traditional method, and the measurement efficiency is greatly improved.The average absolute error of the network prediction absolute phase is about 0.05 rad, and the local error is as low as 0.004 rad, and the accuracy is close to that of the traditional multi-frame method.Meanwhile, the gray-scale fringe coding fundamentally eliminates the interference of the object surface color on the measurement fringe, and can accurately measure the surface topography of various color objects, and the standard ceramic plane reconstruction error is about 0.023 mm, so that the industrial precision detection requirement is met.

Description

Technical Field

[0001] This invention relates to the field of optical three-dimensional sensing, and in particular to a method, apparatus, device, and medium for single-frame three-dimensional reconstruction with resistance to color interference. Background Technology

[0002] Optical 3D measurement technology, with its significant advantages such as non-contact operation, high precision, and full-field acquisition, has been widely used in fields such as cultural relic preservation, industrial quality inspection, intelligent manufacturing, and machine vision. Among the many active optical 3D measurement technologies, fringe projection profilometry (FPP), which uses sinusoidal fringe patterns as a carrier, has become one of the most widely used structured light measurement methods due to its high measurement accuracy and flexible system setup.

[0003] Simultaneously improving measurement efficiency and accuracy has always been a core research focus in the field of fringe projection profilometry (FPP). In recent years, combining fringe projection technology with deep learning for 3D reconstruction has been recognized as a highly promising technical approach. Against this backdrop, single-frame 3D shape measurement technology has gradually become a research hotspot, aiming to meet the real-time requirements of dynamic scenes. Existing single-frame measurement methods typically use colored composite fringe patterns as the projection carrier; however, these methods have significant drawbacks when dealing with objects whose surfaces have bright red, green, blue, and other colors.

[0004] Specifically, the existing technology has the following main shortcomings: 1. Measurement errors caused by uneven reflectivity of colored object surfaces: Objects of different colors have different reflectivity to light waves and absorb different wavelength components of the projected light. This non-uniform reflectivity can lead to problems such as inconsistent brightness, decreased contrast, and non-sinusoidal distortion in the stripe patterns captured by the camera.

[0005] 2. Color crosstalk and decoding difficulties: When measuring colored objects, the color information on the object's surface can severely interfere with the color components of the structured light, causing color deviations. This interference is particularly noticeable at color boundaries, leading to significant errors in the decoding process and even causing phase calculation failures.

[0006] 3. Limitations of existing correction methods: Although existing research has proposed methods based on intensity correction, HSI color space decoding, or dual-projector compensation to try to solve the above problems, these methods often struggle to maintain high efficiency in single-frame measurement while simultaneously achieving high-precision reconstruction under complex color textures. For example, some color-coding-based methods are unable to effectively avoid color crosstalk, while traditional multi-step phase-shifting methods, although highly accurate, require the acquisition of multiple images, which cannot meet the needs of online high-speed detection.

[0007] In summary, overcoming the interference of object surface color on projected stripes and achieving efficient and high-precision 3D topography measurement of colored objects with only a grayscale camera and single-frame acquisition is a technical challenge that urgently needs to be solved in the field of optical 3D sensing. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a method, apparatus, device and medium for single-frame 3D reconstruction with resistance to color interference, so as to improve the above-mentioned problems.

[0009] This invention provides a method for single-frame 3D reconstruction against color interference, comprising the following steps: S1. Binocular vision system calibration: Adjust and fix the relative positions of the two grayscale cameras and the digital projector, acquire calibration images, and use Zhang Zhengyou calibration method to obtain the calibration parameters of the grayscale cameras. S2. Image Acquisition and Preprocessing: A Haar-like composite fringe pattern (HCFP), 12 sinusoidal fringe patterns with three-frequency four-step phase shift, and a 3rd-order Haar-like composite fringe pattern brightness reference pattern (brightnesses of 50, 140, and 230 respectively) are projected onto the surface of a colored object, and deformed images are acquired. Gamma distortion correction is performed on all acquired deformed images. The correction mathematical model is as follows:

[0010] in, To output the ideal grayscale value, The actual grayscale value captured by the camera. γ is the system's Gamma distortion coefficient, which must be the average of values ​​measured for different colors. The Haar-like composite fringe image and 12 sinusoidal fringe images were subjected to segmented normalization. The segmented normalization method was based on the coding characteristics of the Haar-like composite fringe, using the intermediate brightness gray value of the Haar-like composite fringe. Threshold Divide the grayscale range into [ , ]and[ , Piecewise linear normalization is performed on the two segments, and the mathematical model is as follows:

[0011] in, The normalized grayscale value. , The upper and lower limits of the ideal grayscale. To acquire the gamma-corrected grayscale values ​​of the striped image, , These are the highest and lowest grayscale values ​​after Gamma correction of the acquired striped image, respectively. Using simple planes, complex freeform surfaces (opera masks), and sharp-edge structures (car models) as the basic carriers, various objects of different colors are combined. Each model is randomly placed in more than 20 different poses within the measurement field of view. Finally, no less than 720 sets of complex reflectivity objects with different color distributions and poses are collected as samples and divided into training set, validation set and test set in a ratio of 8:1:1. All images are pixel normalized to [0,1] and cropped to a uniform size. S3. Construction of Dual Context Scale Awareness Network: The dual context scale awareness network is a three-stage network composed of three improved U-Nets cascaded together, which sequentially realizes the functions of stripe restoration, frequency separation and phase shift prediction; each improved U-Net includes an encoding layer, a decoding layer, an attention module and an adversarial training module, and the encoding layer introduces a spatial frequency fusion attention module; Based on the U-Net framework, a spatial-frequency fusion attention module is introduced into the encoding layer, and an adversarial training framework is used for optimization (using a combined loss function, which includes pixel-level L2 loss to ensure global structural consistency and adversarial loss to improve the realism of local details). This makes the dual-context scale perception network consist of four parts: an encoding layer, a decoding layer, an attention module, and an adversarial training module. At the same time, the dual-context scale perception network adopts a three-level cascaded structure to realize stripe restoration, frequency separation, and phase shift prediction functions in sequence. S4. Model Training: The dual-context scale perception network is trained using the training dataset until the loss converges, resulting in a mature model capable of reconstructing a complete multi-frequency, multi-step phase-shifting stripe image from a single-frame Haar composite stripe image. S5. Measurement data acquisition: Project a single-frame Haar composite stripe pattern onto the colored object to be measured, and simultaneously acquire deformed Haar composite stripe images using two grayscale cameras; The projector projects a single HCFP pattern onto the surface of the object being measured, and two grayscale cameras simultaneously acquire images of deformed stripes modulated by the object's surface. The HCFP images acquired by the left and right cameras are then input into a trained dual-context scale perception network, which outputs 12 standard three-frequency four-step phase-shifted stripe images. S6. Fringe Prediction and Phase Calculation: Input the deformed Haar composite fringe image into the mature model respectively, output a complete multi-frequency multi-step phase-shifted fringe image, calculate the wrapped phase of each frequency using the phase-shifting method, and perform phase unwrapping using the multi-frequency heterodyne method to obtain the absolute phase map of the two grayscale cameras. The wrapping phase at three frequencies is calculated using a four-step phase shift method; the time phase is then unwrapped using a three-frequency heterodyne method: first, the difference frequency phase is calculated. and Then, heterodyne calculation is performed to obtain the globally unique phase. Guided by this, the highest frequency absolute phase diagram is obtained by unwrapping the package step by step. S7. Binocular 3D Reconstruction: Subpixel-level matching is performed on the absolute phase map to obtain a disparity map. Combined with the calibration parameters, the 3D coordinates are calculated using the principle of binocular stereo vision to complete the shape reconstruction of the colored object under test.

[0012] As a further improvement of the present invention, in S2, the segmented normalization process specifically involves: using the mid-brightness grayscale value of the Haar composite stripe image as the segmentation threshold, dividing the grayscale range of the image into a low grayscale segment and a high grayscale segment; for the low grayscale segment, linearly mapping the corrected grayscale value to a preset ideal low grayscale range according to its relative position within the segment; for the high grayscale segment, linearly mapping the corrected grayscale value to a preset ideal high grayscale range according to its relative position within the segment.

[0013] As a further improvement of the present invention, in S3, the construction method of the dual context scale perception network includes: using U-Net as the basic framework, introducing a spatial frequency fusion attention module in the encoding layer, and optimizing it using an adversarial training framework, so that the dual context scale perception network is divided into four parts: encoding layer, decoding layer, attention module and adversarial training module; at the same time, the dual context scale perception network adopts a three-level cascaded structure to realize stripe restoration, frequency separation and phase shift prediction functions in sequence.

[0014] As a further improvement of the present invention, in S3, the dual-context scale perception network includes: a first-stage network for predicting the Gamma-corrected Haar composite fringe image as an ideal normalized image; a second-stage network for separating the ideal normalized image predicted by the first-stage network into three different frequency first phase-shift fringe images; and a third-stage network for predicting the remaining phase-shift fringe images at each frequency based on the first phase-shift fringe image predicted by the second-stage network.

[0015] As a further improvement of the present invention, the training process of the adversarial training framework is optimized using a combined loss function, which includes pixel-level L2 loss to ensure global structural consistency and adversarial loss to improve the realism of local details.

[0016] As a further improvement of the present invention, in S4, the method for constructing the training dataset includes: based on object models of multiple colors, by combining different colors and changing different poses within the measurement field of view, collecting no less than 720 sets of sample images; after normalizing the pixels of all sample images to the [0,1] interval and cropping them to a uniform size, dividing them into training set, validation set and test set in a ratio of 8:1:1.

[0017] As a further improvement of the present invention, in S6, the specific process of phase unwrapping by the multi-frequency heterodyne method is as follows: the three frequency wrapped phases are arranged in order from low to high frequency, and the low-frequency unwrapping result is used to guide the high-frequency unwrapping in turn. The lowest frequency wrapped phase is used as the initial unwrapped phase, and the process is expanded upward step by step to finally obtain an unambiguous absolute phase map.

[0018] The present invention also provides a single-frame 3D reconstruction device against color interference, comprising: The binocular vision system calibration unit is used to adjust and fix the relative positions of the two grayscale cameras and the digital projector, acquire calibration images, and obtain the calibration parameters of the grayscale cameras using the Zhang Zhengyou calibration method. The image capture and preprocessing unit is used to project the brightness reference pattern of the Haar-like composite fringe pattern, the 12 sinusoidal fringe patterns of the three-frequency four-step phase shift, and the third-order Haar-like composite fringe pattern onto the surface of a colored object and acquire deformed images. The deformed images are then subjected to Gamma distortion correction, and the 12 sinusoidal fringe patterns and the Haar-like composite fringe patterns are then subjected to segmented normalization processing to generate a training dataset. A dual-context scale-aware network construction unit is provided, wherein the dual-context scale-aware network is a three-stage network composed of three cascaded improved U-Nets, which sequentially realize stripe restoration, frequency separation and phase shift prediction functions; each improved U-Net includes an encoding layer, a decoding layer, an attention module and an adversarial training module, and the encoding layer introduces a spatial frequency fusion attention module; The model training unit is used to train the dual-context scale perception network using the training dataset until the loss converges, thereby obtaining a mature model that can reconstruct a complete multi-frequency multi-step phase-shifting stripe image from a single-frame Haar composite stripe image. The measurement data acquisition unit is used to project a single-frame Haar composite stripe pattern onto the colored object to be measured, and to simultaneously acquire deformed Haar composite stripe images through two grayscale cameras. The fringe prediction and phase calculation unit is used to input the deformed Haar composite fringe image into the mature model, output a complete multi-frequency multi-step phase-shifted fringe image, then use the phase-shifting method to calculate the wrapped phase of each frequency, and use the multi-frequency heterodyne method to perform phase unwrapping to obtain the absolute phase map of the two grayscale cameras. The binocular 3D reconstruction unit is used to perform subpixel-level matching on the absolute phase map to obtain a disparity map. Combined with the calibration parameters, it calculates the 3D coordinates through the principle of binocular stereo vision to complete the shape reconstruction of the colored object under test.

[0019] The present invention also provides a single-frame 3D reconstruction device for color objects with anti-color interference, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement any of the above-described methods for single-frame 3D reconstruction with anti-color interference.

[0020] The present invention also provides a computer-readable storage medium storing a computer program that can be executed by a processor of the device in which the computer-readable storage medium is located, to implement any of the color interference-resistant single-frame 3D reconstruction methods described above.

[0021] Compared with the prior art, the present invention has at least the following advantages: (1) Measurement efficiency is greatly improved. This invention only needs to project and acquire a single Haar-like composite fringe pattern, and then the complete multi-frequency multi-step phase-shift fringe sequence can be predicted and reconstructed through the network. Compared with the 12 images required by the traditional three-frequency four-step phase-shift method, the number of image acquisitions is reduced by 11, and the total measurement time is significantly shortened, which can meet the efficiency requirements of online rapid detection in industrial sites.

[0022] (2) High phase prediction accuracy. The Haar composite fringe pattern (HCFP) used in this invention has clear periodic boundary features, which is beneficial for the network to perform accurate frequency separation. Experimental results show that the mean absolute error (MAE) of the absolute phase predicted by the dual context scale sensing network is about 0.05 rad, and can be as low as 0.004 rad in local areas. Its accuracy is close to the measurement level of the traditional multi-frame phase shift method, which fully verifies the feasibility and reliability of the single-frame prediction scheme.

[0023] (3) Strong adaptability to complex color scenes. This invention can accurately measure the surface morphology of red, green, blue and various mixed color objects, effectively solving the problem of measurement failure of traditional structured light methods when facing highly saturated and highly reflective colored objects, and expanding the application scope of structured light three-dimensional measurement technology.

[0024] (4) Excellent three-dimensional reconstruction accuracy. In the verification experiment of standard test pieces, the root mean square error (RMSE) of the reconstruction of the standard ceramic plane is about 0.023 mm, the RMSE of the radius measurement of the standard ceramic ball is about 0.045 mm, and the RMSE of the center distance measurement of the ball is about 0.054 mm. All indicators meet the accuracy requirements of industrial precision testing and have good engineering application value.

[0025] (5) Low hardware cost and strong anti-interference capability. This invention adopts a grayscale stripe encoding scheme, which only requires two grayscale cameras to complete data acquisition, eliminating the need for expensive color cameras and reducing hardware costs by more than 30%. At the same time, the grayscale encoding method fundamentally avoids the inter-channel color crosstalk problem that is common in traditional RGB color encoding, effectively eliminating the interference of the surface color of the measured object on the measurement stripes, and significantly improving the system's adaptability to objects of different colors and measurement robustness. Attached Figure Description

[0026] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a single-frame 3D reconstruction method against color interference according to the present invention; Figure 2 A flowchart for creating the dataset of this invention; Figure 3 This is a structural diagram of the SFFAB of the present invention; Figure 4 This is a structural diagram of the ATF of the present invention; Figure 5 This is a comparison chart showing the first-stage prediction results of the three-color plane in this invention. Figure 6 This is a comparison chart showing the first-stage prediction results of the color face painting method of this invention. Figure 7 This is a diagram illustrating the measurement process of the car model 1 of the present invention. Figure 8 This is a diagram illustrating the measurement process of the car model 2 of this invention. Figure 9 This is a photograph of the standard four-step stepped steel component used in the measurement of the standard four-step stepped steel component of this invention. Figure 10 This is a three-dimensional point cloud diagram of the measurement process in the standard four-step stepped steel component measurement of this invention; Figure 11 This is a thermal distribution diagram of the measurement error in four planes during the standard four-step stepped steel component measurement of this invention. Figure 12 This is a root mean square error diagram of the four planes in the standard four-step measurement of stepped steel parts of this invention; Figure 13 This is a graph showing the measurement results of the standard ceramic ball of this invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example (I) Hardware System Setup The hardware components of the binocular structured light measurement system used in this invention are as follows: - Cameras: 2 ImagingSourceDMK23U274 grayscale cameras, 1600×1200 pixels resolution, 30fps frame rate; - Projector: V1i digital light processing projector, resolution 1600×1200 pixels; - Computer: Intel Core i7-7800X CPU, NVIDIA GeForce GTX 3060 Ti 12GB GPU; -System parameters: Field of view 240mm×180mm, working distance (from camera to the object being measured) approximately 500mm.

[0030] (II) Detailed Implementation Steps like Figure 1 As shown: S1: Binocular system calibration.

[0031] Fix the relative positions of the binocular camera and the projector, and print a 9×12 checkerboard calibration board (square side length 10mm); move the calibration board to 15 different poses (including different angles, distances, and tilts), and simultaneously acquire calibration images from the left and right cameras; use Zhang Zhengyou calibration method to solve for the camera's intrinsic parameters (focal length, principal point coordinates), extrinsic parameters (rotation matrix, translation vector), and radial distortion coefficients; project the checkerboard pattern through the projector to complete the calibration parameters of the projector and the binocular camera, and obtain all the calibration parameters required for the system's 3D reconstruction.

[0032] S2: Image capture and preprocessing.

[0033] This step aims to prepare a high-quality dataset for network training, and the process is as follows: Figure 2 As shown.

[0034] 1. Stripe generation and data acquisition: Select three frequencies =80、 =70、 =61, which meets the frequency selection criteria. =(80 70) (70 61)=1; Generate basic sine fringes:

[0035] The spatial frequency of the stripe.

[0036] Converting sinusoidal fringes into binary square waves using a Haar-like transform:

[0037] Superposition and normalization to generate HCFP:

[0038] Ultimately, the grayscale values ​​of HCFP are only four discrete values: 50, 110, 170, and 230, with obvious grayscale jumps at the periodic boundaries.

[0039] For each sample, perform the following data collection operations: (1) Project three-frequency four-step phase-shifting fringes (80 / 70 / 61 frequencies) and capture 12 phase-shifting fringe images as reference annotation data for subsequent three-dimensional reconstruction; (2) Keep the object still, project Haar composite stripes, and capture one composite stripe image as the raw input data for single-frame measurement; (3) Three uniform grayscale images with projected grayscale values ​​of 50, 140, and 230 were used to capture the corresponding grayscale images, which were then used as calibration data for segmented normalization, such as... Figure 2 The segmented normalization process in part a is shown below. All acquired images are stored in a lossless format to ensure that the original grayscale information is not lost.

[0040] 2. Data Preprocessing like Figure 2 The process of establishing the piecewise normalized training dataset for part b is shown below. The collected original dataset is preprocessed by a combination of Gamma correction and piecewise normalization to generate the preprocessed dataset. The specific processing steps are as follows: (1) Gamma correction: The system-calibrated Gamma coefficient is used to perform inverse Gamma compensation on all original captured images (composite stripes, phase-shifted stripes, uniform grayscale images) to eliminate the nonlinear distortion of the system and obtain a linearized grayscale image; (2) Segmented normalization: Based on the calibration data of the uniform grayscale image, the composite stripe image after Gamma correction is segmented and normalized to eliminate the grayscale deviation caused by color reflectance and obtain a high-quality composite stripe image after calibration. Figure 5 and Figure 6The effect of this step is demonstrated. As can be seen, after Gamma correction and piecewise normalization, the gray-level jumps caused by different object colors in the original image are effectively eliminated, and the stripes restore the ideal gray-level distribution, providing high-quality input and labels for network training.

[0041] S3: Construction of Dual Context Scale Aware Network (DCSA-Net).

[0042] The Dual Context Scale Awareness Network (DTA) is based on the classic U-Net framework and improves upon it by introducing a Spatial Frequency Fusion Attention Module (SFFAB) and an Adversarial Training Framework (ATF) to address the stripe inpainting requirements of objects with complex reflectivity. The network consists of four parts: an encoding layer, a decoding layer, an attention module, and an adversarial training module. The input is a single-frame Haar-type composite stripe image (1 channel), and the output is 12 three-frequency, four-step phase-shifted stripe images (12 channels), which can be directly used for subsequent phase extraction and 3D reconstruction. It also employs a three-level "one-to-three" network structure to achieve a progressive task completion of "stripe inpainting → frequency separation → phase shift prediction".

[0043] (1) Improvements to the encoding and decoding layers of the U-Net framework Retaining the encoder-decoder symmetric structure and skip connections of U-Net, the network depth is set to 5 layers to adapt to the resolution and feature extraction requirements of striped images. In the encoding layers, each layer consists of two 3×3 convolutional blocks (BN+ReLU) and one 2×2 max-pooling layer. The convolution operation can be represented as:

[0044] This is the output feature map after the i-th convolutional layer operation. Let i be the input feature map of the i-th layer. The weights are 3x3 convolutional kernels. This is for batch normalization operations.

[0045] Convolutional blocks perform feature extraction (the first convolutional block doubles the number of channels, and the second convolutional block keeps the number of channels unchanged), and pooling layers perform downsampling, gradually compressing the spatial dimension and enriching the feature representation. The pooling operation can be represented as:

[0046] This is the output feature map after the i-th pooling operation. Let be the input feature map after the i-th convolutional layer operation. This is a max pooling operation.

[0047] After each encoding stage, the feature maps are further enhanced by the SFFAB module, and the original information is preserved through residual connections:

[0048] The i-th layer is the final output feature after processing by the SFFAB module and residual connection. The input features are those after the i-th layer pooling operation. This is the processing function for the spatial frequency fusion attention module.

[0049] The last encoding layer omits pooling and directly feeds the features into the decoding layer; in the decoding layer, each layer consists of one 5×5 transposed convolutional upsampling block;

[0050] This is the output feature map after upsampling at layer i. For a 2x upsampling operation, The weights are 5x5 transposed convolution kernels.

[0051] Then, the SFFAB output of the corresponding layer of the encoder is concatenated with the upsampled features through skip connections:

[0052] This is the feature map after concatenation of the i-th layer. For the upsampled features of the i-th layer decoder, The features are those of the i-th layer encoder after processing by the SFFAB module. This is for channel splicing operations.

[0053] Finally, it is refined by 3×3 convolution, BN, and ReLU:

[0054] The output features of the i-th layer decoder after convolutional refinement are... The input features are the concatenated features of the i-th layer. represents the weights of the 3x3 convolution kernel in the i-th layer.

[0055] The upsampling block doubles the feature space dimension and halves the number of channels. The feature fusion block uses skip connections to concatenate the shallow detail features of the encoding layer with the deep semantic features of the decoding layer, achieving multi-scale feature fusion. In the output layer, a 1×1 convolution is used to map the features of the decoding layer to the target number of channels, outputting the repaired composite stripes, the separated three-frequency stripes, and the predicted phase-shifted stripes, respectively.

[0056] (2) Spatial Frequency Fusion Attention Module (SFFAB) To address the issues of lost local details and insufficient global context awareness in the fringes of complex reflective objects, an SFFAB module is introduced after each layer of the coding layer. This module achieves the fusion of spatial details and frequency domain features through multi-branch feature extraction and dynamic attention weighting.

[0057] The SFFAB module structure comprises four parts: parallel feature extraction branch, feature fusion, attention weight generation, and dynamic fusion. In the parallel branch section, three parallel branches are designed to extract features at different scales: a 3×3 dilated convolution branch extracts local detail features.

[0058] Output map for local features. The weights are local convolution kernels of size 3×3. This is the input feature map.

[0059] 5×5 dilated convolution branches extract global contextual features:

[0060] This is the global feature output map. The weights are 5×5 dilated convolution kernels. This is a dilated convolution operation with an inflation rate of 2.

[0061] Window attention branch extracts long-range dependency features:

[0062] The feature map output by the attention mechanism. For multi-head attention processing functions, These are the query matrix, key matrix, and value matrix, respectively.

[0063] In the feature fusion section, the outputs of the three branches are concatenated to obtain multi-scale fused features:

[0064] The fused feature map These are local features (from the 3×3 convolution branch). These are global features (from the dilated convolution branch). These are attention features (derived from multi-head attention branches). The spatial dimensions of the feature map. The number of channels after splicing.

[0065] The attention weight generation part uses global average pooling (GAP) and global max pooling (GMP) to capture global statistical information of the fused features:

[0066] This is the output after global average pooling. The height of the feature map, The width of the feature map. The fused feature map is in the first The pixel value at row i and column j.

[0067]

[0068] This is the output after global max pooling.

[0069] Channel-level attention weights are generated using a multilayer perceptron (MLP) to achieve dynamic weighting of different features:

[0070] This is the weight matrix (dimension reduced) for the first fully connected layer. This is the weight matrix for the second fully connected layer.

[0071] Finally, in the dynamic fusion process, the attention weights are multiplied by the channels of the fused features, and then fused with the original encoded features through residual connections to ensure that no feature information is lost.

[0072] This is the final output of the entire module.

[0073] The SFFAB module dynamically adjusts the weights of features at different scales based on the grayscale distribution and detail features of the stripes. It enhances the expression of local features in the color boundaries and detail areas of complex reflective objects, and strengthens the perception of global features in smooth areas, effectively improving the network's feature extraction capability for complex stripes. The SFFAB module structure is as follows: Figure 3 As shown.

[0074] (3) Adversarial Training Framework (ATF) To improve the perceived realism and detail reproduction of the restored stripes, the concept of Generative Adversarial Networks (GANs) is introduced to construct an adversarial training framework, consisting of a generator (the main body of DCSA-Net) and a discriminator. The network output is optimized through adversarial training between the two, as shown in the structure below. Figure 4 As shown.

[0075] The generator, which is an encoder-decoder + SFFAB module of a dual-context scale-aware network, is responsible for restoring the input damaged stripes into ideal, distortion-free stripes. The discriminator is constructed using four 4×4 convolutional layers (BN + LeakyReLU) and is responsible for distinguishing whether the input stripes are "real ideal stripes (annotated data)" or "generated restored stripes (model output)", with the output being a probability value between 0 and 1. :

[0076] The output probability of the discriminator for input x. As the input to the discriminator, The weights of the 4x4 convolution kernel in the l-th layer of the discriminator are... The Sigmoid activation function maps the output to the (0, 1) interval.

[0077] Finally, the adversarial loss is presented in the form of a conditional GAN:

[0078] To calculate the expectation of the real data (x, y), To calculate the expectation of the generated data x, The fake image generated by the generator from the input x.

[0079] Combining pixel-level L2 loss and adversarial loss, the mathematical model is as follows:

[0080] in, To achieve pixel-level L2 loss, the global structural consistency between the repaired stripes and the ideal stripes is ensured. To combat the loss, the generator and discriminator work together to improve the realism of local details and perceived quality of the repaired stripes.

[0081] (4) Three-level cascaded "one-to-three" network structure like Figure 2 As shown in the process of establishing the training dataset for frequency separation and phase shift prediction in part c, the dual-context scale-aware network adopts a three-level cascaded "one-to-three" architecture to realize the progressive task of "fringe restoration → frequency separation → phase shift prediction": The first stage network takes a Gamma-corrected Haar composite fringe image as input and labels the darkest and brightest baseline images after correction, as well as the normalized Haar composite fringe image after normalization. It aims to predict from the corrected HCFP image to the ideal normalized image. The second-stage network: The input is the ideal normalized Haar composite fringe image predicted by the first-stage network, and the label is the first phase shift image of each frequency in the three-frequency four-step phase shift after normalization, realizing frequency separation from a single ideal normalized HCFP image to three first phase shift fringe images of different frequencies. The third-stage network takes the first phase shift fringe image for each frequency predicted by the second-stage network as input and labels the remaining three phase shift images for that frequency after segmented normalization. This enables the prediction from the first phase shift fringe of a single frequency to other phase shift fringes in multiple frames at that frequency.

[0082] All three stages adopt a "one-to-three" input-output structure, progressing step by step, and finally reconstructing a complete multi-frequency, multi-step phase-shifting fringe image from a single-frame Haar composite fringe image.

[0083] S4: Building a high-quality training dataset and training the model.

[0084] To ensure the diversity and representativeness of the dataset, six commonly used industrial colors were selected. Using simple planes, complex freeform surfaces (theatrical masks), and sharp-edge structures (car models) as the basic carriers, multi-color composite models were constructed by combining 3 to 6 different colors, resulting in 12 basic color models. These basic models were then randomly arranged into 36 different color distribution models. Each model was randomly placed in 20 different poses within the measurement field of view, ultimately capturing 720 sets of complex reflectivity object samples with different color distributions and poses. For each sample, an HCFP was first projected, and images from the left and right cameras were acquired. Then, 12 striped images with three frequencies and four-step phase shifts were projected sequentially as labels. Three uniform grayscale images with grayscale values ​​of 50, 140, and 230 were projected, and the corresponding grayscale images were captured as segmented normalized calibration data. The construction steps are as follows: (1) Annotation data processing: The same segmented normalization process was performed on the 12 phase shift fringe images after Gamma correction to obtain ideal phase shift fringe annotation data, ensuring the grayscale consistency between the annotation data and the input data; (2) Data pair construction: The original composite fringe image after Gamma correction is used as the model input, the calibrated composite fringe image after segmented normalization is used as the annotation for the first stage repair task, and the phase shift fringe image after segmented normalization is used as the annotation for the subsequent frequency separation and phase shift prediction tasks, thus constructing a three-level progressive input-annotation data pair.

[0085] After processing, all data were unified in resolution and normalized in grayscale (to the [0,1] range). The dataset was then divided into training, validation, and test sets in an 8:1:1 ratio, with 576 training sets, 72 validation sets, and 72 test sets. This ensured that the dataset division conformed to general machine learning principles and avoided overfitting. All image pixel values ​​were normalized to [0,1] and cropped to 1028×968 pixels to accelerate training convergence.

[0086] The specific training design conditions are as follows: (1) Training hardware and software environment Model training was conducted on a desktop computer with an Intel Core i7-9700 CPU and an NVIDIA GeForce RTX 4060 Ti 16GB GPU to ensure computational efficiency. The software environment was Python 3.8, with the network built on the PyTorch 2.10 deep learning framework. PyCharm was used as the development environment, and OpenCV and NumPy were used for image preprocessing and data reading and writing.

[0087] (2) Training hyperparameter settings To ensure effective training and fast convergence of the model, the following core hyperparameters were determined after multiple experiments and optimizations. The optimizer was the Adam optimizer, and the learning rate was set to... The weight decays to Batch size is set to 4 to adapt to GPU memory capacity. Number of training epochs: 400. Early stopping is used; training stops if the validation set loss does not decrease for 50 consecutive epochs to avoid overfitting. Adversarial training strategy: train the generator for 2 epochs and the discriminator for 1 epoch to maintain the training balance between the two. Data loading: A data generator is used to load and augment batch data in real time to improve training efficiency.

[0088] (3) Monitoring of the training process During training, the changes in loss values ​​on the training and validation sets are monitored in real time, and the repair effect is visualized. The L2 loss, adversarial loss, and total loss for each round are recorded, loss curves are plotted, and the convergence status of the model is determined.

[0089] Furthermore, samples from the validation set are randomly selected in each round, and the model's repair results are output and compared with the labeled data to intuitively judge the repair effect. Finally, the model with the lowest loss on the validation set is saved as the optimal model for subsequent stripe repair and 3D reconstruction experiments.

[0090] S5: Measurement data acquisition.

[0091] The projector projects a single HCFP pattern onto the object under test, and the left and right cameras simultaneously acquire the deformed stripe images. The acquired images are input into a trained dual-context scale perception network. The first stage outputs a Haar composite image with the color effect removed. The second stage network predicts the first phase shift image of three frequencies based on the output of the first stage. The third stage network predicts the other three phase shift images of each frequency based on the output of the second stage. The total prediction time is about 0.6 seconds.

[0092] S6: Fringe prediction and phase calculation.

[0093] The wrap-around phase at each frequency is calculated using a four-step phase-shifting method:

[0094] Let be the gradient magnitude of the nth region in direction i. Let i be the sinusoidal component in direction i. Let i be the cosine component of the direction i.

[0095] Phase unwrapping is performed using the three-frequency heterodyne method: first, the difference frequency phase is calculated. and Then calculate the globally unique phase. Guided by this, the highest frequency is obtained by unpacking the package step by step. =80 absolute phase diagram.

[0096] S7: 3D reconstruction.

[0097] Subpixel-level matching of the absolute phase maps of the left and right cameras is performed to obtain a disparity map; combined with the intrinsic and extrinsic parameters calibrated by the system, the three-dimensional coordinates of each pixel are calculated using the principle of triangulation to generate a three-dimensional point cloud; the three-dimensional point cloud is filtered and denoised to obtain the final three-dimensional shape of the object.

[0098] (III) Accuracy Verification Examples To verify the accuracy of this method, the following experiments were conducted, and the results are as follows: Figures 7 to 13 As shown.

[0099] 1. Measurement effect of colored objects Figure 7 and Figure 8 The measurement process for two colored car models is demonstrated. Results show that the proposed method effectively suppresses color-induced stripe distortion, generates a continuous and smooth absolute phase map, and has a mean absolute phase error (MAE) of less than 0.051 rad. Compared with traditional methods, the reconstructed 3D point cloud produced by this method is artifact-free and accurately restores the curvature and edge details of the object's surface.

[0100] 2. Measurement accuracy of standard parts Measurement of stepped steel components: such as Figures 9-12 As shown, ten repeated measurements were performed on a standard four-step steel staircase. The average step heights obtained were 10.0351 mm, 5.0212 mm, and 2.5364 mm, with corresponding average errors of 0.0351 mm, 0.0212 mm, and 0.0364 mm, respectively. The root mean square error (RMSE) of each step surface was within 0.0230 mm.

[0101] Ceramic ball measurement: such as Figure 13 As shown, measurements were taken on a standard ceramic sphere. In the figure, a represents the actual standard part and its true parameters, b represents the 3D reconstructed point cloud and the measured values, and c represents the histogram of the fitting error distribution. The measured center-to-center distance was Dm = 100.1926 mm, with an RMSE of 0.0536 mm. The measured sphere radii were RAm = 19.0174 mm and RBm = 19.0101 mm, with corresponding RMSEs of 0.0461 mm and 0.0438 mm, respectively.

[0102] Experimental results demonstrate that the proposed method can achieve high-quality 3D precision measurement.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for single-frame 3D reconstruction against color interference, characterized in that, Includes the following steps: S1. Binocular vision system calibration: Adjust and fix the relative positions of the two grayscale cameras and the digital projector, acquire calibration images, and use Zhang Zhengyou calibration method to obtain the calibration parameters of the grayscale cameras. S2. Image capture and preprocessing: The brightness reference pattern of the Haar-like composite fringe pattern, the 12 sinusoidal fringe patterns with three-frequency four-step phase shift, and the third-order Haar-like composite fringe pattern are projected onto the surface of the colored object and deformed images are acquired. The deformed images are corrected for Gamma distortion. At the same time, the 12 sinusoidal fringe patterns and the Haar-like composite fringe patterns are segmented and normalized to generate a training dataset. S3. Construction of Dual Context Scale Awareness Network: The dual context scale awareness network is a three-stage network composed of three improved U-Nets cascaded together, which sequentially realizes the functions of stripe restoration, frequency separation and phase shift prediction; each improved U-Net includes an encoding layer, a decoding layer, an attention module and an adversarial training module, and the encoding layer introduces a spatial frequency fusion attention module; S4. Model Training: Using the training dataset, train the dual-context scale perception network until the loss converges to obtain a mature model that can reconstruct a complete multi-frequency multi-step phase-shifting stripe image from a single-frame Haar composite stripe image. S5. Measurement data acquisition: Project a single-frame Haar composite stripe pattern onto the colored object to be measured, and simultaneously acquire deformed Haar composite stripe images using two grayscale cameras; S6. Fringe Prediction and Phase Calculation: Input the deformed Haar composite fringe image into the mature model and output a complete multi-frequency multi-step phase-shifted fringe image. Then, use the phase-shifting method to calculate the wrapped phase of each frequency and use the multi-frequency heterodyne method to unwrap the phase and obtain the absolute phase map of the two grayscale cameras. S7. Binocular 3D Reconstruction: Subpixel-level matching is performed on the absolute phase map to obtain a disparity map. Combined with the calibration parameters, the 3D coordinates are calculated using the principle of binocular stereo vision to complete the shape reconstruction of the colored object under test.

2. The method according to claim 1, characterized in that, In S2, the segmented normalization process specifically involves: using the mid-brightness grayscale value of the Haar composite stripe image as the segmentation threshold, dividing the grayscale range into low grayscale segments and high grayscale segments; for the low grayscale segments, linearly mapping the corrected grayscale values ​​to a preset ideal low grayscale range according to their relative positions within the segment; for the high grayscale segments, linearly mapping the corrected grayscale values ​​to a preset ideal high grayscale range according to their relative positions within the segment.

3. The method according to claim 1, characterized in that, In S2, the training dataset is constructed as follows: based on object models of multiple colors, no less than 720 sets of sample images are collected by combining different colors and changing different poses within the measurement field of view; all sample images are normalized to the 0~1 range and cropped to a uniform size to obtain the training dataset.

4. The method according to claim 1, characterized in that, In S3, the dual-context scale-aware network is optimized using an adversarial training framework. Specifically, it is optimized using a combined loss function, which includes pixel-level L2 loss to ensure global structural consistency and adversarial loss to improve the realism of local details.

5. The method according to claim 1, characterized in that, In S3, the dual-context scale-aware network includes: The first-stage network takes a Gamma-corrected Haar-like composite stripe image as input and an ideal normalized image as a label for supervised training, and outputs a normalized Haar-like composite stripe image. The second-stage network takes the normalized Haar composite fringe image output by the first-stage network as input and uses the first phase shift fringe image of each frequency in the three-frequency four-step phase shift as a label for supervised training, and outputs the first phase shift fringe image of each of the three frequencies. The third-stage network takes the first phase-shift fringe image at each frequency output by the second-stage network as input, and uses the remaining three phase-shift fringe images at that frequency as labels for supervised training, outputting a complete four-step phase-shift fringe image at that frequency.

6. The method according to claim 1, characterized in that, In S6, the specific process of phase unwrapping using the multi-frequency heterodyne method is as follows: first calculate the difference frequency phase, then calculate the globally unique phase, and use this as a guide to unwrap the phase step by step to obtain the absolute phase diagram of the highest frequency.

7. A single-frame 3D reconstruction device resistant to color interference, characterized in that, include: The binocular vision system calibration unit is used to adjust and fix the relative positions of the two grayscale cameras and the digital projector, acquire calibration images, and obtain the calibration parameters of the grayscale cameras using the Zhang Zhengyou calibration method. The image capture and preprocessing unit is used to project the brightness reference pattern of the Haar-like composite fringe pattern, the 12 sinusoidal fringe patterns of the three-frequency four-step phase shift, and the third-order Haar-like composite fringe pattern onto the surface of a colored object and acquire deformed images. The deformed images are then subjected to Gamma distortion correction, and the 12 sinusoidal fringe patterns and the Haar-like composite fringe patterns are then subjected to segmented normalization processing to generate a training dataset. A dual-context scale-aware network construction unit is provided, wherein the dual-context scale-aware network is a three-stage network composed of three cascaded improved U-Nets, which sequentially realize the functions of stripe restoration, frequency separation and phase shift prediction; each improved U-Net includes an encoding layer, a decoding layer, an attention module and an adversarial training module, and the encoding layer introduces a spatial frequency fusion attention module; The model training unit is used to train the dual-context scale perception network using the training dataset until the loss converges, thereby obtaining a mature model that can reconstruct a complete multi-frequency multi-step phase-shifting stripe image from a single-frame Haar composite stripe image. The measurement data acquisition unit is used to project a single-frame Haar composite stripe pattern onto the colored object to be measured, and to simultaneously acquire deformed Haar composite stripe images through two grayscale cameras. The fringe prediction and phase calculation unit is used to input the deformed Haar composite fringe image into the mature model, output a complete multi-frequency multi-step phase-shifted fringe image, then use the phase-shifting method to calculate the wrapped phase of each frequency, and use the multi-frequency heterodyne method to perform phase unwrapping to obtain the absolute phase map of the two grayscale cameras. The binocular 3D reconstruction unit is used to perform sub-pixel-level matching on the absolute phase map to obtain a disparity map. Combined with the calibration parameters, it calculates the 3D coordinates through the principle of binocular stereo vision to complete the shape reconstruction of the colored object under test.

8. A single-frame 3D reconstruction device for color objects, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the anti-color interference single-frame 3D reconstruction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device in which the computer-readable storage medium is located, to implement the anti-color interference single-frame three-dimensional reconstruction method as described in any one of claims 1 to 6.

Images