A color structured light color crosstalk correction method of unsupervised deep learning
By using unsupervised deep learning methods to correct color crosstalk in color structured light images, the problem of large errors in 3D measurement of color structured light is solved, and high-precision 3D topography restoration is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2022-10-12
- Publication Date
- 2026-07-03
AI Technical Summary
In existing color structured light 3D measurement technology, color crosstalk leads to large measurement errors and makes it impossible to accurately recover the 3D surface morphology of the measured object.
An unsupervised deep learning method is adopted to perform color crosstalk correction on color structured light images using a deep neural network. This is achieved by generating composite color phase-shifted fringe images, acquiring deformed color fringe images, separating RGB channel images, performing correction using a deep neural network, and solving the predicted phase using the phase-shifting method, ultimately restoring the three-dimensional morphology of the object under test.
It achieves fast and effective color crosstalk correction, reduces measurement phase error, improves the accuracy and generalization performance of 3D topography measurement, and reduces the cost of dataset label production.
Smart Images
Figure CN115615358B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of structured light three-dimensional measurement, and specifically to an unsupervised deep learning-based method for color crosstalk correction in structured light. Background Technology
[0002] Structured light 3D measurement technology boasts advantages such as non-contact operation, high sensitivity, and high precision, and has been widely applied in industries such as industrial inspection, reverse engineering, and intelligent manufacturing. With the rapid development of computer vision, structured light 3D measurement technology is currently evolving towards high-speed and real-time measurement. Traditional phase-shifting methods require at least three structured light images to reconstruct a measurement result, thus severely limiting its application in high-speed dynamic measurement. Color structured light methods load three phase-shifted images into RGB channels respectively, forming a single color structured light image, thereby enabling the reconstruction of measurement results from a single image, thus meeting the current trend in high-speed dynamic measurement applications.
[0003] However, since the wavelength distribution of natural light is continuous, the distribution of a specific color's wavelength is within a small range. Therefore, the responses of projectors or cameras to the RGB colors inevitably overlap, leading to errors and interference when separating the RGB channels of structured light. This phenomenon is known as color crosstalk. Clearly, without eliminating and correcting color crosstalk in the structured light image after channel separation, the calculated phase distribution of the measured object will inevitably have serious errors, making it impossible to recover the accurate three-dimensional surface morphology. Therefore, how to perform high-quality correction of color crosstalk has become a crucial challenge and an important research direction in structured light measurement technology. Summary of the Invention
[0004] The purpose of this invention is to provide an unsupervised deep learning-based method for color crosstalk correction in structured light, so as to achieve color crosstalk correction in structured light quickly and effectively.
[0005] To achieve the above objectives, the present invention employs the following technical solution:
[0006] An unsupervised deep learning-based method for color crosstalk correction in structured light, comprising:
[0007] Using a computer to generate a composite color phase-shifting fringe structured light image C And transmit it to the measurement system;
[0008] The composite color phase-shifting fringe structured light image I is projected using the projection module of the measurement system. C The image is projected onto the surface of the object being measured, while simultaneously, the color camera of the measurement system acquires a highly modulated, distorted color stripe image of the object from another angle. color ;
[0009] For the deformed color stripe image I color The RGB three channels are separated, and three deformed grayscale striped structured light images I are extracted respectively. R I G and I B Image I R I G and I B The inputs are respectively fed into three deep neural network modules for color crosstalk correction;
[0010] The three deep neural networks output three predicted color crosstalk corrected grayscale images I' R , I' G and I' B The predicted phase Φ' is calculated using the phase-shifting method based on these three grayscale images.
[0011] Computer inverse projection simulation is performed using the predicted phase Φ' to obtain the deformed color stripe image I corresponding to the phase result. rep_color ;
[0012] Using the acquired deformed color stripe image I color The deformed color stripe image I obtained by inverse projection simulation rep_color Construct the loss function for the deep neural network and calculate its loss value;
[0013] When the loss value calculated by the loss function reaches the minimum value, the final corrected crosstalk corrected ideal fringe image is obtained; the final ideal phase Φ is calculated by combining the corrected ideal fringe image with the phase shift method, and the true three-dimensional morphology of the measured object is restored.
[0014] Furthermore, a computer-generated color structured light image can be represented as:
[0015]
[0016] Where I C The composite color phase-shifted fringe structured light image contains three channel images, denoted by I1, I2, and I3, respectively. Each channel is a grayscale sinusoidal fringe phase-shifted image; f represents the frequency of the sinusoidal fringe, x represents the horizontal coordinate index of the image, 2nπ / 3 represents its phase shift amount, and n represents the nth channel.
[0017] Furthermore, the measurement system includes a DLP projection module, a color industrial camera, and a computer. The DLP projection module's optical axis is at a 30-degree angle to the object being measured, projecting the structured light image I. C The light axis of the color industrial camera is perpendicular to the surface of the object being measured and the image is projected onto the object.
[0018] Furthermore, the color camera captures a highly modulated, distorted color stripe image of the object under test. color It can be represented as:
[0019]
[0020] Among them, I R I G I B For I color The RGB three-channel image, where Φ represents the true phase distribution of the measured object, is an unknown quantity that needs to be solved during the three-dimensional measurement process.
[0021] Furthermore, the system responsible for processing the three deformed grayscale stripe structured light images I R I G and I B The three deep neural network sub-modules are all U-shaped networks, consisting of an encoder and a decoder. The encoder has five layers from top to bottom, with each layer connected by feature extraction downsampling to progressively reduce the image size. Each layer contains three sequentially arranged convolutional layers connected by residual blocks. The output of the last convolutional layer in the previous layer serves as the input to the first convolutional layer in the next layer. The decoder has a symmetrical structure to the encoder, with five layers. Each layer is connected by feature extraction upsampling to progressively restore the original image size, ultimately obtaining the predicted output. The output of the last convolutional layer in the next layer serves as the input to the first convolutional layer in the previous layer. The bottom layer of the encoder and the bottom layer of the decoder are connected by an attention mechanism module.
[0022] Furthermore, the weights of the three deep neural subnetworks cannot be shared.
[0023] Furthermore, the step of solving for the predicted phase Φ' based on the three grayscale images using the phase-shifting method includes:
[0024] Using grayscale image I' R , I' G and I' B Substituting into the phase-shifting method formula, the predicted phase Φ' is obtained:
[0025]
[0026] Furthermore, by performing computer inverse projection simulation using the predicted phase Φ', the deformed color stripe image I corresponding to the phase result can be obtained. rep_color The formula is as follows:
[0027]
[0028] Among them, I rep_R I rep_R ,I rep_RFor I rep_color An RGB three-channel image.
[0029] Furthermore, the loss function of a deep neural network is expressed as:
[0030]
[0031] Where x and y represent the horizontal and vertical coordinate indices of the image, and H and W represent the height and width of the image; λ R , λ G and λ B These are the weights of the loss values for the three RGB channels.
[0032] Furthermore, through network training, the three deep neural networks that minimized their loss values are saved as the final correction network; the image I... R I G and I B The input is a correction network, and the three crosstalk correction ideal stripe images corresponding to the three channels of RGB are used to correct the output of the correction network. The phase is solved by the phase shift method, and finally the obtained ideal phase Φ is obtained by nonlinear mapping to obtain the true three-dimensional shape of the object under test.
[0033] Furthermore, the nonlinear calibration model used in the nonlinear mapping is expressed as follows:
[0034]
[0035] Where h represents the three-dimensional topographic depth information, and a, b and c are calibration parameters, all of which are determined during the measurement system calibration process before measurement.
[0036] Compared with the prior art, the present invention has the following technical features:
[0037] 1. This scheme utilizes an unsupervised deep learning mechanism to correct the image distortion and aliasing caused by color crosstalk in colored structured light, thereby reducing the phase error of the measurement and achieving higher precision three-dimensional topography measurement. Compared with traditional color crosstalk correction methods, the method provided by this invention does not require complex crosstalk matrix estimation, but instead utilizes the powerful nonlinear fitting and prediction capabilities of deep neural networks for rapid correction.
[0038] 2. Compared to the traditional deep learning mechanism of one-to-one input-label mapping, the unsupervised learning mechanism provided by this invention can completely discard the label data in the dataset, greatly reducing the manual cost of dataset collection and creation in deep learning technology. Furthermore, the inverse projection simulation results used in the method provided by this invention are strictly simulated according to the physical model of the measurement principle. Therefore, compared with traditional deep learning mechanisms, the method of this invention is more interpretable and is not limited by the training dataset, applicable to any measurement scenario, thus exhibiting good generalization performance. Attached Figure Description
[0039] Figure 1 This is a schematic diagram of the color structured light measurement system used in this invention.
[0040] Figure 2 This is a flowchart of the unsupervised deep learning method of the present invention.
[0041] Figure 3 This is a diagram of the deep neural network structure of the present invention.
[0042] Figure labeling: 1-Color industrial camera, 2-DLP projection module, 3-Computer, 4-R-channel encoded grayscale stripe image, 5-G-channel encoded grayscale stripe image, 6-B-channel encoded grayscale stripe image, 7-Synthesized color encoded stripe image. Detailed Implementation
[0043] This invention provides an unsupervised deep learning-based method for color crosstalk correction in structured light. First, a color phase-shifted fringe structured light image is generated using a computer. Then, a highly modulated, deformed color fringe image of the object under test is acquired using a color camera, and the R, G, and B channels of this color image are separated to extract three grayscale images. These three grayscale images are then input into three deep neural network modules for color crosstalk correction. Subsequently, the deep neural networks output three predicted corrected grayscale images. Based on these three grayscale images, the predicted phase is calculated using the phase-shifting method. Computer inverse projection simulation is performed using this predicted phase to obtain the deformed color fringe image corresponding to the phase result. Finally, the loss value between the acquired deformed color fringe image and the deformed color fringe image obtained from the inverse projection simulation is calculated. The network parameters are iteratively optimized until the loss value is minimized, resulting in the ideal correction result. The unsupervised deep learning mechanism of this invention does not require the creation of large amounts of training data and their corresponding labels, is not dependent on a specific dataset, and significantly improves the operational efficiency and generalization performance of deep learning.
[0044] The specific implementation process of the present invention will be further described in detail below with reference to the accompanying drawings.
[0045] S1, Use a computer to generate a composite color phase-shifting fringe structured light image IC And transmit it to the measurement system:
[0046] Computer-generated color structured light images can be represented as:
[0047]
[0048] Where I C The composite color phase-shifted fringe structured light image contains three channel images, denoted by I1, I2, and I3, respectively. Each channel is a grayscale sinusoidal fringe phase-shifted image; f represents the frequency of the sinusoidal fringe, x represents the horizontal coordinate index of the image, 2nπ / 3 represents its phase shift amount, and n represents the nth channel.
[0049] S2, the composite color phase-shifting fringe structured light image I is projected using the projection module of the measurement system. C The image is projected onto the surface of the object being measured, while simultaneously, the color camera of the measurement system acquires a highly modulated, distorted color stripe image of the object from another angle. color The other angle refers to an angle that is different from the projection angle.
[0050] See Figure 1 The measurement system described in this embodiment includes a DLP projection module, a color industrial camera, and a computer. The DLP projection module's optical axis is at approximately a 30-degree angle to the object being measured, projecting the structured light image I. C The light axis of the color industrial camera is perpendicular to the surface of the object being measured and the image is projected onto the object.
[0051] I. Highly modulated distorted color stripe image of the object under test acquired by a color camera color It can be represented as:
[0052]
[0053] Among them, I R I G I B ( Figure 1 4, 5, and 6 in the text are I color ( Figure 1 The RGB three-channel image in section 7) shows that Φ represents the true phase distribution of the measured object, which is an unknown quantity that needs to be solved during the three-dimensional measurement process; the color image I color By extracting individual images from the RGB channels, three deformed grayscale striped structured light images I can be obtained. R I G and I B .
[0054] S3, for the deformed color stripe image I colorThe RGB three channels are separated, and three deformed grayscale striped structured light images I are extracted respectively. R I G and I B Image I R I G and I B The inputs are fed into three deep neural network modules for color crosstalk correction:
[0055] Responsible for processing three deformed grayscale stripe structured light images I R I G and I B The three deep neural network sub-modules are all U-shaped networks, consisting of an encoder and a decoder. The encoder has five layers from top to bottom, with each layer connected by feature extraction downsampling connections to progressively reduce the image size. Each layer contains three sequentially arranged convolutional layers connected by residual blocks. The output of the last convolutional layer in the previous layer serves as the input to the first convolutional layer in the next layer. The decoder has a symmetrical structure with five layers, with each layer connected by feature extraction upsampling connections to progressively restore the original image size, ultimately obtaining the predicted output. The output of the last convolutional layer in the next layer serves as the input to the first convolutional layer in the previous layer. The bottom layer of the encoder and the bottom layer of the decoder are connected by an attention mechanism module, which improves the network's attention to stripe edge features. Furthermore, corresponding layers in the encoder and decoder use skip connections to directly transmit feature maps of the same size, improving learning speed.
[0056] Due to the three deformed grayscale stripe structured light images I R I G and I B Although all three images are sinusoidal stripe patterns, the crosstalk effects of each image are different in magnitude and characteristics, so the weights of the three deep neural subnetworks cannot be shared.
[0057] S4, the three deep neural networks output three predicted color crosstalk corrected grayscale images I' R , I' G and I' B Based on these three grayscale images, the predicted phase Φ' is calculated using the phase-shifting method:
[0058] Among them, grayscale image I' R , I' G and I' B Substituting into the phase-shifting method formula, the predicted phase Φ' is obtained:
[0059]
[0060] S5. Using the predicted phase Φ', a computer inverse projection simulation can be performed to obtain the deformed color stripe image I corresponding to the phase result. rep_color This step includes:
[0061] The principle of computer inverse projection simulation is to treat the solved predicted phase as a known quantity and add it to the generation formula of the colored stripe structured light to obtain a deformed colored stripe image I modulated by the predicted phase. rep_color The generation formula is as follows:
[0062]
[0063] Among them, I rep_R I rep_R ,I rep_R For I rep_color The RGB three-channel image; theoretically, if the corrected grayscale image predicted by the network is accurate enough, the deformed color stripe structured light image I obtained by inverse projection simulation can be obtained. rep_color It will be compared with the actual acquired deformed color stripe image I color Almost identical.
[0064] S6, using the acquired deformed color stripe image I color The deformed color stripe image I obtained by inverse projection simulation rep_color Constructing the loss function for the deep neural network and calculating its loss value; this step includes:
[0065] Deformed color stripe image I obtained based on inverse projection simulation rep_color With the acquired deformed color stripe image I color Constraints are established to optimize and adjust the parameters of the deep neural network, thereby prompting it to output higher-quality correction results. The loss function of the deep neural network is defined as:
[0066]
[0067] Where x and y represent the horizontal and vertical coordinate indices of the image, and H and W represent the height and width of the image.
[0068] In practical calculations, the loss is calculated separately for each of the RGB channels of the two color stripe images, and then summed. The loss function can be further rewritten as:
[0069]
[0070] Where λ R , λ G and λ B The weights for the loss values of the three RGB channels are set according to the specific performance of the network.
[0071] S7, when the loss value calculated by the loss function reaches its minimum, the final corrected crosstalk-corrected ideal fringe image is obtained; the final ideal phase Φ is calculated using the corrected ideal fringe image combined with the phase shift method, and the true three-dimensional morphology of the object under test is restored; this step includes:
[0072] Through network training, the three deep neural networks that minimized the loss value are saved as the final correction network; image I R I G and I B The input is a calibration network, and the three crosstalk correction ideal fringe images corresponding to the three channels of RGB are used as the output of the calibration network. The phase is solved using the phase shift method in S4. Finally, the obtained ideal phase Φ is mapped to the true three-dimensional shape of the object under test through the following nonlinear calibration model:
[0073]
[0074] Where h represents the three-dimensional topographic depth information, and a, b and c are calibration parameters, all of which are determined during the measurement system calibration process before measurement.
[0075] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A color crosstalk correction method for structured light using unsupervised deep learning, characterized in that, include: Using a computer to generate a composite color phase-shifting fringe structured light image C And transmit it to the measurement system; The composite color phase-shifting fringe structured light image I is projected using the projection module of the measurement system. C The image is projected onto the surface of the object being measured, while simultaneously, the color camera of the measurement system acquires a highly modulated, distorted color stripe image of the object from another angle. color ; For the deformed color stripe image I color The RGB three channels are separated, and three deformed grayscale striped structured light images I are extracted respectively. R I G and I B Image I R I G and I B The inputs are respectively fed into three deep neural network modules for color crosstalk correction; The three deep neural networks output three predicted color crosstalk corrected grayscale images I' R , I' G and I' B The predicted phase Φ' is calculated using the phase-shifting method based on these three grayscale images. Computer inverse projection simulation is performed using the predicted phase Φ' to obtain the deformed color stripe image I corresponding to the phase result. rep_color ; Using the acquired deformed color stripe image I color The deformed color stripe image I obtained by inverse projection simulation rep_color Construct the loss function for the deep neural network and calculate its loss value; When the loss value calculated by the loss function reaches the minimum value, the final corrected crosstalk corrected ideal fringe image is obtained; the final ideal phase Φ is calculated by combining the corrected ideal fringe image with the phase shift method, and the true three-dimensional morphology of the measured object is restored. Responsible for processing three deformed grayscale stripe structured light images I R I G and I B All three deep neural network sub-modules are U-shaped networks, consisting of an encoder and a decoder. The encoder has five layers from top to bottom, with each layer connected by feature extraction downsampling to progressively reduce the image size. Each layer contains three sequentially arranged convolutional layers connected by residual blocks. The output of the last convolutional layer in the previous layer serves as the input to the first convolutional layer in the next layer. The decoder has a symmetrical structure to the encoder, with five layers. Each layer is connected by feature extraction upsampling to progressively restore the original image size, ultimately yielding the predicted output. The output of the last convolutional layer in the next layer serves as the input to the first convolutional layer in the previous layer. The bottom layer of the encoder and the bottom layer of the decoder are connected by an attention mechanism module. The loss function of a deep neural network is expressed as: ， Where x and y represent the horizontal and vertical coordinate indices of the image, and H and W represent the height and width of the image; λ R , λ G and λ B These are the weights of the loss values for the three RGB channels.
2. The unsupervised deep learning-based color structured light color crosstalk correction method according to claim 1, characterized in that, Computer-generated color structured light images can be represented as: ， Where I C The composite color phase-shifted fringe structured light image contains three channel images, denoted by I1, I2, and I3, respectively. Each channel is a grayscale sinusoidal fringe phase-shifted image; f represents the frequency of the sinusoidal fringe, x represents the horizontal coordinate index of the image, 2nπ / 3 represents its phase shift amount, and n represents the nth channel.
3. The unsupervised deep learning-based color structured light color crosstalk correction method according to claim 1, characterized in that, The measurement system includes a DLP projection module, a color industrial camera, and a computer; wherein, the optical axis of the DLP projection module is at a 30-degree angle to the object being measured to project the structured light image I. C The light axis of the color industrial camera is perpendicular to the surface of the object being measured and the image is projected onto the object.
4. The unsupervised deep learning-based color structured light color crosstalk correction method according to claim 2, characterized in that, I. Highly modulated distorted color stripe image of the object under test acquired by a color camera color It can be represented as: ， Among them, I R I G I B For I color The RGB three-channel image, where Φ represents the true phase distribution of the measured object, is an unknown quantity that needs to be solved during the three-dimensional measurement process.
5. The unsupervised deep learning-based color structured light color crosstalk correction method according to claim 1, characterized in that, The step of solving for the predicted phase Φ' based on the three grayscale images using the phase-shifting method includes: Using grayscale image I' R , I' G and I' B Substituting into the phase-shifting method formula, the predicted phase Φ' is obtained: 。 6. The unsupervised deep learning-based color structured light color crosstalk correction method according to claim 2, characterized in that, The computer inverse projection simulation using the predicted phase Φ' can obtain the deformed color stripe image I corresponding to the phase result. rep_color The formula is as follows: ， Among them, I rep_R I rep_R , I rep_R For I rep_color An RGB three-channel image.
7. The unsupervised deep learning-based color structured light color crosstalk correction method according to claim 1, characterized in that, Through network training, the three deep neural networks that minimized the loss value are saved as the final correction network; image I R I G and I B The input is a correction network, and the three crosstalk correction ideal stripe images corresponding to the three channels of RGB are used to correct the output of the correction network. The phase is solved by the phase shift method, and finally the obtained ideal phase Φ is obtained by nonlinear mapping to obtain the true three-dimensional shape of the object under test.
8. The unsupervised deep learning-based color structured light color crosstalk correction method according to claim 7, characterized in that, The nonlinear calibration model used in the nonlinear mapping is expressed as follows: ， Where h represents the three-dimensional topographic depth information, and a, b and c are calibration parameters, all of which are determined during the measurement system calibration process before measurement.