Method for on-line measurement of three-dimensional outer contour and defect detection of composite striped projection steel pipe and application thereof

Abstract

This invention discloses a method and application for online measurement and defect detection of the three-dimensional outer contour of a steel pipe using composite stripe projection. The steps include: first, calibrating the camera's intrinsic parameters using the standard Zhang Zhengyou calibration method; then, calibrating the camera-projector system by projecting a transversely and longitudinally orthogonal sinusoidal stripe structure; constructing a steel pipe surface dataset by combining a designed single-frame structured light and 20-step phase-shifting sinusoidal stripes; training three deep learning networks for phase extraction, absolute phase range acquisition, and phase unwrapping, respectively; and finally, reconstructing the steel pipe surface based on the final unwrapped phase and the system's calibration parameters, and comparing it with a standard steel pipe surface to identify defects. This invention utilizes a single-frame structured light and three different deep learning networks for decoding, effectively avoiding the superposition of errors in phase extraction and absolute phase range acquisition, and significantly improving the accuracy of three-dimensional reconstruction of the steel pipe surface.

Description

Technical Field

[0001] This invention relates to the field of structured light 3D reconstruction, specifically to a method and application for online measurement and defect detection of the 3D outer contour of a composite stripe projection steel pipe. Background Technology

[0002] Achieving high-precision 3D reconstruction of dynamic scenes has always been a crucial requirement in industrial inspection. Traditional multi-frequency phase-shifting fringe profilometry requires projecting multiple structured light patterns, necessitating that the object be stationary. However, if the object moves during image projection, points with the same pixel coordinates in consecutive frames will correspond to different positions of the object, leading to reconstruction errors. To address dynamic scenes, Fourier transform profilometry is used for single-frame reconstruction, requiring careful design and calibration of the system's pose. Furthermore, reconstructed objects often have complex surfaces. When extracting the wrapper phase using Fourier transform, spectral leakage at frequency 0 introduces errors in the extracted phase, affecting the reconstruction accuracy. Summary of the Invention

[0003] To overcome the shortcomings of the prior art, this invention provides a method and application for online measurement and defect detection of the three-dimensional outer contour of a composite stripe projection steel pipe, aiming to achieve high-precision single-frame structured light three-dimensional reconstruction, thereby enabling accurate detection of three-dimensional defects in the outer contour of the steel pipe.

[0004] The technical solution adopted by this invention to solve the technical problem is as follows:

[0005] The present invention provides a method for online measurement and defect detection of the three-dimensional outer contour of a composite stripe projection steel pipe, characterized by its application in a camera-projector system and the following steps:

[0006] Step 1: In the camera-projector system, calibrate the camera's intrinsic parameters A based on the pinhole camera model. c ;

[0007] Step 2: Calibrate the parameters of the camera-projector system;

[0008] Step 2.1: Based on the image size of the projector in the camera-projector system, generate a sinusoidal fringe image with orthogonal horizontal and vertical axes using equation (1):

[0009]

[0010] Equation (1), where x and y are the pixel coordinates of the sinusoidal fringe image, a(x,y) is the background light intensity at pixel coordinate (x,y), and b(x,y) is the modulation index at pixel coordinate (x,y). W-n (x,y) represents the grayscale value of the horizontal stripe at pixel coordinates (x,y). H-n(x,y) represents the gray value of the vertical stripe at pixel coordinates (x,y); f is the set sinusoidal stripe frequency; W and H are the total number of horizontal and vertical coordinates of the sinusoidal stripe image; and n and N are the phase shift number and phase shift step number, respectively.

[0011] Step 2.2: Project the sinusoidal fringe image onto the calibration plate and acquire the calibration plate image using a camera, thereby calculating the pixel coordinates (x, y, x) of the upper corner point of the calibration plate image using equation (2). c ,y c The pixel coordinates (x) of the projected image corresponding to ) p ,y p ):

[0012]

[0013] In equation (2), Φ v (x c ,y c ) and Φ u (x c ,y c ) represents the pixel coordinates (x, y) of the top corner of the calibration board image. c ,y c The horizontal and vertical sinusoidal phases;

[0014] Step 2.3 According to (x) c ,y c ), (x p ,y p The projector's intrinsic parameters A were calibrated using a pinhole camera model. p And the rotation matrix R and translation vector T between the projector and the camera;

[0015] Step 3: Construction of the steel pipe dataset;

[0016] Step 3.1: Shift the phase value π of the two standard sinusoidal fringe patterns with a frequency period difference of 1 to obtain the corresponding sinusoidal phase patterns. Modulate the orthogonal sinusoidal signals in the two standard sinusoidal fringe patterns and their corresponding sinusoidal phase patterns, and combine them into a single image to obtain a composite fringe image.

[0017] Step 3.2: Use a projector to project the composite stripe image and its standard sinusoidal stripe pattern onto the surface of the steel pipe, respectively, to obtain the composite stripe image and standard stripe image on the surface of the steel pipe.

[0018] The composite stripe image of the steel pipe surface is used as the input data of the dataset, and the wrapping phase value, absolute phase value, and processed stripe order calculated from the standard stripe image of the steel pipe surface are used as the ground values ​​of the dataset, thereby constructing the steel pipe dataset.

[0019] Step 4: Construct and train deep learning networks, including: a wrap phase extraction network, an absolute phase estimation network, and a fringe order unfolding network;

[0020] Both the packaged phase extraction network and the absolute phase estimation network are U-shaped encoder-decoder networks.

[0021] The encoding layer contains c1 downsampling modules, each of which consists of a convolutional layer, a ReLU activation layer, a batch normalization layer, and a residual layer; the kernel size of the convolutional layer in the encoding layer is ks1, and the stride is s1.

[0022] The decoding layer contains c1 upsampling modules, each of which consists of a linear interpolation module, a convolutional layer, a ReLU activation layer, a batch normalization layer, and a residual layer; the convolutional kernel size of the convolutional layer in the decoding layer is ks2, and the stride is s2.

[0023] The striped hierarchical unfolding network is a pyramid transformer structure encoder-decoder network;

[0024] The coding layer contains c2 transformer coding modules. Each transformer coding module consists of a slice layer, a position coding layer, and a transformer layer. The slice size of the slice layer is ks2, and the number of transformer layers is ks3.

[0025] Step 4.1: Input the input data of the dataset into the package phase extraction network and the absolute phase estimation network respectively, and use the package phase value in the ground truth of the dataset as the output of the package phase extraction network and the absolute phase value in the ground truth of the dataset as the output of the absolute phase estimation network to train the package phase extraction network and the absolute phase estimation network to obtain the trained package phase extraction network and the absolute phase estimation network.

[0026] Step 4.2: Using the wrapped phase output of the trained wrapped phase extraction network as the input of the stripe level unfolding network, and using the stripe level in the ground truth of the dataset as the output of the stripe level unfolding network, train the stripe level unfolding network to obtain the trained stripe level unfolding network.

[0027] Step 5: Input the composite stripe image to be predicted into the trained wrap phase extraction network and the trained absolute phase estimation network respectively, and obtain the predicted wrap phase value and the estimated absolute phase range.

[0028] The predicted package phase value is input into the trained stripe level unfolding network to obtain the unfolded stripe level map. Based on the estimated absolute phase range, the unfolded stripe level map is restored to the true absolute phase level and used to unfold the predicted package phase to obtain the true absolute phase.

[0029] Step 6: Based on the true absolute phase and intrinsic parameter A p Internal Reference A c The point cloud coordinates of the object are calculated using the rotation matrix R and the translation vector T, and a cylindrical fitting is performed to obtain the fitting result. The fitting result is compared with a pre-set threshold, and points with large deviations are selected as defect areas in the composite stripe image to be predicted.

[0030] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the online measurement and defect detection method of the three-dimensional outer contour, and the processor is configured to execute the program stored in the memory.

[0031] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the online measurement and defect detection method for three-dimensional outer contour.

[0032] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0033] 1. This invention proposes to use deep learning to analyze single-frame images, which effectively reduces the problem of needing to project multiple structured light patterns in dynamic scenes and can be effectively applied in dynamic scenes.

[0034] 2. This invention designs a composite structured light for the surface of steel pipes, avoiding the problem of poor projection effect of structured light patterns due to the texture of the steel pipe surface itself;

[0035] 3. This invention designs three deep learning networks. The striped-level unfolding network with the encoder-decoder structure of the pyramid transformer can obtain global information, effectively avoiding the superposition of errors in phase extraction and absolute phase range acquisition, and achieving better reconstruction. The striped-level unfolding network is used to process the results of the wrapping phase extraction network, which effectively reduces the two errors of directly extracting the unfolded phase and improving the accuracy of reconstruction. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention;

[0037] Figure 2 Images captured by structured light projection designed for the inventive method onto a steel pipe;

[0038] Figure 3 The image shows the 3D reconstruction effect after neural network processing in the invention method.

[0039] Figure 4 This is a flowchart illustrating the processing of the wrapper phase extraction network and the absolute phase estimation network in the invention method;

[0040] Figure 5 This is a flowchart illustrating the processing of the striped hierarchical unfolded network in the invention method. Detailed Implementation

[0041] In this embodiment, a method for online measurement and defect detection of the three-dimensional outer contour of a composite striped projection steel pipe is applied to a camera-projector system. The camera acquires coded structured light projected onto the surface of the steel pipe by the projector. A deep learning network extracts the structured light code modulated by the steel pipe surface, thereby obtaining the depth information of the steel pipe surface. Based on the system calibration results, the depth is reconstructed into a point cloud to obtain the morphological information of the steel pipe surface. This information is then fitted and compared with a standard model to identify defect areas. Specifically, for example... Figure 1 As shown, the method is performed according to the following steps:

[0042] Step 1: In the camera-projector system, calibrate the camera's intrinsic parameters A using the Zhang Zhengyou calibration method based on the pinhole camera model. c ;

[0043] Step 2: Calibrate the parameters of the camera-projector system;

[0044] Step 2.1: Based on the image size of the projector in the camera-projector system, generate a sinusoidal fringe image with orthogonal horizontal and vertical axes using equation (1):

[0045]

[0046] Equation (1), where x and y are the pixel coordinates of the sinusoidal fringe image, a(x,y) is the background light intensity at pixel coordinate (x,y), and b(x,y) is the modulation index at pixel coordinate (x,y). W-n (x,y) represents the grayscale value of the horizontal stripe at pixel coordinates (x,y). H-n (x,y) represents the gray value of the vertical stripe at pixel coordinates (x,y); f is the set sinusoidal stripe frequency; W and H are the total number of horizontal and vertical coordinates of the sinusoidal stripe image; and n and N are the phase shift number and phase shift step number, respectively.

[0047] Step 2.2: Project a sinusoidal fringe image onto the calibration plate and use a camera to capture the image of the calibration plate. Treat the projector as a virtual camera and calculate the pixel coordinates (x, y, x) of the upper corner of the calibration plate image according to equation (2). c ,yc The pixel coordinates (x) on the corresponding projected image p ,y p ):

[0048]

[0049] In equation (2), Φ v (x c ,y c ) and Φ u (x c ,y c ) represents the pixel coordinates (x, y) of the top corner of the calibration board image. c ,y c The horizontal and vertical sinusoidal phases;

[0050] Step 2.3 According to (x) c ,y c ), (x p ,y p The projector's intrinsic parameters A were calibrated using a pinhole camera model. p The pose relationship between the projector and the calibration board is determined, and then the positional relationship R and T between the projector and the camera is obtained based on the poses of the camera and the calibration board and the poses of the projector and the calibration board.

[0051] Step 3: Construction of the steel pipe dataset;

[0052] Step 3.1: Shift the phase value by π between two standard sinusoidal fringe patterns with a frequency period difference of 1 to obtain the corresponding sinusoidal phase patterns. This modulates the orthogonal sinusoidal signals in the two standard sinusoidal fringe patterns and their corresponding sinusoidal phase patterns, and combines them into a single image to obtain a composite fringe image. The two different frequency periods differ by 1, which satisfies the condition for phase unwrapping. The two signals with a phase value difference of π effectively reduce spectral aliasing and improve the accuracy of wrapped phase extraction. In this example, vertical stripes with frequencies of 31 and 32 are used. Through equation (3), horizontal sinusoidal signals with carrier frequencies of 30, 60, 90, and 120 are used to composite the encoded signals into a single image.

[0053]

[0054] In equation (3), I p For the final composite stripe image, (x p ,y p ) represents the image coordinates, I b For the coded stripes, I cb The carrier stripes represent the total number of encoded images, and m is the encoding sequence number.

[0055] Step 3.2: Use a projector to project the composite fringe image and its standard sinusoidal fringe pattern onto the surface of the steel pipe. The standard sinusoidal fringe pattern is projected using a 20-step phase-shifting method to obtain the composite fringe image and the standard fringe image on the surface of the steel pipe, for a total of 41 images.

[0056] The composite stripe image of the steel pipe surface is used as the input data of the dataset, and the wrapping phase value, absolute phase value, and processed stripe order calculated from the standard stripe image of the steel pipe surface are used as the ground values ​​of the dataset, thereby constructing the steel pipe dataset.

[0057] Step 4: Construct and train deep learning networks, including: a wrap phase extraction network, an absolute phase estimation network, and a fringe order unfolding network;

[0058] Both the wrapper phase extraction network and the absolute phase estimation network are U-shaped encoder-decoder networks.

[0059] The encoding layer contains c1 downsampling modules, each of which consists of a convolutional layer, a ReLU activation layer, a batch normalization layer, and a residual layer; the kernel size of the convolutional layer in the encoding layer is ks1, and the stride is s1.

[0060] The decoding layer contains c1 upsampling modules, each of which consists of a linear interpolation module, a convolutional layer, a ReLU activation layer, a batch normalization layer, and a residual layer; the convolutional kernel size of the convolutional layer in the decoding layer is ks2, and the stride is s2.

[0061] Convolution can better focus on local information in an image and extract the necessary phase information from the local area; the U-shaped structure can effectively integrate shallow and deep features, and can better perceive shallow texture information such as edges, making training more stable.

[0062] The striped hierarchical unfolded network is a pyramid transformer structure encoder-decoder network;

[0063] The encoding layer contains c2 transformer encoding modules. Each transformer encoding module consists of a slice layer, a position encoding layer, and a transformer layer. The slice size of the slice layer is ks2, and the number of transformer layers is ks3. If the results of the absolute phase estimation network are directly used to guide the wrapping phase extraction network's wrapping phase unfolding, there will be errors in some parts of the phase estimation network's results due to the surface texture of the steel pipe. To avoid the superposition of errors between the absolute phase estimation network and the wrapping phase extraction network, a stripe level unfolding network is used to continuously unfold the prediction results of the wrapping phase extraction network. The encoding layer of the stripe level unfolding network consists of pyramid-shaped transformers. Transformers have strong global modeling capabilities. Using the Transformer structure can help to better focus on the wrapping phase on the entire image and better predict the level of the current pixel. Using convolutional layers as decoding modules helps to better predict the level relationship between adjacent pixels, and finally obtain the level information of the stripes.

[0064] Step 4.1: Input the input data of the dataset into the wrap phase extraction network and the absolute phase estimation network respectively, and use the wrap phase value in the ground truth of the dataset as the output of the wrap phase extraction network and the absolute phase value in the ground truth of the dataset as the output of the absolute phase estimation network to train the wrap phase extraction network and the absolute phase estimation network to obtain the trained wrap phase extraction network and the absolute phase estimation network.

[0065] Step 4.2: Use the wrapped phase output of the trained wrapped phase extraction network as the input of the stripe level unfolding network, and use the stripe level in the ground truth of the dataset as the output of the stripe level unfolding network to train the stripe level unfolding network and obtain the trained stripe level unfolding network.

[0066] Step 5: Image of the composite stripes to be predicted as follows Figure 2 The values ​​are input into the trained package phase extraction network and the trained absolute phase estimation network, respectively, to obtain the predicted package phase value and the estimated absolute phase range, as shown below. Figure 4 As shown; the predicted package phase value is input into the trained stripe level unfolding network, and the unfolded stripe level map is obtained, as shown. Figure 5 As shown, there is a series difference between the stripe series level map and the true phase level. The series difference is obtained by taking the mode of the difference between the stripe series level map and the absolute phase range. The package phase predicted by the package phase extraction network is expanded based on the series difference and the stripe series level map.

[0067] Step 6: Based on the true absolute phase and intrinsic parameter A p Internal Reference Ac The rotation matrix R and translation vector T are used to calculate the point cloud coordinates of an object, as shown below. Figure 3 Cylindrical fitting is then performed to obtain the fitting results. These results are compared with a pre-set threshold, and points with larger deviations are selected as defect areas in the composite stripe image to be predicted.

Claims

1. A method for online measurement and defect detection of the three-dimensional outer contour of a composite stripe projection steel pipe. Its characteristic is that it is applied to a camera-projector system and is carried out in the following steps: Step 1: In the camera-projector system, calibrate the camera's intrinsic parameters based on the pinhole camera model. ; Step 2: Calibrate the parameters of the camera-projector system; Step 2.1: Based on the image size of the projector in the camera-projector system, generate a sinusoidal fringe image with orthogonal horizontal and vertical axes using equation (1): (1) Equation (1), where x and y are the pixel coordinates of the sinusoidal fringe image, respectively. pixel coordinates Background light intensity at that location pixel coordinates The adjustment system of the department, pixel coordinates The grayscale value of the horizontal stripes. pixel coordinates The grayscale value of the vertical stripes; The frequency of the sinusoidal fringe is set, W and H are the total number of horizontal and vertical coordinates of the sinusoidal fringe image, and n and N are the phase shift number and phase shift step number, respectively. Step 2.2: Project the sinusoidal fringe image onto the calibration plate and acquire the calibration plate image using a camera, thereby calculating the pixel coordinates of the upper corner points of the calibration plate image using equation (2). The pixel coordinates of the corresponding projected image : (2) In equation (2), and The pixel coordinates of the top corner of the calibration board image The horizontal and vertical sinusoidal phases; Step 2.3 According to , The projector's intrinsic parameters were calibrated using a pinhole camera model. and the rotation matrix between the projector and the camera Translation vector ; Step 3: Construction of the steel pipe dataset; Step 3.1: Shift the phase value of the two standard sinusoidal fringe patterns whose frequency periods differ by 1. The corresponding sinusoidal phase diagram is then obtained, and the orthogonal sinusoidal signals in the two standard sinusoidal fringe diagrams and their corresponding sinusoidal phase diagrams are modulated and combined into a single image to obtain a composite fringe image. Step 3.2: Use a projector to project the composite stripe image and its standard sinusoidal stripe pattern onto the surface of the steel pipe, respectively, to obtain the composite stripe image and standard stripe image on the surface of the steel pipe. The composite stripe image of the steel pipe surface is used as the input data of the dataset, and the wrapping phase value, absolute phase value, and processed stripe order calculated from the standard stripe image of the steel pipe surface are used as the ground values ​​of the dataset, thereby constructing the steel pipe dataset. Step 4: Construct and train deep learning networks, including: a wrap phase extraction network, an absolute phase estimation network, and a fringe order unfolding network; Both the packaged phase extraction network and the absolute phase estimation network are U-shaped encoder-decoder networks. The coding layer contains There are three downsampling modules, each consisting of a convolutional layer, a ReLU activation layer, a batch normalization layer, and a residual layer; the kernel size in the encoding layer is [size missing]. Step size is ; The decoding layer contains Each upsampling module consists of a linear interpolation module, a convolutional layer, a ReLU activation layer, a batch normalization layer, and a residual layer; the kernel size of the convolutional layer in the decoding layer is [missing information]. Step size is ; The striped hierarchical unfolding network is a pyramid transformer structure encoder-decoder network; The coding layer contains Each transformer encoding module consists of a slice layer, a positional encoding layer, and a transformer layer, with the slice size of the slice layer being [missing information]. The number of transformer layers is ; Step 4.1: Input the input data of the dataset into the package phase extraction network and the absolute phase estimation network respectively, and use the package phase value in the ground truth of the dataset as the output of the package phase extraction network and the absolute phase value in the ground truth of the dataset as the output of the absolute phase estimation network to train the package phase extraction network and the absolute phase estimation network to obtain the trained package phase extraction network and the absolute phase estimation network. Step 4.2: Using the wrapped phase output of the trained wrapped phase extraction network as the input of the stripe level unfolding network, and using the stripe level in the ground truth of the dataset as the output of the stripe level unfolding network, train the stripe level unfolding network to obtain the trained stripe level unfolding network. Step 5: Input the composite stripe image to be predicted into the trained wrap phase extraction network and the trained absolute phase estimation network respectively, and obtain the predicted wrap phase value and the estimated absolute phase range. The predicted package phase value is input into the trained stripe level unfolding network to obtain the unfolded stripe level map. Based on the estimated absolute phase range, the unfolded stripe level map is restored to the true absolute phase level and used to unfold the predicted package phase to obtain the true absolute phase. Step 6: Based on the true absolute phase and internal parameters Internal Reference Rotation matrix Translation vector The point cloud coordinates of the object are calculated and cylindrical fitting is performed to obtain the fitting result. The fitting result is compared with a pre-set threshold, and points with large deviations are selected as defect areas in the composite stripe image to be predicted.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports a processor in executing the method of claim 1, the processor being configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the method of claim 1.

Images