Line structured light vision system calibration method based on mixed projection

Abstract

The line structured light vision system calibration method based on mixed projection specifically comprises the following steps: S1, generating a phase displacement grating pattern and a fixed spot pattern, projecting a mixed pattern of the phase displacement grating pattern and the fixed spot pattern onto a flat plate, and capturing a projection image by a camera; s2, acquiring pixels of a projection image captured by the camera, calculating an absolute phase, establishing a corresponding relation between the camera and the projector, and regarding the projector and the camera as a virtual camera; s3, projecting a plurality of checkerboard patterns in different directions on the flat plate, and capturing projection images of the patterns one by one by a camera; mapping a projection image captured by the camera into a coordinate system of the projector according to the corresponding relation between the camera and the projector established in the step S2, and converting the projection image into a virtual image of the projector; s4, inputting the virtual image into a calibration tool box of matlab, calculating internal parameters and external parameters of the virtual camera, and completing calibration; the operation process is simple and the calibration precision is high.

Description

Technical Field

[0001] The present invention relates to the technical field of line structured light vision system calibration, and in particular to a line structured light vision system calibration method based on hybrid projection. Background Art

[0002] With the rapid development of computer technology, computer vision theory has begun to be applied to the field of precision testing, thus forming an emerging technology - visual inspection technology. Among them, the line structured light vision system has the advantages of non-contact, fast speed and high precision, and is widely used in welding guidance, industrial measurement and quality inspection and other fields. The line structured light vision system usually consists of two parts: a projector and a camera, which must be calibrated before use; the purpose of calibration is to project a specific light signal onto the surface of the object, which is collected by the camera, and the position and depth of the object are calculated according to the changes in the light signal caused by the object, thereby restoring the entire three-dimensional space.

[0003] When calibrating the line structured light vision system, a projector needs to project a stable image (such as stripe patterns and spot patterns) and project this image onto a calibration board (generally a chessboard calibration board). The traditional method generally uses time multiplexing technology or spatial neighborhood technology to generate a specific projection pattern. Among them, time multiplexing technology is to continuously project a series of patterns within a certain period of time. Representative patterns include binary coding patterns, n-array coding patterns, grayscale coding patterns, and phase shift patterns; the advantage of the coding method used for these patterns is that the pixels of each camera are different and independent, so they can be easily and accurately decoded to achieve high-precision results; however, its encoding process is time-consuming, making it difficult to apply time multiplexing technology to scenes that require high-speed or real-time measurement. Spatial neighborhood technology only projects a well-designed pattern, such as color coding and symbol coding patterns, which can be used in dynamic scenes, but its resolution and accuracy are limited.

[0004] In addition, the traditional calibration method requires the projector and camera to be calibrated separately. For example, Chinese patent CN202310048348.7 discloses a method for calibrating a line structured light sensor. First, the camera is calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficient, and then the image of the calibration plate under the external light source at the same position and the image of the calibration plate projected with laser stripes are collected for distortion correction; then the plane coordinates of the feature points in the local coordinate system are solved using the homography matrix to obtain the three-dimensional coordinates of the feature points in the world coordinate system, and finally the principal component analysis method is used to reproject the three-dimensional coordinates of the feature points in the world coordinate system into plane coordinates, and the homography matrix from the light plane to the image plane is calculated to complete the light plane calibration. The operation process of this calibration method is relatively complicated. Summary of the invention

[0005] The present application provides a calibration method for a line structured light vision system based on hybrid projection that is flexible in calculation and high in accuracy, and is used to overcome the above-mentioned problems existing in the prior art. The calibration method for a line structured light vision system based on hybrid projection of the present application combines time multiplexing technology and spatial neighborhood technology to generate a specific hybrid pattern, and at the same time utilizes the correspondence between the projector and the camera, regards the projector and the camera as a virtual camera, and uses the matlab calibration toolbox to directly calibrate the virtual camera, without the need to calibrate the projector and the camera separately, which simplifies the operation process, is simple in calculation, flexible in calibration, and has high calibration accuracy.

[0006] The technical solution of this application is:

[0007] The calibration method of the line structured light vision system based on hybrid projection specifically includes the following steps:

[0008] S1, generating a phase-shifted grating pattern and a fixed spot pattern, and projecting a mixed pattern of the phase-shifted grating pattern and the fixed spot pattern onto a flat plate, and then capturing the projected image by a camera;

[0009] S2, collects the pixels of the projected image captured by the camera, calculates the absolute phase, establishes the corresponding relationship between the camera and the projector, and regards the projector and the camera as a virtual camera;

[0010] S3, projecting multiple checkerboard patterns in different directions on the flat plate, and capturing the projection images of these patterns one by one by the camera; and then mapping the projection images captured by the camera into the coordinate system of the projector according to the correspondence between the camera and the projector established in step S2, and converting them into virtual images of the projector;

[0011] S4, input the virtual image into the calibration toolbox of MATLAB, calculate the internal and external parameters of the virtual camera, and complete the calibration.

[0012] Compared with the prior art, the calibration method of the line structured light vision system based on hybrid projection of the present application combines time multiplexing technology and spatial neighborhood technology to generate a specific hybrid pattern, and at the same time, utilizes the correspondence between the projector and the camera, regards the projector and the camera as a virtual camera, and directly calibrates the virtual camera using the MATLAB calibration toolbox, and no longer needs to calibrate the projector and the camera separately, which simplifies the operation process, is simple to calculate, and is flexible in calibration, and has high calibration accuracy. Specifically, first, a hybrid pattern is generated and projected onto a flat plate, and the projected image is captured by the camera, and then the absolute phase is calculated based on the pixels of the projected image to establish the correspondence between the camera and the projector; then, multiple checkerboard patterns are projected in different directions, and the captured checkerboard projection image is converted into a virtual image of the projector by utilizing the correspondence between the camera and the projector, and finally, the internal and external parameters of the virtual camera are calculated using the MATLAB calibration toolbox to complete the calibration.

[0013] As an optimization, in the aforementioned calibration method of the line structured light vision system based on hybrid projection, in the step S1, a three-step phase shift method is used to generate a phase-shifted grating pattern, a pseudo-random number generator is used to generate a fixed spot pattern, and the phase-shifted grating pattern and the fixed spot pattern are overlapped to obtain a mixed pattern. The three-step phase shift method obtains phase information by decoding by acquiring a three-step phase-shifted picture. This method can improve the stability and reliability of the image while ensuring accuracy; thereby, the generated phase-shifted grating pattern has high precision and high stability. The pseudo-random number generator has the advantages of strong controllability, high repeatability and high speed when generating images; the use of a pseudo-random number generator to generate a fixed spot pattern is simple to operate and easy to implement. As a result, the generation of the mixed pattern in the present application is relatively easy and has high stability.

[0014] Furthermore, the generation formula of the phase-shifted grating pattern is as follows: Where A is the amplitude, C is a constant scalar, f is the frequency of the fringe pattern, 2πi / 3 are the 3 different phase offsets, and (x,y) are the pixels captured by the camera.

[0015] Furthermore, the spots in the fixed spot pattern are a zero matrix; the size of the spots is determined by the resolution of the projector and the camera and the speckle density, and the spot density can be selected according to the specific application.

[0016] As an optimization, in the aforementioned hybrid projection-based line structured light vision system calibration method, in step S2, a virtual image of the projector is modeled according to the pixels of the projection image captured by the camera: Among them, I A (u,v)=α(u,v)A,I C(u,v)=α(u,v)[C+β(u,v)]; (u,v) is the pixel of the virtual image, α(u,v) represents the reflectivity of each pixel, and β(u,v) represents the ambient illumination function of each pixel; the phase relationship between the camera and the projector is:

[0017] Furthermore, the normalized cross-correlation algorithm is used to calculate the similarity of each region of the projected image and the virtual image: Among them, the target block m (virtual image) and the reference block n (projected image) are two image blocks of the same size. and is the average intensity of the two image blocks. The normalized cross-correlation algorithm is used to calculate the similarity of each area of ​​the image, which can be used to match the feature points in the virtual image and the projected image captured by the camera, and these feature points can be identified and located by calculating the correlation between them; thus, the impact of image deformation and noise problems in the line structured light vision system can be reduced.

[0018] As an optimization, in the aforementioned hybrid projection-based line structured light vision system calibration method, in step S4, the internal and external parameters of the virtual camera are calculated by minimizing the reprojection error. Minimizing the reprojection error can gradually adjust the parameters of the virtual camera in an iterative manner to achieve the best matching result, thereby improving the accuracy and efficiency of the calibration; at the same time, by continuously optimizing the external parameters of the virtual camera, the stability of the entire system can be enhanced, so that good performance can be maintained even in complex environments.

[0019] Furthermore, the internal parameters are the focal length and image parameters of the virtual camera, and the external parameters are the rotation matrix and translation vector of the virtual camera.

[0020] As an optimization, in the aforementioned hybrid projection-based line structured light vision system calibration method, in step S3, 5-10 checkerboard patterns in different directions are projected on the flat plate. When there are too few projected checkerboard patterns, there may be a certain degree of randomness, which in turn affects the calibration result; and when there are too many projected checkerboard patterns, it will lead to excessive calculations, affecting the calculation efficiency; the inventors have shown through experiments that it is more appropriate to select 5-10 patterns.

[0021] As an optimization, in the aforementioned hybrid projection-based line structured light vision system calibration method, the flat board in steps S1 and S3 is a flat white board, thereby preventing the hybrid pattern or checkerboard pattern from being deformed when projected onto the flat board, thereby affecting the subsequent calculation results, and ensuring the accuracy of the calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1It is a flow chart of the calibration method of the line structured light vision system based on hybrid projection of the present application;

[0023] Figure 2 is a phase-shifted grating pattern generated by a three-step phase shift method in an embodiment of the present application;

[0024] Figure 3 is a fixed speckle pattern generated by a pseudo-random number generator in an embodiment of the present application;

[0025] Figure 4 Yes Figure 2 The phase-shifted grating pattern and Figure 3 The mixed pattern is obtained by overlapping the fixed spot patterns in . DETAILED DESCRIPTION

[0026] The present application is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to be limiting of the present application; all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. In the following embodiments, the contents not described in detail are all common technical knowledge in the field.

[0027] Line structured light vision systems are generally divided into two parts: a projector and a camera. When calibrating a line structured light vision system, the projector needs to project a stable image (such as a stripe pattern and a spot pattern) and project this image onto a calibration board (generally a chessboard calibration board). Traditional methods generally use time multiplexing technology or spatial neighborhood technology to generate a specific projection pattern, and require the projector and camera to be calibrated separately.

[0028] In view of this, the present application provides a calibration method for a line structured light vision system based on hybrid projection, which combines time multiplexing technology and spatial neighborhood technology to generate a specific hybrid pattern. At the same time, the correspondence between the projector and the camera is utilized, the projector and the camera are regarded as a virtual camera, and the MATLAB calibration toolbox is used to directly calibrate the virtual camera. There is no need to calibrate the projector and camera separately, which simplifies the operation process, makes the calculation simple, the calibration flexible, and the calibration accuracy high.

[0029] Example:

[0030] See also Figure 1 The line structured light vision system calibration method based on hybrid projection in this embodiment specifically includes the following steps.

[0031] S1, generating a phase-shifted grating pattern and a fixed spot pattern, and projecting a mixed pattern of the phase-shifted grating pattern and the fixed spot pattern onto a flat panel, and then capturing the projected image by a camera.

[0032] Specifically, S11: Generate a phase-shifted grating pattern using a three-step phase shift method (see Figure 2 ), the generating formula is as follows: Where A is the amplitude, C is a constant scalar, f is the frequency of the fringe pattern, 2πi / 3 are three different phase offsets, and (x, y) are the pixels captured by the camera; S12: Generate a fixed speckle pattern using a pseudo-random number generator, where the spots in the fixed speckle pattern are an M×M zero matrix (see Figure 3 , the spots are the white dots in the figure); the size of the spots is determined by the resolution of the projector and the camera and the speckle density, and the density of the spots is selected according to the specific application; S13: overlapping the phase-shifted grating pattern and the fixed spot pattern to obtain a mixed pattern (see Figure 4 ).

[0033] The implementation of the three-step phase shift method includes two main steps: making a phase-shifted coded image and decoding the acquired phase-shifted image. It can effectively extract the phase information of the surface of the object, and the solved value has a good effect on anti-noise; and this method can improve the stability and reliability of the image while ensuring accuracy; thus, the generated phase-shifted grating pattern has high accuracy and high stability. The pseudo-random number generator has the advantages of strong controllability, high repeatability and high speed when generating images; the pseudo-random number generator is used to generate a fixed spot pattern, which is simple to operate and easy to implement. As a result, the generation of the mixed pattern in this application is relatively easy and has high stability.

[0034] S2, collects the pixels of the projected image captured by the camera, calculates the absolute phase, establishes the correspondence between the camera and the projector, and regards the projector and the camera as a virtual camera.

[0035] Specifically, based on the pixels of the projected image captured by the camera, a virtual image of the projector is modeled: Among them, I A (u,v)=α(u,v)A,I C (u,v)=α(u,v)[C+β(u,v)]; (u,v) is the pixel of the virtual image, α(u,v) represents the reflectivity of each pixel, and β(u,v) represents the ambient illumination function of each pixel; at the same time, the normalized cross-correlation algorithm is used to calculate the similarity of each area of ​​the projected image and the virtual image: Among them, the target block m (virtual image) and the reference block n (projected image) are two image blocks of the same size. and is the average intensity of the two image blocks; according to the above two formulas, the phase relationship between the camera and the projector can be obtained as:

[0036] In this application, the normalized cross correlation (NCC) algorithm is used to calculate the similarity of each area of ​​the image, which can be used to match the feature points in the virtual image and the projected image captured by the camera, and these feature points are identified and located by calculating the correlation between them; thereby reducing the impact of image deformation and noise problems in the line structured light vision system.

[0037] S3, projecting 8 checkerboard patterns in different directions onto the flat panel, and capturing the projection images of these patterns one by one by the camera; and then mapping the projection images captured by the camera into the coordinate system of the projector according to the correspondence between the camera and the projector established in step S2, and converting them into virtual images of the projector.

[0038] S4, input the virtual image into the calibration toolbox of MATLAB, calculate the internal and external parameters of the virtual camera by minimizing the reprojection error, and complete the calibration; the internal parameters are the focal length and image parameters of the virtual camera, and the external parameters are the rotation matrix and translation vector of the virtual camera. Minimizing the reprojection error can gradually adjust the parameters of the virtual camera in an iterative manner to achieve the best matching result, thereby improving the accuracy and efficiency of the calibration; at the same time, by continuously optimizing the external parameters of the virtual camera, the stability of the entire system can be enhanced, so that good performance can be maintained even in complex environments.

[0039] In this embodiment, the flat board in steps S1 and S3 is a flat white board, thereby preventing the mixed pattern or the checkerboard pattern from being deformed when projected onto the flat board, thereby affecting subsequent calculation results, and ensuring calibration accuracy.

[0040] The above general description of the invention involved in this application and the description of its specific implementation methods should not be understood as limiting the technical solutions of the invention. Based on the disclosure of this application, those skilled in the art can add, reduce or combine the disclosed technical features in the above general description and / or specific implementation methods (including examples) without violating the constituent elements of the invention involved, to form other technical solutions within the scope of protection of this application.

Claims

1. A line structured light vision system calibration method based on hybrid projection, characterized in that: The specific steps include: S1, generating a phase-shifted grating pattern and a fixed spot pattern, and projecting a mixed pattern of the phase-shifted grating pattern and the fixed spot pattern onto a flat plate, and then capturing the projected image by a camera; S2, collects the pixels of the projected image captured by the camera, calculates the absolute phase, establishes the corresponding relationship between the camera and the projector, and regards the projector and the camera as a virtual camera; S3, projecting multiple checkerboard patterns in different directions onto the flat panel, and using the camera to capture the projected images of these patterns one by one; Then, according to the correspondence between the camera and the projector established in step S2, the projection image captured by the camera is mapped into the coordinate system of the projector and converted into a virtual image of the projector; S4, input the virtual image into the calibration toolbox of MATLAB, calculate the internal and external parameters of the virtual camera, and complete the calibration.

2. The method for calibrating a line structured light vision system based on hybrid projection according to claim 1, characterized in that: In the step S1, a three-step phase shift method is used to generate a phase-shifted grating pattern, and a pseudo-random number generator is used to generate a fixed spot pattern; The phase-shifted grating pattern and the fixed spot pattern are then overlapped to obtain a mixed pattern.

3. The method for calibrating a line structured light vision system based on hybrid projection according to claim 2, characterized in that: The generation formula of the phase shift grating pattern is as follows: ;in, is the amplitude, is a constant scalar, is the frequency of the fringe pattern, There are three different phase shifts. are the pixels captured by the camera.

4. The method for calibrating a line structured light vision system based on hybrid projection according to claim 2, characterized in that: The spots in the fixed spot pattern are a matrix of zeros.

5. The method for calibrating a line structured light vision system based on hybrid projection according to claim 3, characterized in that: In step S2, a virtual image of the projector is modeled according to the pixels of the projection image captured by the camera: ； in, , ; is the pixel of the virtual image, Represents the reflectivity of each pixel. Represents the ambient lighting function of each pixel; the phase relationship between the camera and the projector is obtained as: 。 6. The method for calibrating a line structured light vision system based on hybrid projection according to claim 5, characterized in that: The normalized cross-correlation algorithm is used to calculate the similarity of each region between the projected image and the virtual image: ； The target block and reference blocks are two image blocks of the same size, and is the average intensity of the two image patches.

7. The method for calibrating a line structured light vision system based on hybrid projection according to claim 1, characterized in that: In step S4, the internal and external parameters of the virtual camera are calculated by minimizing the reprojection error.

8. The method for calibrating a line structured light vision system based on hybrid projection according to claim 7, characterized in that: The internal parameters are the focal length and image parameters of the virtual camera, and the external parameters are the rotation matrix and translation vector of the virtual camera.

9. The method for calibrating a line structured light vision system based on hybrid projection according to claim 1, characterized in that: In step S3, 5 to 10 checkerboard patterns in different directions are projected onto the flat panel.

10. The method for calibrating a line structured light vision system based on hybrid projection according to claim 1, characterized in that: The flat board in steps S1 and S3 is a flat white board.

Images