Imaging method based on line shift Gray code

Abstract

The invention provides an imaging method based on a line shift Gray code. The imaging method comprises the following steps: constructing a structured light three-dimensional imaging system, wherein the structured light three-dimensional imaging system comprises a camera, a projector and a computer; projecting a line shift image and a Gray code image to a target object according to the structured light three-dimensional imaging system and collecting the line shift image and the Gray code image; performing feature point decoding on the target object according to the collected line shift image and the Gray code image to obtain decoding information of the target object; manufacturing a calibration plate according to the measurement view of the structured light three-dimensional imaging system, and obtaining the internal and external parameters of the camera and the internal and external parameters of the projector according to the calibration plate; and according to the decoding information of the target object, the internal and external parameters of the camera and the internal and external parameters of the projector, reconstructing a depth map and a three-dimensional point cloud of the target object based on a triangulation principle. According to the method, the characteristics of line shifting and Gray codes are combined, the sampling point density can be improved while the decoding accuracy is not changed, the better anti-interference capability is achieved, and the high measurement precision is achieved for the object surface with the large reflectivity change range and even the stainless steel surface.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and structured light three-dimensional imaging, and more specifically, relates to an imaging method based on line-shifted Gray code. Background Art

[0002] In recent years, with the acceleration of the computing and processing speed of computers, key breakthroughs and progress have been made in the research of optoelectronic components, digital imaging devices, lasers, and other light sources, enabling the gradual integration of optical three-dimensional reconstruction technology into our production and life. Compared with two-dimensional images, three-dimensional reconstruction can obtain the depth information of the object to be measured, and has high application value in many fields such as machine vision, industrial inspection, reverse engineering, medical diagnosis, biometric identification, digitalization of cultural relics, and film and television special effects. Among them, structured light technology is an important research field of three-dimensional reconstruction and plays an important role in many intelligent devices and practical applications. The structured light method has the advantages of low cost, high resolution, and fast speed, and is the most practical optical non-contact three-dimensional measurement technology. Currently, the existing structured light three-dimensional imaging methods mainly use the combination of Gray code and phase shift. This method is easily affected by the discontinuity of the object shape change, the non-sinusoidality of the acquired fringe image, and image noise, and has poor anti-interference ability. Summary of the Invention

[0003] In view of this, the present invention proposes an imaging method based on line-shifted Gray code, and the imaging method includes the following steps:

[0004] Step S100, constructing a structured light three-dimensional imaging system, where the structured light three-dimensional imaging system includes a camera, a projector, and a computer;

[0005] Step S200, projecting and collecting line-shifted images and Gray code images to a target object according to the structured light three-dimensional imaging system;

[0006] Step S300, decoding the feature points of the target object according to the collected line-shifted images and Gray code images to obtain the decoded information of the target object;

[0007] Step S400, making a calibration board according to the measurement field of view size of the structured light three-dimensional imaging system, and obtaining the internal and external parameters of the camera and the internal and external parameters of the projector according to the calibration board;

[0008] Step S500, reconstructing the depth map and three-dimensional point cloud of the target object according to the decoded information of the target object, the internal and external parameters of the camera, and the internal and external parameters of the projector, and based on the principle of triangulation.

[0009] Optionally, in step S200, according to the resolution of the projector, a corresponding number of horizontal Gray codes and vertical Gray codes are projected so that each pixel of the projection image can be uniquely encoded with a Gray code value; the line shift pattern performs pixel translation according to the number of periods of the Gray code.

[0010] Optionally, in step S300, using the image collected by the camera, the Gray code is unfolded by level, the line shift pattern extracts the center points of the line shift stripes according to the Steger algorithm, and the pixel points on the image are unfolded to obtain the code value corresponding to each pixel.

[0011] Optionally, in step S400, a triangulation relationship is established between the camera and the projector to fabricate a calibration board for the structured light three-dimensional imaging system, the internal and external parameters of the camera are calibrated according to the Zhang Zhengyou calibration method, and the internal and external parameters of the projector are deduced according to the encoded phase information of the feature points of the calibration board.

[0012] The beneficial effects of the present invention are as follows:

[0013] The imaging method based on line shift Gray code of the present invention first constructs a structured light three-dimensional imaging system, and the structured light three-dimensional imaging system includes a camera, a projector and a computer; secondly, projects and collects line shift images and Gray code images to a target object according to the structured light three-dimensional imaging system; thirdly, decodes the feature points of the target object according to the collected line shift images and Gray code images to obtain the decoded information of the target object; fourthly, fabricates a calibration board according to the measurement field of view size of the structured light three-dimensional imaging system, and obtains the internal and external parameters of the camera and the internal and external parameters of the projector according to the calibration board; finally, based on the decoded information of the target object, the internal and external parameters of the camera and the internal and external parameters of the projector, and based on the principle of triangulation, reconstructs the depth map and three-dimensional point cloud of the target object.

[0014] The imaging method based on line shift Gray code of the present invention utilizes the characteristics of line shift combined with Gray code, can improve the sampling point density without changing the decoding accuracy, solves well the problem that the three-dimensional shape of highly reflective objects cannot be reconstructed in the prior art, has better anti-interference ability, and has high measurement accuracy for the surface of objects with a large range of reflectivity changes, even for the stainless steel surface.

[0015] Other features and advantages of the present invention will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention can be better understood by referring to the following description made in conjunction with the drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar components.

[0017] Figure 1 shows an implementation flowchart of an imaging method based on line-shifted Gray code according to an embodiment of the present invention;

[0018] Figure 2 shows a schematic structural diagram of a structured light three-dimensional imaging system according to an embodiment of the present invention, where 1 is a camera, 2 is a projector, 3 is a computer, and 4 is an object;

[0019] Figure 3 shows a schematic diagram of a Gray code pattern according to an embodiment of the present invention;

[0020] Figure 4 shows a schematic diagram of a line-shifted pattern according to an embodiment of the present invention. Detailed implementation manners

[0021] In order to enable those skilled in the art to more fully understand the technical solution of the present invention, the exemplary embodiments of the present invention will be described more comprehensively and in detail below with reference to the accompanying drawings. Obviously, one or more of the embodiments of the present invention described below are merely one or more of the specific ways to implement the technical solution of the present invention, and are not exhaustive. It should be understood that other ways belonging to a general inventive concept can be used to implement the technical solution of the present invention, and should not be limited by the exemplary embodiments described. Based on one or more embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0022] Embodiment: Figure 1 shows an implementation flowchart of an imaging method based on line-shifted Gray code according to an embodiment of the present invention, Figure 2 shows a schematic structural diagram of a structured light three-dimensional imaging system according to an embodiment of the present invention. Referring to Figure 1 and Figure 2 , the imaging method based on line-shifted Gray code according to an embodiment of the present invention includes:

[0023] Step S100, constructing a structured light three-dimensional imaging system, which includes a camera, a projector, and a computer;

[0024] Step S200, projecting and collecting line-shifted images and Gray code images to the object according to the structured light three-dimensional imaging system;

[0025] Step S300, decoding the feature points of the object according to the collected line-shifted images and Gray code images to obtain the decoded information of the object;

[0026] Step S400: Make a calibration board according to the measurement field of view size of the structured light three-dimensional imaging system, and obtain the internal and external parameters of the camera and the internal and external parameters of the projector based on the calibration board;

[0027] Step S500: Based on the decoded information of the target object, the internal and external parameters of the camera, and the internal and external parameters of the projector, and based on the principle of triangulation, reconstruct the depth map and three-dimensional point cloud of the target object.

[0028] Specifically, in the embodiment of the present invention, a system is built in which a computer controls a projector to project multiple encoded patterns onto a target object, and at the same time, a camera captures images and performs decoding operations. As Figure 2 shown, the computer generates encoded images, and sequentially projects the images onto the object surface by the projector in order. The striped pattern is deformed due to the change in the height of the object surface. Each modulated striped image is recorded by the camera, and the computer is used for data processing. Finally, the height information of the object to be measured in the three-dimensional world is obtained through the corresponding mathematical relationship for three-dimensional reconstruction. For the structured light three-dimensional imaging system, it is necessary to restore the three-dimensional shape of the object through processes such as encoding and decoding of projection patterns, calibration of the camera and projector, and three-dimensional reconstruction.

[0029] Further, in step S200 of the embodiment of the present invention, according to the resolution of the projector, a corresponding number of horizontal Gray codes and vertical Gray codes are projected so that each pixel of the projection image can be uniquely encoded with a Gray code value; the line shift pattern performs pixel translation according to the period number of the Gray code. When the camera takes pictures of the object, there will always be shadow areas in the pictures due to different light angles. In order to identify and eliminate the shadow areas at the beginning to reduce the calculation workload, a pure black and white pattern is projected onto the object, which can be used to subtract the shadow area.

[0030] Specifically, in step S200 of the embodiment of the present invention, for any two adjacent code values of the Gray code, only one code value is different and the other code values are the same. The n-bit Gray code has 2 n encoded numbers. Taking the four-bit Gray code as an example, sixteen encoded numbers will be generated, which are composed of the decimal integers from zero to fifteen, and are converted into the corresponding binary numbers in the order of numerical size, and then the binary numbers generate the corresponding Gray code numbers through exclusive OR operations. During measurement, the black part in the image is the number 0, and the white part is the number 1. An N-order Gray code pattern is projected onto the object surface to encode the object information. The specific pattern is as Figure 3 shown. The corresponding line shift pattern generates a specific pattern according to the Gray code period number. In step S200, an eight-bit Gray code is generated, and the resolution of the projector is 1280*720. Therefore, the line shift pattern is designed with one cycle for every 5 pixels, including 1 bright stripe with a pixel value of 1 and 4 dark stripes with a pixel value of 0. The specific pattern shape is as Figure 4As shown, each time one pixel is moved, and according to the designed structured light three-dimensional imaging system, it is moved 5 times in total, and the bright line can traverse all pixel points in space.

[0031] Furthermore, in step S300 of the embodiment of the present invention, using the image collected by the camera, the Gray code is expanded according to levels. The line shift pattern extracts the center points of the line shift stripes according to the Steger algorithm, and the pixel points on the image are expanded to obtain the code value corresponding to each pixel.

[0032] Specifically, in step S300 of the embodiment of the present invention, each pixel encoded by the projector is decoded one by one, and the corresponding feature points are found in the camera system and the projector system respectively through the stripe pattern of the deformed object surface captured. To find the "homonymous points" of the optical machine corresponding to the camera vision system, it is necessary to project stripe patterns in horizontal and vertical directions for decoding. The center of the bright stripe of the line shift pattern can be extracted by the Steger algorithm. The captured picture is subjected to Gaussian filtering, and then the Hessian matrix is calculated to obtain the normal direction of the light stripe. Finally, the sub-pixel position is obtained by using Taylor expansion in the normal direction. According to the projected Gray code pattern, the pixels in each period can be decoded to obtain the corresponding code value.

[0033] Furthermore, in step S400 of the embodiment of the present invention, a calibration board for the structured light three-dimensional imaging system is made according to the triangular relationship between the camera and the projector. The internal and external parameters of the camera are calibrated according to the Zhang Zhengyou calibration method, and the internal and external parameters of the projector are deduced according to the encoded phase information of the feature points of the calibration board.

[0034] Specifically, in step S400 of the embodiment of the present invention, the calibration board composed of two-dimensional grids is used for calibration by the Zhang's calibration method. The pictures of the calibration board in different poses are collected, the pixel coordinates of the corner points in the pictures are extracted, the initial values of the internal and external parameters of the camera are calculated through the homography matrix, the distortion coefficients are estimated by the nonlinear least squares method, and finally the parameters are optimized by the maximum likelihood estimation method to calculate the internal and external parameters of the camera, where a calibration board with a suitable specification is designed according to the built system. The decoding values obtained in step S300 are subjected to bilinear interpolation to obtain the decoding values of the feature points of the calibration board. Since the projector is different from the camera and cannot directly determine the corresponding code of the object, the unique "homonymous point" on the projector is determined through horizontal and vertical directions, and then the internal and external parameters of the projector are calculated in the same way as calibrating the camera.

[0035] Specifically, in step S500 of the embodiment of the present invention, through the known parameter information obtained in step S300 and step S400, the three-dimensional points of the object to be detected are deduced according to the mathematical formula to satisfy the following formula:

[0036]

[0037] In the above formula, represents the elements in the camera projection matrix, represents the elements in the projector matrix, (u c , v c ) represents the pixel points on the camera, and u p is the abscissa of the corresponding pixel point on the projector. Using the parameters obtained through system calibration, the coordinates of the object to be measured in the world coordinate system can be calculated using the above formula, realizing the three-dimensional reconstruction of the object to be measured. At this time, only the line shift and Gray code pattern in one direction need to be projected.

[0038] Specifically, in the embodiments of the present invention, an 8-level Gray code pattern is designed, in both horizontal and vertical directions, and 5 line shift patterns, in both horizontal and vertical directions. Each Gray code pattern requires two positive and negative images, making the decoding more robust. The Zhang Zhengyou calibration method is used to calibrate the internal and external parameters of the camera. Using the principle of homologous points of the coding values, the feature points of the calibration board from the perspective of the projector are extracted to calibrate the internal and external parameters of the projector. The Steger algorithm is used to extract the center points of the bright stripes. According to the periodicity, the coding of each pixel point on the image is expanded. Due to the discrete characteristics of the Gray code, the bilinear interpolation method is used for coding completion. The decoding value of the center of the circular dot calibration board is extracted for projector calibration.

[0039] Let B represent the binary code and G represent the converted Gray code, where B = B n-1 B n-2 …B1B0, B n-1 represents the highest bit of the binary code, G = G n-1 G n-2 …G1G0, G n-1 represents the highest bit of the Gray code, B0 and G0 are the corresponding lowest bits, and the highest bits of both are kept unchanged during the conversion (B n-1 = G n-1 ), the second highest bit is obtained by the exclusive OR of B n-1 and B n-2 , and the following digits are also converted according to this rule. Then the expression of the conversion relationship is:

[0040]

[0041] The projector projects the projection patterns as shown Figure 3 at times t0, t1, t2, t3..., and the camera sequentially acquires and stores the pictures at the corresponding times. The projection pattern of the projector at the last time divides all the patterns into 2 n striped areas. The line shift pattern shifts a bright stripe with a pixel value of 1 by one pixel every 5 pixels. The bilinear interpolation is used to extract the center points of the stripes. According to the Steger algorithm, finally, the period is expanded:

[0042] linegray = 5 × T + k

[0043] Where T refers to the decoded value of the Gray code, the initial value of k is 0, and it is incremented by 1 each time a line shift pattern is read.

[0044] For the center of the bright stripe of the line shift pattern, it is necessary to find the normal direction of the line shift bright stripe from the Hessian matrix. Expand its gray level distribution according to the Taylor polynomial in the normal direction of the optical stripe, and the obtained maximum value is the sub-pixel coordinate of the optical stripe in this normal direction. For a two-dimensional discrete image I(u, v), the Hessian matrix can be expressed as:

[0045]

[0046] Where u and v represent the row coordinate and column coordinate of the pixel, and I uv represents the gray level of the pixel (u, v), which can also be called the gray level distribution function. And I uu , I uv and I vv can all be obtained through the convolution operation of I uv and the two-dimensional Gaussian function G(u, v).

[0047] The main function of the two-dimensional Gaussian function is to make the gray level distribution characteristics of the optical stripe more obvious. At the pixel (u, v), the Hessian matrix has two eigenvectors, one of which has a larger absolute value and is the normal direction vector at this pixel, while the other is the tangential direction vector. Therefore, the normal direction can be calculated by finding the eigenvectors of the Hessian matrix. For a certain pixel point H(u0, v0), the Hessian matrix in the second-order Taylor expansion is:

[0048]

[0049] The eigenvalues and eigenvectors obtained from the Hessian matrix at this point correspond to the normal direction at this point and the second-order directional derivative in this direction respectively. The unit vector of the normal direction is: e = [e u , e v , and the pixel point I = (u0 + t·e u , v0 + t·e v ) in the normal direction of the optical stripe can be represented by the gray level of the pixel (u0, v0)

[0050] and the second-order Taylor expansion polynomial as:

[0051] I = (u0 + t·e u , v0 + t·e v ) = I(u0, v0) + t·e[I u , I u T+t·e·H(u,v)·e T

[0052]

[0053] Then substitute t (i.e., the Taylor expansion) into it to obtain the sub-pixel coordinates of the center of the light stripe.

[0054] For the calibration of the camera system and the opto-mechanical system, as well as the three-dimensional reconstruction of the object to be detected, more specifically, it is achieved by analyzing the imaging model of the system equipment, constructing coordinate systems, performing mutual conversions between coordinate systems, obtaining the relevant parameters for system calibration, and realizing three-dimensional imaging. Regarding the projector as an inverse camera, after system calibration, the projection matrix can be obtained.

[0055]

[0056] Where A c represents the projection matrix of the camera, and A p represents the projector matrix. Then there are the following functional relationships among the world coordinate system, the pixel coordinate systems of the camera and the projector:

[0057] s c [u c v c 1] T = A c [X W Y W Z W 1] T

[0058] s p [u p v p 1] T = A p [X W Y W Z W 1] T

[0059] In the formula, s c and s p represent the scale factors of the camera and the projector respectively. Through the line shift method combined with the Gray code, the encoded value φ a of each pixel point can be obtained. According to mathematical formula derivation, the three-dimensional points of the object to be detected satisfy the following formula:

[0060]

[0061] In the formula represents the elements in the camera projection matrix, represents the elements in the projector matrix, (u c, v c ) represents the pixel points on the camera, u p is the abscissa of the corresponding pixel point on the projector. With the parameters obtained through system calibration, the coordinates of the object to be measured in the world coordinate system can be calculated using the above formula, realizing the three-dimensional reconstruction of the object to be measured. At this time, only the line shift and Gray code pattern in one direction need to be projected.

[0062] Although one or more embodiments of the present invention have been described above, those of ordinary skill in the art should be aware that the present invention can be implemented in any other form without departing from its gist and scope. Therefore, the embodiments described above are illustrative rather than restrictive, and many modifications and substitutions will be obvious to those of ordinary skill in the art without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. An imaging method based on line-shifted Gray code, characterized in that, Including: Step S100: Construct a structured light three-dimensional imaging system, which includes a camera, a projector, and a computer; Step S200: Project and collect line shift images and Gray code images onto the target object according to the structured light three-dimensional imaging system; Step S300: Decode the feature points of the target object based on the collected line shift images and Gray code images to obtain the decoded information of the target object; Step S400: Make a calibration board according to the measurement field of view size of the structured light three-dimensional imaging system, and obtain the internal and external parameters of the camera and the internal and external parameters of the projector according to the calibration board; Step S500: Reconstruct the depth map and three-dimensional point cloud of the target object based on the decoded information of the target object, the internal and external parameters of the camera, and the internal and external parameters of the projector, and based on the principle of triangulation.

2. The imaging method based on line-shifted Gray code according to claim 1, wherein, In step S200, project a corresponding number of horizontal Gray codes and vertical Gray codes according to the resolution of the projector, so that each pixel of the projection image can be uniquely encoded with a Gray code value; The line shift pattern performs pixel translation according to the period number of the Gray code.

3. The imaging method based on line-shifted Gray code according to claim 2, wherein In step S300, use the images collected by the camera to expand the Gray code by level. The line shift pattern extracts the center points of the line shift stripes according to the Steger algorithm, and expand the pixel points on the image to obtain the code value corresponding to each pixel.

4. The imaging method based on line-shifted Gray code according to claim 3, wherein In step S400, establish a triangular relationship between the camera and the projector to make the calibration board of the structured light three-dimensional imaging system, calibrate the internal and external parameters of the camera according to the Zhang Zhengyou calibration method, and calculate the internal and external parameters of the projector according to the encoded phase information of the calibration board feature points.

Images