A three-dimensional imaging device and method based on displacement imaging and structured light fusion

Abstract

The present invention discloses a three-dimensional imaging device and method based on displacement imaging and structured light fusion, the method comprising: obtaining a three-dimensional depth image I of a calibration plate by a structured light imaging method, obtaining a three-dimensional depth image II of the calibration plate by a displacement imaging method; calculating a transformation matrix between the three-dimensional depth image I and the three-dimensional depth image II; obtaining a three-dimensional depth image III of a product to be tested by a structured light imaging method, obtaining a three-dimensional depth image IV of the product to be tested by a displacement imaging method; and fusing the three-dimensional depth image III and the three-dimensional depth image IV using the transformation matrix obtained in step S2. The height information of the mirror part of the product obtained by the displacement imaging method is fused with the height information of the regular part obtained by the structured light imaging method by the transformation matrix, so as to realize the three-dimensional imaging of the product with the mirror, and the average distance of the fusion of the two imaging methods is less than 1 micron, so as to ensure the accuracy of the three-dimensional imaging of the product with the mirror.

Description

Technical Field

[0001] The present invention relates to the field of image acquisition technology, and in particular to a three-dimensional imaging device and method based on displacement imaging and structured light fusion. Background Art

[0002] Visual inspection refers to the technology of converting the captured target into image signals through machine vision products, transmitting them to the image processing system to extract the target features, and then realizing on-site control based on the judgment results.

[0003] Structured light and displacement imaging are commonly used 3D imaging methods. Structured light is a method in which a camera collects specific light information projected by a projector onto the surface of an object and the background, and then calculates the position and depth of the object based on the changes in the light signal caused by the object, thereby restoring the entire 3D space. However, since structured light relies on the reflected light of an object and obtains height information through stripe distortion, when capturing images of products with mirrors, the stripes reflected by the mirrors are too weak, making it difficult to obtain and resulting in poor calculation results, or the brightness of the mirror reflections is too strong, resulting in overexposure, and accurate phases cannot be obtained. Although displacement imaging does not rely on the reflection of an object, it relies on the surrounding pixels to calculate the focusing factor, and is greatly affected by noise and displacement accuracy, and its absolute accuracy is not as good as that of structured light.

[0004] Based on the above technical problems, the present application proposes a three-dimensional imaging device and method based on displacement imaging and structured light fusion. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional imaging device and method based on displacement imaging and structured light fusion to solve the technical problems mentioned in the background technology. The purpose of the present invention is achieved through the following technical solutions:

[0006] A three-dimensional imaging device based on displacement imaging and structured light fusion includes an industrial computer, a control mechanism, a motion mechanism, a projector, a light source and a camera. The control mechanism, the motion mechanism and the projector are all electrically connected to the industrial computer. The projector and the camera are both installed on the motion mechanism. A grating ruler is installed on the motion mechanism. The grating ruler, the light source and the camera are all electrically connected to the control mechanism.

[0007] Furthermore, the control mechanism includes a power supply module, a control module, a grating scale IO module and a communication module. The power supply module is electrically connected to the control module, the grating scale IO module and the communication module respectively. The grating scale is electrically connected to the control module through the grating scale IO module. The control module is electrically connected to the communication module, the camera and the light source respectively. The communication module is electrically connected to the industrial control computer.

[0008] Furthermore, the control module includes a control unit, a storage unit, a camera trigger unit and a light source trigger unit. The control unit is electrically connected to the power supply module, the storage unit, the camera trigger unit and the light source trigger unit, respectively. The communication module is electrically connected to the storage unit, the grating scale IO module is electrically connected to the control unit, the camera is electrically connected to the camera trigger unit, and the light source is electrically connected to the light source trigger unit.

[0009] A three-dimensional imaging method based on displacement imaging and structured light fusion includes the following steps:

[0010] Step S1, obtaining a three-dimensional depth image I of a calibration plate by a structured light imaging method, and obtaining a three-dimensional depth image II of the calibration plate by a displacement imaging method;

[0011] Step S2, calculating the transformation matrix between the three-dimensional depth image I and the three-dimensional depth image II;

[0012] Step S3, obtaining a three-dimensional depth image III of the product to be tested by a structured light imaging method, and obtaining a three-dimensional depth image IV of the product to be tested by a displacement imaging method;

[0013] Step S4: fuse the three-dimensional depth image III and the three-dimensional depth image IV using the transformation matrix obtained in step S2.

[0014] Furthermore, the structured light imaging method includes:

[0015] Step S01, projecting grating stripes onto a calibration plate through a projector, and acquiring a stripe image through a camera;

[0016] Step S02, synchronously moving the projector and the camera, and recording the moving height;

[0017] Step S03, repeating step S01 and step S02 until the measurement range of the camera is exceeded;

[0018] Step S04, calculating the relationship between the phase shift value and the moving height of the grating fringe, and saving it as a calibration file;

[0019] Step S05: project grating stripes onto the calibration plate or the product to be tested through a projector, obtain a stripe image through a camera, and calculate a three-dimensional depth image I of the calibration plate or a three-dimensional depth image III of the product to be tested according to the calibration file.

[0020] Further, the displacement imaging method includes:

[0021] Step S11, moving the camera through the motion mechanism to obtain sequence images at different heights;

[0022] Step S12: using a focus evaluation algorithm to evaluate the focus value of the pixel point in each sequence image;

[0023] Step S13: acquiring a focus curve according to the focus value, and using the focus curve to acquire a three-dimensional depth image II of the calibration plate or a three-dimensional depth image IV of the product to be measured.

[0024] Furthermore, step S2 specifically includes:

[0025] Step S21, converting the three-dimensional depth image I into a point cloud P, and converting the three-dimensional depth image II into a point cloud Q;

[0026] Step S22: for each point q in the point cloud Q i , find the point p closest to it in the point cloud P j ;

[0027] Step S23, calculating the transformation matrix from point cloud Q to point cloud P through the nearest points, and calculating the average distance;

[0028] Step S24: Apply the transformation matrix obtained in step S23 to the point cloud Q, and repeat steps S22 to S24 until the average distance is less than a set threshold.

[0029] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0030] By integrating the height information of the mirror part of the product obtained by the displacement imaging method with the height information of the regular part obtained by the structured light imaging method through the transformation matrix, three-dimensional imaging of products with mirror surfaces is achieved. The average distance of the fusion of the two imaging methods is less than 1 micron, which ensures the accuracy of three-dimensional imaging of products with mirror surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings in the specification, which constitute a part of this application, are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 This is a schematic diagram of the structure of Example 1 of the present application;

[0033] Figure 2 This is a flow chart of the three-dimensional imaging method according to an embodiment of the present application;

[0034] Figure 3 This is a flow chart of the structured light imaging method according to an embodiment of the present application;

[0035] Figure 4 This is a flow chart of the displacement imaging method according to an embodiment of the present application;

[0036] Figure 5 This is a flow chart of the transformation matrix calculation method according to an embodiment of the present application.

[0037] Markings in the attached figure: 1. industrial computer; 2. control mechanism; 21. power supply module; 22. control module; 221. control unit; 222. storage unit; 223. camera trigger unit; 224. light source trigger unit; 23. communication module; 24. grating ruler IO module; 3. motion mechanism; 4. projector; 5. light source; 6. camera; 7. grating ruler. DETAILED DESCRIPTION

[0038] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the drawings and specific implementation methods of the specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1

[0040] like Figure 1 A three-dimensional imaging device based on displacement imaging and structured light fusion is shown, comprising an industrial computer 1, a control mechanism 2, a motion mechanism 3, a projector 4, a light source 5 and a camera 6. The control mechanism 2, the motion mechanism 3 and the projector 4 are all electrically connected to the industrial computer 1. The motion mechanism 3 is a Z-axis linear module, and the projector 4 and the camera 6 are both installed on the Z-axis linear module. The industrial computer 1 controls the movement of the Z-axis linear module to realize the up and down movement of the projector 4 and the camera 6. A grating ruler 7 is installed on the Z-axis linear module, and the grating ruler 7 is electrically connected to the control mechanism 2, and is used to feed back the position information of the motion mechanism 3 to the control mechanism 2. The projector 4 is a DLP projection optical machine. The industrial computer 1 pre-burns the projection stripes into the projector 4 and selects the projection. The projection stripes are distorted on the object, and the height information of the product to be tested is obtained by calculating the degree of distortion. The light source 5 is installed above the product to be tested, and is used to illuminate the product to be tested. The camera 6 obtains the image of the product to be tested and saves it to the industrial computer 1, and obtains the height information of the product to be tested through image processing software.

[0041] like Figure 1 As shown, the control mechanism 2 includes a power supply module 21, a control module 22, a communication module 23 and a grating scale IO module 24. The power supply module 21 is electrically connected to the control module 22, the communication module 23 and the grating scale IO module 24 respectively. The power supply module 21 is responsible for receiving the 24V DC power provided by the electromechanical control part, and outputting high-precision voltage and current to each module after filtering.

[0042] The control module 22 includes a control unit 221, a storage unit 222, a camera trigger unit 223 and a light source trigger unit 224. The control unit 221 is electrically connected to the storage unit 222, the camera trigger unit 223 and the light source trigger unit 224 respectively. The communication module 23 is a serial communication module, which is electrically connected to the storage unit 222 and is used to write control parameters into the control module 22. The grating ruler IO module 24 is electrically connected to the control unit 221. The grating ruler IO module 24 is responsible for receiving the differential signal of the grating ruler 7, obtaining the moving distance of the motion mechanism 3 by calculating the differential signal, and cooperating with the control module 22 to trigger the camera to obtain a sequence of images with equal or unequal spacing. The camera 6 is electrically connected to the camera trigger unit 223, and the light source 5 is electrically connected to the light source trigger unit 224. The control unit 221 controls the camera trigger unit 223 and the light source trigger unit 224 to synchronously send a trigger signal to achieve synchronous triggering of the camera 6 and the light source 5.

[0043] Example 2

[0044] like Figure 2 A three-dimensional imaging method based on displacement imaging and structured light fusion is shown, comprising the following steps:

[0045] Step S1, obtaining a three-dimensional depth image I of a calibration plate by a structured light imaging method, and obtaining a three-dimensional depth image II of the calibration plate by a displacement imaging method.

[0046] like Figure 3 As shown, the structured light imaging method includes:

[0047] Step S01, adjust the projector and the camera to the same focal plane, and move to the highest point of the range through the motion mechanism, project grating stripes onto the calibration plate through the projector, and the camera acquires the stripe image and saves it to the industrial computer;

[0048] Step S02, synchronously moving the projector and the camera downward, and recording the moving heights of the projector and the camera;

[0049] Step S03, repeating step S01 and step S02 until the measurement range of the camera is exceeded;

[0050] Step S04, calculating the relationship between the phase shift value and the moving height of the grating fringe, and saving it as a calibration file;

[0051] Step S05: project grating stripes onto the calibration plate through a projector, obtain a stripe image through a camera, and calculate a three-dimensional depth image I of the calibration plate according to the calibration file.

[0052] like Figure 4 As shown, the displacement imaging method includes:

[0053] Step S11, the camera is moved to the highest point of the measuring range by the motion mechanism, the motion mechanism controls the camera to descend, and the control mechanism synchronously triggers the camera and the light source through the position feedback of the grating ruler to obtain a sequence of images at different height positions of the calibration plate;

[0054] Step S12: using a focus evaluation algorithm to evaluate the focus value of the pixel point in each sequence image;

[0055] Focus evaluation algorithms can be roughly divided into four categories: evaluation algorithms based on gradient edges (spatial domain), frequency domain evaluation algorithms, evaluation algorithms based on information entropy, and evaluation algorithms based on image statistics. Among them, evaluation algorithms based on gradient edges (spatial domain) include the sum of absolute grayscale difference function (SMD), energy gradient function (EOG), Roberts function, Krish function, Tenengrad function, Brenner function, Laplace function, improved Laplace function (SML) and FSWM function; frequency domain evaluation algorithms include operators based on DCT transform (discrete cosine transform) and operators based on Fourier transform; evaluation algorithms based on image statistics include variance function, autocorrelation function, Range function, Menmay function and Masgrn function. All of the above are existing technologies and will not be described here.

[0056] Step S13, obtain the focus curve of the pixel point according to the focus value, extract the peak position of the focus curve, use it as the rough focus position of the pixel point, use the Gaussian curve fitting algorithm to locate the precise focus position, and traverse all the pixels to obtain the three-dimensional depth image II of the calibration plate.

[0057] Step S2: Calculate the transformation matrix between the three-dimensional depth image I and the three-dimensional depth image II.

[0058] like Figure 5 As shown, step S2 specifically includes:

[0059] Step S21, converting the three-dimensional depth image I into a point cloud P, and converting the three-dimensional depth image II into a point cloud Q;

[0060] Step S22: For each point in the point cloud Q, , find the point closest to it in the point cloud P ;

[0061] Step S23, establish a least squares equation through the nearest points, calculate the transformation matrix from point cloud Q to point cloud P, and calculate the average distance;

[0062] Step S24: Apply the transformation matrix obtained in step S23 to the point cloud Q, and repeat steps S22 to S24 until the average distance is less than a set threshold.

[0063] Step S3: obtaining a three-dimensional depth image III of the product to be tested by a structured light imaging method, and obtaining a three-dimensional depth image IV of the product to be tested by a displacement imaging method.

[0064] The structured light imaging methods for the products to be tested include:

[0065] The grating stripes are projected onto the product to be tested through a projector, the stripe image is acquired through a camera, and the three-dimensional depth image I of the product to be tested is calculated according to the calibration file.

[0066] like Figure 4 As shown, the displacement imaging method of the product to be tested includes:

[0067] Step S11, the camera is moved to the highest point of the measuring range by the motion mechanism, the motion mechanism controls the camera to descend, and the control mechanism synchronously triggers the camera and the light source through the position feedback of the grating ruler to obtain sequence images of the product to be tested at different height positions;

[0068] Step S12: using a focus evaluation algorithm to evaluate the focus value of the pixel point in each sequence image;

[0069] Step S13, obtaining a focus curve of the pixel point according to the focus value, extracting the peak position of the focus curve, using this as the rough focus position of the pixel point, locating the precise focus position in combination with the Gaussian curve fitting algorithm, and traversing all the pixels to obtain the three-dimensional depth image IV of the product to be tested.

[0070] Step S4, converting the three-dimensional depth image III and the three-dimensional depth image IV into point clouds, and then fusing the three-dimensional depth image III and the three-dimensional depth image IV using the transformation matrix obtained in step S2 to obtain a three-dimensional depth image based on the fusion of displacement imaging and structured light imaging.

[0071] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0072] By integrating the height information of the mirror part of the product obtained by the displacement imaging method with the height information of the regular part obtained by the structured light imaging method through the transformation matrix, three-dimensional imaging of products with mirror surfaces is achieved. The average distance of the fusion of the two imaging methods is less than 1 micron, which ensures the accuracy of three-dimensional imaging of products with mirror surfaces.

[0073] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional imaging method based on displacement imaging and structured light fusion, characterized in that: The following steps are involved: Step S1, obtaining a three-dimensional depth image I of a calibration plate by a structured light imaging method, and obtaining a three-dimensional depth image II of the calibration plate by a displacement imaging method; Step S2, calculating the transformation matrix between the three-dimensional depth image I and the three-dimensional depth image II; Step S3, obtaining a three-dimensional depth image III of the product to be tested by a structured light imaging method, and obtaining a three-dimensional depth image IV of the product to be tested by a displacement imaging method; Step S4, using the transformation matrix obtained in step S2 to fuse the three-dimensional depth image III and the three-dimensional depth image IV; The structured light imaging method comprises: Step S01, projecting grating stripes onto a calibration plate through a projector, and acquiring a stripe image through a camera; Step S02, synchronously moving the projector and the camera, and recording the moving height; Step S03, repeating step S01 and step S02 until the measurement range of the camera is exceeded; Step S04, calculating the relationship between the phase shift value and the moving height of the grating fringe, and saving it as a calibration file; Step S05, projecting grating stripes onto the calibration plate or the product to be tested through a projector, acquiring a stripe image through a camera, and calculating a three-dimensional depth image I of the calibration plate or a three-dimensional depth image III of the product to be tested according to the calibration file; The displacement imaging method comprises: Step S11, moving the camera through the motion mechanism to obtain sequence images at different heights; Step S12: using a focus evaluation algorithm to evaluate the focus value of the pixel point in each sequence image; Step S13: acquiring a focus curve according to the focus value, and using the focus curve to acquire a three-dimensional depth image II of the calibration plate or a three-dimensional depth image IV of the product to be measured.

2. The three-dimensional imaging method based on displacement imaging and structured light fusion according to claim 1, characterized in that: Step S2 specifically includes: Step S21, converting the three-dimensional depth image I into a point cloud P, and converting the three-dimensional depth image II into a point cloud Q; Step S22: for each point in the point cloud Q , find the point closest to it in the point cloud P ; Step S23, calculating the transformation matrix from point cloud Q to point cloud P through the nearest points, and calculating the average distance; Step S24: Apply the transformation matrix obtained in step S23 to the point cloud Q, and repeat steps S22 to S24 until the average distance is less than a set threshold.

3. A device using the three-dimensional imaging method based on displacement imaging and structured light fusion as described in any one of claims 1 or 2, characterized in that: It includes an industrial computer, a control mechanism, a motion mechanism, a projector, a light source and a camera. The control mechanism, the motion mechanism and the projector are all electrically connected to the industrial computer. The motion mechanism is a Z-axis linear module. The projector and the camera are both installed on the Z-axis linear module. The industrial computer controls the movement of the Z-axis linear module to achieve the up and down movement of the projector and the camera. A grating ruler is installed on the motion mechanism. The grating ruler, the light source and the camera are all electrically connected to the control mechanism. The grating ruler is used to feed back the position information of the motion mechanism to the control mechanism. The projection stripes are pre-burned into the projector and the projection is selected by the industrial computer. The projection stripes are distorted on the object, and the height information of the product to be tested is obtained by calculating the degree of distortion. The light source is installed above the product to be tested and is used to illuminate the product to be tested; the camera is used to obtain an image of the product to be tested and save it to the industrial computer, and obtain height information of the product to be tested through image processing software.

4. The three-dimensional imaging device based on displacement imaging and structured light fusion according to claim 3, characterized in that: The control mechanism includes a power supply module, a control module, a grating ruler IO module and a communication module. The power supply module is electrically connected to the control module, the grating ruler IO module and the communication module respectively. The grating ruler is electrically connected to the control module through the grating ruler IO module. The control module is electrically connected to the communication module, the camera and the light source respectively. The communication module is electrically connected to the industrial computer.

5. The three-dimensional imaging device based on displacement imaging and structured light fusion according to claim 4, characterized in that: The control module includes a control unit, a storage unit, a camera trigger unit and a light source trigger unit. The control unit is electrically connected to the power supply module, the storage unit, the camera trigger unit and the light source trigger unit respectively. The communication module is electrically connected to the storage unit, the grating ruler IO module is electrically connected to the control unit, the camera is electrically connected to the camera trigger unit, and the light source is electrically connected to the light source trigger unit.

Images