Multi-lens coplanar correction system and method

Abstract

The invention discloses a multi-lens coplane correction system and a multi-lens coplane correction method, the multi-lens coplane correction method is suitable for a three-dimensional imaging device with a plurality of lenses, and the method comprises the following steps: the three-dimensional imaging device shoots a reference surface of an object through the plurality of lenses so as to respectively generate a plurality of three-dimensional images, the arithmetic device calculates a plurality of surface equations according to three-dimensional information of the three-dimensional image and calculates a plurality of coordinate systems corresponding to the plurality of lenses according to the surface equations, one of the coordinate systems is a first coordinate system, and the other coordinate systems are second coordinate systems. The arithmetic device calculates a correction matrix of each second coordinate system relative to the first coordinate system at least according to the coordinate systems, and the actuating device performs adjustment according to the correction matrixes.

Description

Technical Field

[0001] The present invention relates to an array optical system, and in particular to a multi-lens coplanarity correction system and method. Background Art

[0002] An array optical system can be used, for example, in the field of white light interference. The main components of this system are multiple channels arranged in an array form, and each channel is provided with a lens or a camera. These channels must act simultaneously to obtain complete information of each channel. The multiple lenses of the array optical system must have the same focal length, and when establishing the system, it is necessary to ensure that these lenses are located on the same plane. During the assembly process of the array optical system, it is necessary to adjust the tilt and height of the lenses in each channel to ensure that the lenses in different channels meet the coplanarity property.

[0003] In the prior art, the laser alignment method can only provide error information of the lens height. Moreover, the optical path design of laser alignment is difficult. Once the optical path is offset during erection, it may never be possible to achieve coplanarity of multiple lenses. On the other hand, the method of adjusting coplanarity by observing the imaging clarity between different lenses relies on the personal experience of the operator, and this method cannot be quantified, making the operation difficult. Summary of the Invention

[0004] In view of this, the present invention provides a multi-lens coplanarity correction system and method, which can obtain the height and angle information of each lens for coplanarity adjustment. It not only has a higher success rate of coplanarity adjustment, but also can quantitatively present the height error and angle error of each lens, reducing the operation threshold of users and being more suitable for introducing an automatic adjustment mechanism.

[0005] A multi-lens coplanarity correction method according to an embodiment of the present invention is applicable to a three-dimensional imaging device having multiple lenses. The method includes: the three-dimensional imaging device captures an object through the multiple lenses to respectively generate multiple three-dimensional images. The object includes a reference surface with known spatial information. The computing device calculates multiple surface equations based on the multiple three-dimensional information of the multiple three-dimensional images. The computing device calculates multiple coordinate systems corresponding to the multiple lenses based on the multiple surface equations. Each coordinate system has a specific point as the origin. One of the multiple coordinate systems is the first coordinate system, and each of the multiple coordinate systems other than the first coordinate system is the second coordinate system. The computing device calculates at least a correction matrix of the second coordinate system relative to the first coordinate system based on the multiple coordinate systems. The actuating device adjusts the lens corresponding to each second coordinate system according to the correction matrix.

[0006] A multi-lens coplanarity correction system according to an embodiment of the present invention includes: an object, a three-dimensional imaging device, an arithmetic device, and an actuating device. The object includes a reference surface with known spatial information. The three-dimensional imaging device has a plurality of lenses for photographing the reference surface to respectively generate a plurality of three-dimensional images. The arithmetic device is electrically connected to the three-dimensional imaging device to obtain the plurality of three-dimensional images. The arithmetic device is used to execute a plurality of instructions to cause a plurality of operations, and these operations include: calculating a plurality of surface equations based on the three-dimensional information of the plurality of three-dimensional images; calculating a plurality of coordinate systems corresponding to the plurality of lenses based on these surface equations, wherein each coordinate system has a specific point as the origin, one of the plurality of coordinate systems is the first coordinate system, and each of the plurality of coordinate systems other than the first coordinate system is the second coordinate system; calculating at least a correction matrix of the second coordinate system relative to the first coordinate system based on the plurality of coordinate systems. The actuating device is electrically connected to the arithmetic device. The actuating device adjusts the lens corresponding to the second coordinate system according to the correction matrix.

[0007] The above description of the content of the present invention and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and provide a further explanation of the scope of the patent application of the present invention. Brief Description of the Drawings

[0008] Figure 1 is a block diagram of a multi-lens coplanarity correction system shown in an embodiment of the present invention;

[0009] Figure 2 is a flowchart of a multi-lens coplanarity correction method shown in an embodiment of the present invention;

[0010] Figure 3 is a top view example after reconstructing the point cloud of the three-dimensional images captured by multiple lenses;

[0011] Figure 4 is a side view example after reconstructing the point cloud of the three-dimensional images captured by multiple lenses;

[0012] Figure 5 is Figure 2 a detailed flowchart of a step in; and

[0013] Figure 6 is a schematic diagram of a global coordinate system and a plurality of coordinate systems.

[0014] Symbol Description

[0015] 100: Multi-lens coplanarity correction system

[0016] 10: Object

[0017] 11: Reference surface

[0018] 30: Three-dimensional imaging device

[0019] 32, 34: Lens

[0020] 50: Arithmetic device

[0021] 70: Actuating device

[0022] A1, A2: Shooting range

[0023] A1’, A2’: Point cloud

[0024] R1, R2: Region of interest

[0025] P1, P2: Center point

[0026] S1 - S6, S31 - S33: Steps

[0027] ΔT y : Displacement correction amount

[0028] ΔR z : Rotation correction amount Detailed implementation manner

[0029] The detailed features and advantages of the present invention are described in detail in the following embodiments. The content is sufficient for any person skilled in the relevant art to understand the technical content of the present invention and implement it accordingly. And according to the content disclosed in this specification, claims and drawings, any person skilled in the relevant art can easily understand the relevant purposes and advantages of the present invention. The following embodiments further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention in any way.

[0030] Figure 1 It is a structural diagram of a multi - lens coplanar correction system shown according to an embodiment of the present invention. The multi - lens coplanar correction system 100 includes an object 10, a three - dimensional imaging device 30, an arithmetic device 50, and an actuating device 70.

[0031] The object 10 has a reference surface 11 with known spatial information. In one embodiment, the reference surface 11 is a plane, and this plane includes a plurality of positions with the same height. In one embodiment, the object 10 is a step - height standard produced by VLSI Company, which is a standard part with a processed and flat surface. In another embodiment, the reference surface 11 is a surface that can be defined by a mathematical model, such as a curved surface, but the present invention is not limited thereto.

[0032] The three - dimensional imaging device 30 has a plurality of lenses 32, 34, and these lenses 32, 34 are used to photograph the reference surface 11 of the object 10 to respectively generate a plurality of three - dimensional images. As Figure 1As shown, the number of lenses 32 and 34 is at least two, but the present invention does not limit the upper limit of the number of lenses 32 and 34. In one embodiment, the three-dimensional imaging device 30 is a three-dimensional camera equipped with multiple lenses. In another embodiment, the three-dimensional imaging device 30 is a combination of multiple three-dimensional cameras. A three-dimensional camera generally refers to a camera that uses three-dimensional measurement technology to capture three-dimensional images. The three-dimensional measurement technology includes, for example, stereo vision, structured light, or time-of-flight (ToF), and the present invention is not limited thereto.

[0033] The three-dimensional image means that each pixel in the image contains three-dimensional coordinate information of the world coordinate system. In one embodiment, each lens (32 or 34) of the three-dimensional imaging device 30 directly captures an object and generates a three-dimensional image. In another embodiment, the three-dimensional image is generated by actuating the device 70 to adjust the height of the lenses 32 and 34 relative to the reference surface 11, capturing multiple two-dimensional images of the object 10 at multiple different heights, and the three-dimensional imaging device 30 then establishes a three-dimensional image corresponding to each lens (32 or 34) based on these two-dimensional images.

[0034] Please note that the size of the reference surface 11 must be larger than the visible range formed by all the lenses 32 and 34, as Figure 1 shown. In other words, when capturing the object 10, it is not necessary to move the three-dimensional imaging device 30 to enable all the lenses 32 and 34 to capture the reference surface 11. Regarding the installation positions of the lenses 32 and 34, it is necessary to ensure that there is no overlapping part of the reference surface 11 captured by any two of all the lenses 32 and 34.

[0035] The computing device 50 is electrically connected to the three-dimensional imaging device 30. The computing device 50 obtains the multiple three-dimensional images captured by the three-dimensional imaging device 30, and executes multiple instructions based on these images to cause multiple operations, and finally outputs multiple correction matrices. The multiple operations correspond to the multiple steps included in the multi-lens coplanarity correction method according to an embodiment of the present invention, which will be described in detail later.

[0036] In one embodiment, the computing device 50 may be implemented using at least one of the following examples: a personal computer, a network server, a microcontroller (MCU), an application processor (AP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functions. The present invention does not limit the hardware type of the computing device 50.

[0037] The actuating device 70 is electrically connected to the computing device 50. The actuating device 70 is adjusted according to the calibration matrix to make all the lenses 32, 34 coplanar. In one embodiment, the actuating device 70 is, for example, a three-axis servo motor or any mechanical device capable of achieving three-axis adjustment. The present invention does not limit this.

[0038] Figure 2 FIG. is a flowchart of a multi-lens coplanarity calibration method according to an embodiment of the present invention, including steps S1 to S6. The method is applicable to a three-dimensional imaging device 100 having a plurality of lenses 32, 34.

[0039] Step S1, the three-dimensional imaging device 30 captures an object 10 through a plurality of lenses 32, 34 to respectively generate a plurality of three-dimensional images. The object 10 includes a reference surface 11 with known spatial information.

[0040] Step S2, the computing device 50 calculates a plurality of surface equations based on the three-dimensional information of the plurality of three-dimensional images. In one embodiment, the surface equation is a plane equation, and the following will be described with an example of a plane equation.

[0041] In one embodiment, before actually calculating the plane equation, the computing device 50 decodes the three-dimensional image to obtain a plurality of three-dimensional information of the reference surface 11. In one embodiment, this three-dimensional information is a point cloud.

[0042] The computing device 50 sets a region of interest (ROI) in the plurality of three-dimensional information (such as point cloud) corresponding to each three-dimensional image. Since the object being photographed is the reference surface 11, the computing device 50 can generate a virtual surface based on the plurality of three-dimensional information corresponding to each three-dimensional image, and then calculate the surface equation corresponding to this virtual surface.

[0043] Figure 3 It is a top - view example after reconstructing the point cloud from the three - dimensional images captured by lenses 32 and 34. Among them, R1 and R2 respectively represent the regions of interest set by the computing device 50 in the three - dimensional image (corresponding to the aforementioned virtual surface), and P1 and P2 respectively represent the center points of the regions of interest R1 and R2.

[0044] Figure 4 It is a side - view example after reconstructing the point cloud from the three - dimensional images captured by lenses 32 and 34. Please refer to Figure 1 and Figure 4 , the shooting ranges of lenses 32 and 34 on the reference surface 11 are A1 and A2 respectively. The point clouds corresponding to the shooting ranges A1 and A2 are A1’ and A2’ respectively. Since the spatial information of the reference surface 11 is known, when the reference surface 11 is a plane, if all the lenses 32 and 34 are coplanar, the point clouds reconstructed from multiple three - dimensional images should have the same height. And in the Figure 4 example, the point cloud A1’ has a displacement correction amount ΔT on the Y - axis y and a rotation correction amount ΔR on the Z - axis z , which means that lenses 32 and 34 are not coplanar, and one of them (such as lens 34) needs to be adjusted by the actuating device 70 to align with the other (such as lens 32).

[0045] Since multiple three - dimensional information used to calculate the plane equation is within the regions of interest R1 and R2, the computing device 50 can calculate multiple plane equations corresponding to the multiple lenses 32 and 34, as in the following method one.

[0046]

[0047] where, f 1 , f 2 , …, f N represent the plane equations, a 1 …a N , b 1 …b N , c 1 …c N , d 1 …d N represent the coefficients in the plane equation, and N represents the number of lenses. In the aforementioned embodiment, N = 2.

[0048] Step S3, the computing device 50 calculates multiple coordinate systems according to the multiple plane equations. Figure 5 It is a detailed flowchart of step S3, including steps S31 to S34. The computing device 50 executes once for each plane equation Figure 5The process shown is used to obtain a coordinate system. If the number of lenses is N, the arithmetic unit 50 will execute the process shown N times Figure 5 to calculate N coordinate systems. Each coordinate system has a specific point as its origin, and the specific point is, for example, the aforementioned center point P1 or P2.

[0049] In step S31, the arithmetic unit 50 determines the first vector based on the coefficients of the corresponding plane equation. The first vector is the normal vector of the plane. For example, for the plane equation f 1 = a 1 x + b 1 y + c 1 z + d 1 = 0, the first vector V z1 = (a, b, c).

[0050] In step S32, the second vector is randomly determined. The value of the second vector can be arbitrarily determined. For example, the second vector V x1 = (1, 0, 0).

[0051] In step S33, the arithmetic unit 50 calculates the cross product of the first vector and the second vector to obtain the third vector. In other words, the third vector V y1 = V z1 × V x1 . After executing step S33, it is at least ensured that the first vector V z1 is orthogonal to the third vector V y1 .

[0052] In step S34, the arithmetic unit 50 calculates the cross product of the first vector and the third vector to correct the second vector. In other words, the corrected second vector V x1 = V y1 × V z1 . After executing step S34, it can be ensured that the first vector V z1 is orthogonal to the second vector V x1 , and the third vector V y1 is orthogonal to the second vector V x1 , so that the three vectors V x1 , V y1 , V z1 are orthogonal to each other and form a coordinate system.

[0053] Each of the multiple coordinate systems mentioned in step S3 is composed of a first vector, a second vector, and a third vector. Each coordinate system corresponds to a lens (such as 32 or 34). In order to make all the lenses 32 and 34 coplanar, one of these coordinate systems is taken as a reference and called the first coordinate system, and each of the remaining coordinate systems that need to be adjusted is called the second coordinate system. The goal of the adjustment is to align each second coordinate system with the first coordinate system.

[0054] In the processes of step S4 and step S5, the arithmetic device calculates at least a correction matrix of the second coordinate system relative to the first coordinate system based on the plurality of coordinate systems.

[0055] Step S4, the arithmetic device calculates a plurality of transformation matrices based on the global coordinate system and the plurality of coordinate systems. Figure 6 is a schematic diagram of the global coordinate system (i.e., the world coordinate system) and the plurality of coordinate systems. Each transformation matrix is used to transform a corresponding coordinate system, such as {V x1 ,V y1 ,V z1} or {V x2 ,V y2 ,V z2}, to the global coordinate system {x, y, z}. The calculation method of the transformation matrix is as shown in Method 2 below:

[0056]

[0057] where M represents the transformation matrix, (x 0 ,y 0 ,z 0 ), (x 1 ,y 1 ,z 1 ), (x 2 ,y 2 ,z 2 ) represent the three-axis vectors constituting the coordinate system, and (P x ,P y ,P z ) represents a specific point, and the specific point is the coordinate of the center point P1 of the region of interest R1.

[0058] Step S5, the arithmetic device calculates a correction matrix of the second coordinate system relative to the first coordinate system based on the inverse matrix of the transformation matrix of the first coordinate system and the transformation matrix of the second coordinate system. The correction matrix is used to transform the second coordinate system into the first coordinate system, and its calculation method is as shown in Method 3 below:

[0059] M C =M 1 -1 ×M 2 (Equation 3)

[0060] where, M C represents the correction matrix, M 1 represents the transformation matrix of the first coordinate system, and M 2 represents the transformation matrix of the second coordinate system.

[0061] Step S6, the actuating device 70 is based on the correction matrix M CAdjust the lens corresponding to the second coordinate system. Specifically, the arithmetic device 50 can calculate the rotation correction amount and the displacement correction amount according to Equation 3, as shown in the following Equation 4:

[0062]

[0063] Where represents the rotation correction matrix, represents the displacement correction matrix. According to the rotation correction matrix R and the following Equation 5, the rotation correction amounts of the three axes can be calculated.

[0064]

[0065] Where θ x represents the rotation correction amount of the X axis, θ y represents the rotation correction amount of the Y axis, θ z represents the rotation correction amount of the Z axis, (θ x , θ y , θ z ) constitutes the rotation correction amount ΔR. The displacement correction amount ΔT Z can be directly obtained from the displacement correction matrix T in the calibration matrix M C . Generally speaking, this displacement correction amount is only applicable to the Z axis.

[0066] Finally, the actuating device 70 adjusts the lens corresponding to each second coordinate system according to the rotation correction amount ΔR and the displacement correction amount ΔT Z to align it with the lens corresponding to the first coordinate system.

[0067] In summary, an embodiment of the present invention provides a multi-lens coplanar calibration system and method, which can obtain the height sum and angle information of each lens for coplanar adjustment. It not only has a higher coplanar adjustment success rate, but also can quantitatively present the height error and angle error of each lens, reducing the operation threshold of users and being more suitable for introducing an automatic adjustment mechanism.

Claims

1. A multi-lens coplanarity correction method applicable to a three-dimensional imaging device having multiple lenses, the method comprises: using the three-dimensional imaging device to photograph an object through the lenses to respectively generate a plurality of three-dimensional images, wherein the object includes a reference surface with known spatial information; using an arithmetic device to calculate a plurality of surface equations based on the three-dimensional information of the three-dimensional images; using the arithmetic device to calculate a plurality of coordinate systems corresponding to the lenses based on the surface equations, wherein each of the coordinate systems has a specific point as the origin, one of the coordinate systems is the first coordinate system, and each of the coordinate systems other than the first coordinate system is a second coordinate system; using the arithmetic device to calculate at least based on the coordinate systems a correction matrix of the second coordinate system relative to the first coordinate system; and using an actuating device to adjust one of the lenses corresponding to the second coordinate system according to the correction matrix.

2. The multi-lens coplanarity correction method according to claim 1, wherein using the arithmetic device to calculate at least based on the coordinate systems a correction matrix of the second coordinate system relative to the first coordinate system comprises: calculating a plurality of transformation matrices based on a global coordinate system and the coordinate systems, wherein each of the transformation matrices is used for transformation between the corresponding one of the coordinate systems and the global coordinate system; and calculating the correction matrix based on the transformation matrix of the first coordinate system and the transformation matrix of the second coordinate system.

3. The multi-lens coplanarity correction method according to claim 2, wherein calculating the correction matrix based on the transformation matrix of the first coordinate system and the transformation matrix of the second coordinate system comprises: calculating an inverse matrix based on the transformation matrix of the first coordinate system; and calculating the correction matrix of the second coordinate system relative to the first coordinate system based on the inverse matrix and the transformation matrix of the second coordinate system.

4. The multi-lens coplanarity correction method according to claim 1, wherein the three-dimensional information is point cloud, and using the arithmetic device to calculate the surface equations based on the three-dimensional information of the three-dimensional images comprises: using the arithmetic device to respectively generate a plurality of virtual surfaces based on the point cloud of the three-dimensional images; and using the arithmetic device to calculate the surface equations corresponding to the virtual surfaces.

5. The multi-lens coplanarity correction method according to claim 1, wherein each of the coordinate systems includes a first vector, a second vector and a third vector, and using the arithmetic device to calculate the coordinate systems based on the surface equations comprises: for each of the coordinate systems, determining the first vector according to the coefficients of one of the corresponding surface equations; randomly determining the second vector; calculating the outer product of the first vector and the second vector to obtain the third vector; and calculating the outer product of the first vector and the third vector to correct the second vector.

6. The multi-lens coplanarity correction method according to claim 2, further comprises: before using the arithmetic device to calculate the surface equations based on the three-dimensional information of the three-dimensional images, using the arithmetic device to set an interested region in each of the three-dimensional images.

7. The multi-lens coplanarity correction method according to claim 6, wherein each of the transformation matrices is: where (x 0 , y 0 , z 0 ), (x 1 , y 1 , z 1 ), and (x 2 , y 2 , z 2 ) are the three-axis vectors that make up each of these coordinate systems, (P x , P y , P z ) is the specific point, and the specific point is the center point of the region of interest.

8. The multi-lens coplanarity correction method according to claim 1, wherein the reference surface is a plane, and the plane includes a plurality of positions having the same height.

9. A multi-lens coplanarity correction system comprising: an object including a reference surface with known spatial information; a three-dimensional imaging device having a plurality of lenses, wherein the lenses are used to photograph the reference surface to respectively generate a plurality of three-dimensional images; an arithmetic device electrically connected to the three-dimensional imaging device to obtain the three-dimensional images, the arithmetic device being used to execute a plurality of instructions to cause a plurality of operations, wherein the operations include: calculating a plurality of surface equations based on the three-dimensional information of the three-dimensional images; calculating a plurality of coordinate systems corresponding to the lenses based on the surface equations, wherein each of the coordinate systems has a specific point as the origin, one of the coordinate systems is the first coordinate system, and each of the coordinate systems other than the first coordinate system is a second coordinate system; calculating at least a correction matrix of the second coordinate system relative to the first coordinate system based on the coordinate systems; and an actuating device electrically connected to the arithmetic device, the actuating device adjusting one of the lenses corresponding to the second coordinate system according to the correction matrix.

10. The multi-lens coplanarity correction system according to claim 9, wherein in the operations, calculating at least the correction matrix of the second coordinate system relative to the first coordinate system based on the coordinate systems comprises: calculating a plurality of transformation matrices based on the global coordinate system and the coordinate systems, wherein each of the transformation matrices is used for the transformation between the corresponding one of the coordinate systems and the global coordinate system; and calculating the correction matrix based on the transformation matrix of the first coordinate system and the transformation matrix of the second coordinate system.

11. The multi-lens coplanarity correction system according to claim 10, wherein in the operations, calculating the correction matrix based on the transformation matrix of the first coordinate system and the transformation matrix of the second coordinate system comprises: calculating an inverse matrix based on the transformation matrix of the first coordinate system; and calculating the correction matrix of the second coordinate system relative to the first coordinate system based on the inverse matrix and the transformation matrix of the second coordinate system.

12. The multi-lens coplanarity correction system according to claim 9, wherein the three-dimensional information is point cloud, and in the operations, the arithmetic device calculates the surface equations based on the three-dimensional information of the three-dimensional images comprises: the arithmetic device respectively generating a plurality of virtual surfaces based on the point cloud of the three-dimensional images; and the arithmetic device calculating the surface equations corresponding to the virtual surfaces.

13. The multi-lens coplanarity correction system according to claim 9, wherein each of the coordinate systems includes a first vector, a second vector, and a third vector, and in the operations, the arithmetic device calculates the coordinate systems based on the surface equations comprises: for each of the coordinate systems, determining the first vector according to the coefficients of one of the corresponding surface equations; randomly determining the second vector; calculating the cross product of the first vector and the second vector to obtain the third vector; and calculating the cross product of the first vector and the third vector to correct the second vector.

14. The multi-lens coplanarity correction system according to claim 10, the operations further include: Before calculating the surface equations based on the three-dimensional information of the three-dimensional images by the arithmetic device, setting a region of interest in each of the three-dimensional images by the arithmetic device.

15. The multi-lens coplanarity correction system according to claim 14, wherein each of the transformation matrices is: where (x 0 , y 0 , z 0 ), (x 1 , y 1 , z 1 ), and (x 2 , y 2 , z 2 ) are the three-axis vectors that make up each of these coordinate systems, (P x , P y , P z ) is the specific point, and the specific point is the center point of the region of interest.

16. The multi-lens coplanarity correction system according to claim 9, wherein the reference surface is a plane, and the plane includes a plurality of positions having the same height.

Images