Checkerboard multi-camera fusion calibration method, visual fusion calibration system and multi-camera quantity detection device

Abstract

The invention provides a checkerboard multi-camera fusion calibration method, a visual fusion calibration system and a multi-camera quantity detection device, and belongs to the technical field of computer visual fusion calibration. The checkerboard multi-camera fusion calibration method comprises the steps of composite calibration board manufacturing, product pasting board manufacturing, picture collection, picture corresponding two-dimensional code coordinate calculation, angular point extraction, camera internal and external parameter calculation and fusion calibration. A world coordinate system automatic calibration technology based on a two-dimensional code mismatching correction mechanism is adopted, a sub-pixel-level angular point fusion positioning algorithm is combined, a camera is accurately calibrated, multiple pictures are fused, the method can be applied to an image quantity detection technology in batches at low cost, and popularization and application in the fields of semiconductors, PCBs, 3C and the like are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision fusion calibration, and in particular relates to a checkerboard multi-camera fusion calibration method, a vision fusion calibration system and a multi-camera quantity detection device. Background Art

[0002] Traditional checkerboard calibration methods in existing technologies have the following drawbacks: the world coordinate system origin must be manually specified, the calibration plate must be strictly aligned with the measured surface, and corner point matching is easily affected by rotational symmetry. Existing high-precision industrial inspection systems often use multiple cameras to capture images of the same product and then stitch them together. However, this stitching process is currently inefficient and suffers from large errors. Therefore, a new, less susceptible to interference fusion calibration technology is needed. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a checkerboard multi-camera fusion calibration method, a visual fusion calibration system and a multi-camera quantity detection device, which can solve the above problems.

[0004] Design principle: This invention introduces coded QR codes, and realizes fusion positioning of multiple cameras or multiple images through the mismatch correction mechanism of the QR code and sub-pixel corner points. Each camera calculates the internal and external parameters to realize the conversion from the camera to the calibration plate coordinate system, completing the fusion calibration of multiple cameras.

[0005] A QR code-assisted checkerboard multi-camera fusion calibration method comprises the following steps: a. preparing a composite calibration plate, wherein each checkerboard on the calibration plate is of uniform size and evenly distributed, and a coded QR code is embedded at a preset position of the checkerboard pattern; b. product mounting, wherein the composite calibration plate is placed flat on the plane where the product to be measured is located; c. image acquisition, wherein multiple images of the product to be measured are acquired at different angles by using multiple cameras; d. calculating the coordinates of the QR code corresponding to the image, extracting the pixel coordinates of the QR code vertices from the image through an image processing algorithm, and decoding the image to obtain the world coordinates; e. corner point extraction, wherein the sub-pixel coordinates of the corner points are obtained, and the true value of the calibration plate corresponding to each corner point is deduced by combining the coordinates of the QR code; f. calculating the intrinsic and extrinsic parameters of the camera, and solving the camera intrinsic parameter matrix and extrinsic parameter matrix; g. fusion calibration, wherein the mapping relationship between the pixel coordinates and the calibration plate coordinates is calculated based on the intrinsic and extrinsic parameters of the camera, thereby realizing multi-camera fusion calibration.

[0006] Furthermore, the QR code carries the corresponding coordinate values ​​in the coordinate system of the chessboard, and the QR code is set at four adjacent fixed points of the chessboard.

[0007] Furthermore, the method for calculating the coordinates of the QR code corresponding to the image is: extracting the pixel coordinates corresponding to the four vertices of the QR code through image processing and visual algorithms, and solving the code value corresponding to the QR code.

[0008] Furthermore, the sub-pixel coordinates of the corner points are obtained by calculating the sub-pixel difference algorithm.

[0009] Furthermore, the method also includes: h. splicing optimization, applying a distance weighted algorithm to eliminate splicing errors.

[0010] The present invention also provides a visual fusion calibration system for implementing the aforementioned method, the system comprising: a composite calibration plate module, which integrates a decodable QR code and a high-precision checkerboard pattern; an image acquisition module, which uses multiple cameras with different fields of view or different perspectives to capture images of the calibration plate; a data processing module, which performs QR code decoding, corner point extraction, and calculation of internal and external camera parameters to achieve multi-camera calibration and image fusion; and a calibration verification module, which evaluates calibration accuracy through reprojection error.

[0011] The present invention also provides a multi-camera quantity detection device, including a longitudinal transfer line, a carrier and a quantity detection module; multiple longitudinal transfer lines are arranged side by side on a frame to form multi-channel loading; multiple carriers are arranged on the corresponding active ends of the longitudinal transfer line, and the carrier includes a hollow fixture plate group with an avoidance recess and a flexible contoured pressure plate assembly for supporting and leveling the product to be tested; the quantity detection module uses multiple cameras to capture surface images of the product to be tested on the carrier, and the images captured by the multiple cameras are fused and calibrated based on the aforementioned checkerboard multi-camera fusion calibration method to calculate and output the size of the entire surface of the product.

[0012] Compared with the existing technology, the beneficial effect of the present invention is that: this application adopts the world coordinate system automatic calibration technology based on the QR code mismatch correction mechanism, combined with the sub-pixel corner fusion positioning algorithm, accurately calibrates the camera and fuses multiple pictures, and can be applied in batches at low cost to image quantity detection technology, which is convenient for promotion and application in semiconductors, PCB, 3C and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is the calculation logic flow chart of the checkerboard multi-camera fusion calibration method of this application;

[0014] Figure 2 is a schematic diagram of a composite calibration plate with a checkerboard pattern;

[0015] Figure 3 This is a schematic diagram of the QR code extraction results;

[0016] Figure 4 and Figure 5 Schematic diagram for corner point extraction;

[0017] Figure 6 Schematic diagram of a multi-camera measurement detection device based on fusion calibration;

[0018] Figure 7 This is a schematic diagram of the quantity detection module;

[0019] Figure 8 Schematic diagram of the retest unit.

[0020] In the figure,

[0021] 1. Longitudinal transfer line;

[0022] 2. Carrier;

[0023] 3. Detection module; 31. Detection frame; 32. Detection installation beam plate; 33. Camera lens assembly; 34. Coaxial optical unit; 35. Detection adapter plate;

[0024] 4. Re-inspection unit; 41. Re-inspection collector with wide field of view; 42. Re-inspection of installation parts. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0026] It should be understood that the terms "system," "device," "mechanism," "component," "module," and / or "module" used in this specification are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0027] Checkerboard multi-camera fusion calibration method

[0028] A QR code-assisted checkerboard multi-camera fusion calibration method includes the following steps (not necessarily in the correct order). The general process can be found in Figure 1 .

[0029] a. Make a composite calibration plate. Each checkerboard square on the calibration plate is of the same size and evenly distributed. Embed a coded QR code at the preset position of the checkerboard pattern.

[0030] The QR code stores its absolute coordinate information in the checkerboard coordinate system. The QR code contains the coordinate values ​​corresponding to the checkerboard coordinate system. The QR code is placed at four adjacent fixed points on the checkerboard. In other words, the decoded QR code contains the coordinate values ​​corresponding to the QR code in the checkerboard coordinate system.

[0031] In a specific example, the QR code adopts the QRCode encoding format, and its physical size maintains an integer multiple ratio relationship with the checkerboard unit.

[0032] b. Place the composite calibration plate flat on the surface of the product to be measured.

[0033] c. Image acquisition: Use multiple cameras to capture multiple images of the product under test from different angles.

[0034] d. Calculate the coordinates of the QR code corresponding to the image, extract the pixel coordinates of the QR code vertices through the image processing algorithm, and decode them to obtain the world coordinates.

[0035] The method for calculating the coordinates of the QR code corresponding to the image is: extract the pixel coordinates corresponding to the four vertices of the QR code through image processing and visual algorithms, and calculate the code value corresponding to the QR code.

[0036] e. Corner point extraction: obtain the sub-pixel coordinates of the corner points, combine them with the QR code coordinates, and calculate the true value of the calibration plate corresponding to each corner point.

[0037] The sub-pixel coordinates of the corner points are obtained by calculating the sub-pixel difference algorithm. Alternatively, algorithms such as Harris and Shi-Tomasi can also be used to extract sub-pixel corner points.

[0038] A sub-pixel corner detection algorithm is used to locate the coordinates of the checkerboard corners. The 3D-2D correspondence is established by combining the coordinates decoded from the QR code.

[0039] Specifically, a local coordinate system is first established based on the coordinates of the four QR code vertices. Then, the world coordinates of each corner point are calculated using the checkerboard spacing parameters.

[0040] f. Calculate the internal and external parameters of the camera and solve the camera intrinsic parameter matrix K and extrinsic parameter matrix [R|t] through the PnP algorithm.

[0041] Among them, the camera intrinsic parameter matrix K is:

[0042] Where f x 、f y is the equivalent focal length of the camera in the X-axis and Y-axis directions; u0 and v0 are the image principal point offsets, that is, the coordinates of the image center point in the pixel coordinate system (u0, v0).

[0043] Note: The equivalent focal length unit is pixel. Usually it is calculated by dividing the physical focal length f by the pixel size, such as f x =f / d x , d x The width of a single sensor pixel. Its function is to control the scaling of 3D points on the imaging plane, affecting depth estimation and image distortion correction.

[0044] Among them, the camera extrinsic parameter matrix [R|t] is:

[0045] Where R is a 3×3 rotation matrix that describes the rotation transformation from the world coordinate system to the camera coordinate system. Its function is to eliminate the influence of the camera posture (such as pitch, yaw, and roll) on the projection.

[0046] t is a 3×1 translation vector that represents the position of the world coordinate system origin in the camera coordinate system. Its function is to compensate for the relative displacement between the camera and the target object.

[0047] 1 is the homogeneous coordinate expansion term, which is used to integrate the translation transformation into the matrix multiplication.

[0048] Specifically, the camera's internal and external parameter matrix is ​​solved using the following algorithm formula.

[0049] Where Z c is the depth value of the three-dimensional point in the camera coordinate system, which serves as a normalization factor; u and v represent the horizontal pixel coordinates and vertical pixel coordinates of the target point on the image plane; X W 、Y W , Z W Represent the coordinates of the target point in the three-dimensional world coordinate system.

[0050] g. Fusion calibration: Based on the internal and external parameters of the camera, the mapping relationship between pixel coordinates and calibration plate coordinates is calculated to achieve multi-camera fusion calibration.

[0051] Based on the camera imaging principle, the mapping relationship between pixel coordinates and calibration plate coordinates is calculated. By calculating the internal and external parameters of each camera, each camera can achieve the conversion to the calibration plate coordinate system, thus completing the camera fusion calibration. From an algorithmic perspective, the coordinate system conversion model is completed.

[0052] Furthermore, the method also includes: h. splicing optimization, applying a distance weighted algorithm to eliminate splicing errors.

[0053] Vision fusion calibration system

[0054] A visual fusion calibration system for implementing the aforementioned method includes the following modules.

[0055] Composite calibration plate module, see Figure 2 , integrating a decodable QR code with a high-precision checkerboard pattern. Using a 30mm x 30mm checkerboard as an example: Creating a calibration plate: Printing accuracy ≤ 0.001mm. Multiple cameras capture images of the calibration plate with varying fields of view or angles.

[0056] The data processing module performs QR code decoding, corner extraction, and camera internal and external parameter calculations, enabling multi-camera calibration and image fusion. Furthermore, the data processing module integrates the solvePnP function from the OpenCV library for parameter optimization.

[0057] The calibration verification module evaluates the calibration accuracy through the reprojection error.

[0058] Specifically, the final result is verified, optimized or determined by minimizing the reprojection error.

[0059] Multi-camera measurement detection device based on fusion calibration

[0060] For a specific application example, a multi-camera measurement detection device based on fusion calibration is shown in Figure 6-Figure 8 The multi-camera quantity detection device includes a longitudinal transfer line 1, a carrier 2 and a quantity detection module 3.

[0061] Layout relationship: multiple longitudinal transfer lines 1 are arranged side by side on the frame to form multi-channel loading; multiple carriers 2 are arranged on the corresponding active ends of the longitudinal transfer lines 1, and the carriers 2 include a hollow fixture plate group with an avoidance recess and a flexible contoured pressure plate assembly for supporting and leveling the product to be tested; the measurement detection module 3 uses multiple cameras to capture the surface image of the product to be tested on the carrier 2, and fuses and calibrates the images captured by multiple cameras to calculate and output the size of the entire surface of the product.

[0062] See also Figure 5 The measurement detection module 3 includes a detection frame 31, a detection installation beam plate 32, a camera lens assembly 33 and a coaxial light unit 34; the detection installation beam plate 32 is horizontally installed on the top of the detection frame 31, and multiple groups of camera lens assemblies 33 are installed on the front and rear panels of the detection installation beam plate 32; the coaxial light unit 34 is set directly below the camera lens assembly 33 through the detection adapter plate 35; the acquisition field of view of multiple camera lens assemblies 33 covers the entire panel surface of the test product.

[0063] In the illustrated example, three camera lens assemblies 33 are used. Depending on the accuracy requirements, the cameras may be 2-megapixel, 6-megapixel, or 12-megapixel global shutter cameras.

[0064] For further information, see Figure 1 and Figure 6 The multi-camera quantity detection device also includes a re-inspection unit 4, which uses a large-field camera or an array laser to regularly verify and calibrate the quantity detection module 3.

[0065] Specifically, the re-inspection unit 4 is integrated and installed in the quantity detection module 3. The re-inspection unit 4 includes a large-field-of-view re-inspection collector 41 and a re-inspection mounting member 42. The large-field-of-view re-inspection collector 41 is installed vertically downward on the detection mounting crossbeam plate 32 through the re-inspection mounting member 42, and is located in front of or behind the camera lens assembly 33. The visual field of the large-field-of-view re-inspection collector 41 covers the entire board surface of the product to be tested.

[0066] The image processor compares and verifies the image of the re-inspection unit 4 and the image fused and calibrated by the quantity detection module 3 to determine whether the quantity detection module 3 is accurate and shuts down for maintenance when necessary.

[0067] Currently, this solution is being used on-site for mass inspection of semiconductor wafers (with and without images), mid-frame panels for mobile phones and pads, and large PCBs. The coordinate error of the fused images can be reduced to nanometers.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A QR code-assisted checkerboard multi-camera fusion calibration method, characterized by ,methods include: a. Make a composite calibration plate, where each checkerboard grid is of uniform size and evenly distributed, and embed a QR code at a preset position on the checkerboard pattern; b. Product board placement: Place the composite calibration board flat on the surface of the product to be measured; c. Image acquisition: Use multiple cameras to capture multiple images of the product under test from different angles; d. Calculate the coordinates of the QR code corresponding to the image, extract the coordinates of the QR code vertices through the image processing algorithm, and decode them to obtain the world coordinates; e. Corner point extraction: obtain the sub-pixel coordinates of the corner points, combine them with the QR code coordinates, and calculate the true value of the calibration plate corresponding to each corner point; f. Calculate the internal and external parameters of the camera and solve the camera internal parameter matrix and external parameter matrix; g. Fusion calibration: Based on the internal and external parameters of the camera, the mapping relationship between pixel coordinates and calibration plate coordinates is calculated to achieve multi-camera fusion calibration.

2. The checkerboard multi-camera fusion calibration method according to claim 1, characterized in that: The two-dimensional code carries the coordinate values ​​corresponding to the coordinate system of the chessboard, and the two-dimensional code is set at four adjacent fixed points of the chessboard.

3. The checkerboard multi-camera fusion calibration method according to claim 1, characterized in that: The method for calculating the coordinates of the QR code corresponding to the image is: extract the pixel coordinates corresponding to the four vertices of the QR code through image processing and visual algorithms, and calculate the code value corresponding to the QR code.

4. The checkerboard multi-camera fusion calibration method according to claim 1, characterized in that: The sub-pixel coordinates of the corner points are obtained by calculating the sub-pixel difference algorithm.

5. The checkerboard multi-camera fusion calibration method according to claim 1, characterized in that: The camera intrinsic parameter matrix K is: Where f x 、f y is the equivalent focal length of the camera in the X-axis and Y-axis directions; u0 and v0 are the image principal point offsets, that is, the coordinates of the image center point in the pixel coordinate system (u0, v0).

6. The checkerboard multi-camera fusion calibration method according to claim 1, characterized in that: The camera extrinsic parameter matrix [R|t] is: Where R is a 3×3 rotation matrix, describing the rotation transformation from the world coordinate system to the camera coordinate system; t is a 3×1 translation vector, which represents the position of the world coordinate system origin in the camera coordinate system; 1 is the homogeneous coordinate expansion term, which is used to integrate the translation transformation into the matrix multiplication.

7. The checkerboard multi-camera fusion calibration method according to claim 1, characterized in that: The method also includes: h. Stitching optimization, applying distance weighted algorithm to eliminate stitching errors.

8. A visual fusion calibration system implementing the method according to any one of claims 1 to 7, comprising: Composite calibration plate module: integrates decodable QR code and high-precision checkerboard pattern; Image acquisition module: multiple cameras with different fields of view or different perspectives capture the calibration plate image; Data processing module: performs QR code decoding, corner point extraction, and camera internal and external parameter calculation to achieve multi-camera calibration and image fusion; Calibration Verification Module: evaluates calibration accuracy through reprojection error.

9. A multi-camera measurement detection device, characterized in that: The multi-camera measurement detection device comprises a longitudinal transfer line (1), a carrier (2) and a measurement detection module (3); a plurality of longitudinal transfer lines (1) are arranged side by side on a frame to form a multi-channel loading; a plurality of the carriers (2) are arranged on the corresponding movable ends of the longitudinal transfer line (1), and the carrier (2) comprises a hollow fixture plate group with an avoidance recess and a flexible contoured pressure plate assembly for supporting and leveling the product to be tested; the measurement detection module (3) adopts a plurality of cameras for collecting surface images of the product to be tested on the carrier (2), and performs fusion calibration on the images collected by the plurality of cameras based on the checkerboard multi-camera fusion calibration method described in any one of claims 1 to 7 to calculate and output the size of the entire surface of the product.

Images