Calibration method, device, equipment and medium for automatic optical inspection equipment

Abstract

Embodiments of the present application disclose a method, device, equipment and medium for calibrating an automatic optical inspection (AOI) device. In the case that the 3D AOI device is provided with multiple side cameras, multiple fixed calibration boards are carried by a calibration support in the calibration process, and the projection and image acquisition of each calibration board are sequentially completed in the detection area. The calibration boards are in different pose states, and the central camera and the multiple side cameras respectively complete image acquisition in each pose state, and calibration is performed based on the respective corresponding image acquisition results, so that various conversion matrices required for generating surface morphology parameters of a measured object based on optical detection in a subsequent detection process can be obtained, and the inefficiency of manual adjustment of the orientation of the calibration board in the calibration process is avoided, and the calibration efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of industrial testing technology, and in particular to calibration methods, apparatus, equipment and media for automated optical testing equipment. Background Technology

[0002] In the field of optical inspection, the first type of 3D AOI (Automated Optical Inspection) equipment generally includes a central camera and four side projectors, with each side projector and the central camera forming a subsystem. The second type of 3D AOI equipment includes a central camera, a central projector, and four side cameras, with each side camera and the central projector forming a subsystem. The height map generated by each subsystem needs to be converted to the pixel coordinate system of the central camera to complete the inspection. Regardless of the first or second approach, accurate inspection requires a calibration method specific to the hardware solution. The entire 3D AOI equipment must first be optically calibrated based on a calibration board to accurately detect the shape of the target object.

[0003] The inventors discovered during the calibration of 3D AOI equipment that, compared to the first method which only required keeping the calibration plate stationary, the second method required multiple manual adjustments to the orientation of the calibration plate to ensure it was at a strictly controlled angle for calibration to be completed. This high degree of precision required for manual adjustment made the calibration process more complex. Summary of the Invention

[0004] This invention provides a calibration method, apparatus, device, and medium for automated optical inspection equipment, to solve the technical problem that the calibration process of existing 3D AOI equipment, which includes a central camera, a central projection, and four side cameras, is relatively complex.

[0005] In a first aspect, embodiments of this application provide a calibration method for an automated optical inspection device. The automated optical inspection device includes a central camera and a central projector sharing a single lens, as well as multiple side cameras arranged around the central camera. The calibration method for the automated optical inspection device includes:

[0006] Multiple sinusoidal stripes are encoded based on a preset frequency and a preset number of steps. The multiple sinusoidal stripes are projected sequentially onto the calibration plate in each pose state through central projection. Multiple calibration points are drawn on the surface of the calibration plate. The calibration plate in each pose state is fixed to the calibration bracket and faces the central camera or different side cameras respectively. The calibration bracket is used to carry the calibration plates into the detection area sequentially during the calibration process.

[0007] In each pose state, the first image of the calibration plate under the illumination of the light source is acquired by the central camera and the side camera, and the second image of the calibration plate under the sinusoidal fringe projection is acquired by the side camera each time.

[0008] Based on the associated images of the side camera, the center camera, and the center projection in the image set, the transformation matrices from the pixel coordinate systems of the side camera, the center camera, and the center projection to the corresponding target coordinate systems are determined respectively. The transformation matrices are used as the calibration results of the automatic optical inspection equipment. The image set includes the first image, the second image, and the projection content of the center projection.

[0009] As described above, when the 3D AOI equipment has multiple side cameras, during the calibration process, multiple calibration plates are fixed and supported by a calibration bracket. These plates sequentially enter the detection area to complete the projection and image acquisition of each calibration plate. This allows the calibration plates to quickly and accurately be in different poses. In each pose, the central camera and multiple side cameras are controlled to complete image acquisition. Based on the corresponding image acquisition results, calibration is performed, which yields various transformation matrices required for generating the surface morphology parameters of the measured object based on optical detection in the subsequent detection process. This avoids the inefficiency of manually adjusting the orientation of the calibration plates during the calibration process and improves calibration efficiency.

[0010] Specifically, the homography matrix between the pixel coordinate system and the world coordinate system of each side camera is determined based on the first image, and the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each side camera is determined based on the homography matrix.

[0011] By using the second image captured by each side camera, the phase of the center point coordinates of each calibration point in the first image is calculated, and the coordinates of each center point in each pose state are transformed to the pixel coordinate system corresponding to the center projection to obtain the corresponding projection point coordinates based on the phase.

[0012] Based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system, the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera is determined.

[0013] Based on the coordinates of the projection points and the pixel coordinates of each calibration point in the first image captured by the central camera in the corresponding pixel coordinate system, the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system is determined.

[0014] As described above, by moving the calibration bracket carrying the calibration plate, the calibration plates at different angles enter the detection area in sequence to obtain images of various pose states required for calibration, and the calibration process is completed quickly based on the images.

[0015] Specifically, the homography matrix between the pixel coordinate system and the world coordinate system of each side camera is determined based on the first image, and the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each side camera is determined based on the homography matrix, including:

[0016] By calibrating the first image corresponding to each side camera using Zhang Zhengyou, the intrinsic and extrinsic parameters of each side camera are obtained.

[0017] Based on the product of the intrinsic and extrinsic parameters corresponding to each side camera, the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system for each side camera is obtained.

[0018] As described above, the calibration of the side camera can be quickly completed by solving for the intrinsic and extrinsic parameters using the homography matrix of the side camera.

[0019] Specifically, based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system, a second transformation matrix is ​​determined from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera, including:

[0020] Based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system, the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera is determined by the least squares method.

[0021] As mentioned above, the calibration of the central camera can be quickly confirmed using the least squares method.

[0022] Specifically, by using the second image captured by each side camera, the phase of the circle center point corresponding to each calibration point in the first image is calculated, including:

[0023] By using the second image corresponding to the sinusoidal stripes at each frequency acquired by each side camera, the wrapping phase of each side camera at each frequency is confirmed.

[0024] The wrapped phase corresponding to the target frequency is unfolded to obtain the unfolded phase of the sinusoidal stripes within one period, and the target frequency is the frequency with the largest value.

[0025] The phase identified by the image corresponding to the sinusoidal fringes at each frequency, as described above, allows for rapid confirmation of the expanded phase.

[0026] Specifically, based on the projection point coordinates and the pixel coordinates of each calibration point in the first image captured by the central camera in the corresponding pixel coordinate system, the third transformation matrix from the corresponding pixel coordinate system to the world coordinate system of the central projection is determined, including:

[0027] The third transformation matrix from the corresponding pixel coordinate system to the world coordinate system was confirmed by using a telecentric lens imaging model.

[0028] As mentioned above, the central projection can be accurately calibrated using the telecentric lens imaging model.

[0029] Secondly, embodiments of this application provide a calibration device for an automated optical inspection device. The automated optical inspection device includes a central camera and a central projector sharing a single lens, as well as multiple side cameras arranged around the central camera. The calibration device for the automated optical inspection device includes:

[0030] The stripe coding unit is used to encode multiple sinusoidal stripes based on a preset frequency and a preset number of steps. The multiple sinusoidal stripes are projected sequentially onto the calibration plate in each pose state through central projection. Multiple calibration points are drawn on the surface of the calibration plate. The calibration plate in each pose state is fixed to the calibration bracket and faces the central camera or different side cameras respectively. The calibration bracket is used to carry the calibration plates into the detection area sequentially during the calibration process.

[0031] The image acquisition unit is used to acquire a first image of the calibration plate under light illumination through the central camera and the side camera in each pose state, and to acquire a second image of the calibration plate under sinusoidal fringe projection each time through the side camera.

[0032] The matrix verification unit is used to verify the transformation matrix from the pixel coordinate system of the side camera, the center camera, and the center projection to the corresponding target coordinate system based on the associated images of the side camera, the center camera, and the center projection in the image set, and to use the transformation matrix as the calibration result of the automatic optical inspection device; the image set includes the first image, the second image, and the projection content of the center projection.

[0033] The matrix confirmation unit includes:

[0034] The first transformation confirmation module is used to confirm the homography matrix between the pixel coordinate system and the world coordinate system of each side camera based on the first image, and to confirm the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each side camera based on the homography matrix.

[0035] The coordinate confirmation module is used to calculate the phase of the center point coordinates of each calibration point in the first image through the second image acquired by each side camera, and to transform the center point coordinates of each circle in each pose state to the pixel coordinate system corresponding to the center projection to obtain the corresponding projection point coordinates based on the phase.

[0036] The second transformation confirmation module is used to confirm the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera, based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system.

[0037] The third transformation confirmation module is used to confirm the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system based on the coordinates of the projection point and the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system.

[0038] The first conversion confirmation module includes:

[0039] The parameter confirmation submodule is used to obtain the intrinsic and extrinsic parameters of each side camera by calibrating the first image corresponding to each side camera using Zhang Zhengyou's calibration.

[0040] The product confirmation submodule is used to obtain the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system for each side camera based on the product of the intrinsic and extrinsic parameters corresponding to each side camera.

[0041] The second conversion confirmation module includes:

[0042] The least squares processing submodule is used to determine the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera by using the least squares method, based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system.

[0043] The coordinate confirmation module includes:

[0044] The first phase confirmation submodule is used to confirm the wrapping phase of each side camera at each frequency by using the second image corresponding to the sinusoidal stripe acquired by each side camera at each frequency.

[0045] The second phase confirmation submodule is used to expand the wrapping phase corresponding to the target frequency to obtain the expanded phase of the sine stripes within one period, where the target frequency is the frequency with the largest value.

[0046] The third conversion confirmation module includes:

[0047] The imaging model application submodule is used to confirm the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system through the telecentric lens imaging model.

[0048] Thirdly, embodiments of this application also provide an electronic device, which includes:

[0049] One or more processors;

[0050] Memory, used to store one or more computer programs;

[0051] When one or more computer programs are executed by one or more processors, electronic devices enable calibration methods for automatic optical inspection devices as described in the first aspect.

[0052] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the calibration method of the automatic optical inspection device as described in the first aspect. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating a calibration method for an automated optical inspection device provided in an embodiment of this application.

[0054] Figure 2 This is a schematic diagram of the structure of a calibration device for an automatic optical inspection equipment provided in an embodiment of this application.

[0055] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention and not the entire structure.

[0057] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0058] The embodiments of the present invention will be described in detail below.

[0059] In the field of automated optical inspection, 3D AOI equipment mainly includes an architecture with one central camera and four side projectors, each side projector forming a subsystem with the central camera (hereinafter referred to as the first architecture); and an architecture including a central camera, a central projector, and four side cameras (hereinafter referred to as the second architecture). For 3D AOI equipment implemented with the first architecture, calibration can be completed simply by keeping the calibration plate stationary. However, calibrating 3D AOI equipment with the second architecture requires multiple manual adjustments to the orientation of the calibration plate to ensure it is at a relatively precise angle. This high degree of precision required for manual adjustment of the calibration plate makes the calibration process more complex.

[0060] To address the aforementioned technical issues, this application proposes a calibration method for an automated optical inspection (AOI) device. When the 3D AOI device has multiple side cameras, multiple calibration plates are fixed and supported by a calibration bracket during the calibration process. These plates sequentially enter the inspection area to complete projection and image acquisition for each calibration plate. By positioning the calibration plates in different poses, and controlling the central camera and multiple side cameras to acquire images in each pose, calibration is performed based on the corresponding image acquisition results. This yields various transformation matrices required for generating surface morphology parameters of the measured object based on optical detection during subsequent inspection processes. This avoids the inefficiency of manually adjusting the orientation of the calibration plates during calibration, thus improving calibration efficiency.

[0061] Figure 1 This is a flowchart illustrating a calibration method for an automated optical inspection device provided in an embodiment of this application. The automated optical inspection device in this real-time example includes a central camera and a central projector sharing a single lens, as well as multiple side cameras arranged around the central camera. The central projector is a device for projecting images onto the center of the area to be measured, and can be implemented using a projector. The central camera is a device for acquiring images of the center of the area to be measured. The central projector and the central camera using the same lens can be achieved during hardware assembly; for example, a 45° semi-transparent mirror lens can be used, with the central projector and the central camera sharing a telecentric lens. Additionally, the side cameras are used to acquire images from the sides of the central camera. The number of side cameras can be set according to actual needs; this embodiment uses four side cameras as an example, and the side cameras can employ wide-angle lenses.

[0062] like Figure 1 As shown, the calibration method for this automatic optical inspection equipment includes:

[0063] Step S110: Encode multiple sinusoidal stripes based on a preset frequency and a preset number of steps. Project the multiple sinusoidal stripes sequentially onto the calibration plate in each pose state through central projection. Multiple calibration points are drawn on the surface of the calibration plate. The calibration plate in each pose state is fixed to the calibration bracket and faces the central camera or different side cameras respectively. The calibration bracket is used to carry the calibration plates into the detection area sequentially during the calibration process.

[0064] To achieve the desired calibration in this embodiment, sinusoidal fringes need to be encoded. Sinusoidal fringes are fringes in the shape of a sinusoid. Encoding sinusoidal fringes requires encoding sinusoidal fringes with multiple steps at multiple frequencies. The number of steps in the sinusoidal fringes refers to the distance between two adjacent points in phase in the sinusoidal fringes diagram. The number of steps in the sinusoidal fringes determines the measurement accuracy and resolution. For example, this embodiment uses a five-frequency, four-step sinusoidal fringes as an example. A five-frequency, four-step sinusoidal fringes has five frequencies, each with four steps. This embodiment can encode the sinusoidal fringes using the following formula:

[0065]

[0066] Where A is the background intensity, B is the amplitude, and Φ is the phase. f For frequency, k The number of steps (in this embodiment, 1 <= k <=4, k ∈Z), x, y () are the coordinates in the pixel coordinate system of the center projection.

[0067] In this embodiment, the sinusoidal fringes include horizontally expanding stripes and vertically expanding stripes; therefore, both directions of stripes need to be encoded during encoding. In one embodiment, it is assumed that the resolution of the line bundle projected by the central projection is 1920. The frequencies of the five horizontally unfolded fringes are 240 Hz, 235 Hz, 192 Hz, 45 Hz, and 44 Hz, respectively. The highest frequency used for the horizontal fringes is 240 Hz because 1920 / 240 = 8, meaning one cycle of the horizontal fringes includes 8 pixels (higher frequencies would reduce fringes contrast). 192 Hz means one cycle includes 10 pixels; 235 Hz is an empirical value. The difference between 45 Hz and 44 Hz is 1, also an empirical value; a frequency difference of 1 allows for the determination of a complete unfolding phase, which is used to guide the high-frequency unfolding. The frequencies of the five vertically unfolded fringes are 135 Hz, 130 Hz, 108 Hz, 25 Hz, and 24 Hz, with a similar principle to the horizontal fringes. Combining these frequencies provides better measurement accuracy and stability. It is understood that the frequencies of the five-frequency, four-step sinusoidal fringes can also be set according to actual needs; this embodiment does not impose specific limitations. Sine fringes can be encoded and projected at various stages of automated optical inspection.

[0068] After encoding the five-frequency, four-step sinusoidal stripes, the stripes need to be projected sequentially onto the surface of a calibration board or the object under test using a central projection. The calibration board is a flat plate structure with calibration points drawn according to strict layout requirements. The object under test refers to the object to be reconstructed in 3D, such as a circuit board or components. In this embodiment, sinusoidal stripes of four steps at one frequency are sequentially projected onto the surface of the object under test until sinusoidal stripes of all steps at all frequencies have been projected. By projecting the five-frequency, four-step sinusoidal stripes onto the object under test, the true shape of the object can be reflected based on the deflection of the sinusoidal stripes on the object. The calibration board is the object of photography during the calibration phase, and the object under test is the object of photography during the actual testing phase after calibration. This embodiment mainly describes the process of projection, image acquisition, and image processing using the calibration board as the object of photography during the calibration phase.

[0069] In this embodiment, considering the need to change the orientation of the calibration plate multiple times for calibration, multiple calibration plates are used, with the same distribution of calibration points on each plate. Multiple calibration plates are fixed to a calibration support, and each plate corresponds to a different pose state. During calibration, the calibration support can be placed on a conveyor belt. Controlling the 3D AOI equipment to start calibration drives the conveyor belt to move the calibration support towards the detection area. Multiple calibration plates follow the position changes of the calibration support and enter the detection area sequentially. The operator only needs to manually confirm the start of calibration to automatically complete the calibration process without manual adjustment of the calibration plates, effectively improving calibration speed and allowing for better control of the calibration process.

[0070] Step S120: In each pose state, the first image of the calibration plate under the illumination of the light source is acquired by the central camera and the side camera, and the second image of the calibration plate under the sinusoidal fringe projection is acquired by the side camera each time.

[0071] Both the first and second images are images captured from the calibration board. The main difference between them lies in the projection of the central projection onto the calibration board. The first image has a light source but no sinusoidal fringes; the light source can be white light or other colored light. The second image has sinusoidal fringes. Considering that the overall processing approach for each side camera and different sinusoidal fringes is the same, the images of the calibration board captured by all side cameras when different sinusoidal fringes are projected are collectively referred to as the second image. In the specific calibration process, the order of shooting, based on the changes in pose and the content of the central projection, is not limited; for example, the first image can be captured first, or the second image can be captured first. For the first image, each side camera needs to acquire 5 images for each of the 5 pose states. For the second image, taking the five-frequency four-step method as an example, there are 20 sinusoidal fringes in the five-frequency four-step method, which correspond to the horizontal and vertical directions. Each side camera needs to acquire 40 second images for each pose state. A total of 200 second images need to be acquired for the five pose states. If there are a total of 4 side cameras, a total of 800 second images will be acquired. However, the 800 second images are divided into 4 groups based on the side cameras to obtain the calibration results of the 4 side cameras.

[0072] Step S130: Based on the associated images of the side camera, the center camera, and the center projection in the image set, determine the transformation matrix from the pixel coordinate system of the side camera, the center camera, and the center projection to the corresponding target coordinate system, and use the transformation matrix as the calibration result of the automatic optical inspection device; the image set includes the first image, the second image, and the projection content of the center projection.

[0073] In 3D AOI equipment, each camera and the central projection can only perform planar imaging. To confirm the solid shape from planar imaging and complete the transformation from planar position to solid position, the transformation matrix obtained through calibration is used. Each camera and the central projection have a directly related coordinate system, namely the corresponding target coordinate system. The target coordinate system is mainly the world coordinate system or the pixel coordinate system of other hardware. For different cameras and central projections, calibration has its own corresponding associated images. For different cameras or central projections, the associated images can refer to the pattern on the calibration board, the sinusoidal fringe image projected by the central projection, and / or the corresponding acquired images in steps S110 and S120. Through the regular changes in the content of these images and the imaging information of these contents in the acquired images (such as the position of the origin in the calibration board), the transformation matrix from the pixel coordinate system of the side camera, the central camera, and the central projection to the corresponding target coordinate system can be confirmed through various calibration methods such as Zhang Zhengyou calibration.

[0074] Corresponding to the specific hardware type in the automated optical inspection equipment, the calibration results in step S130 include three transformation matrices: a first transformation matrix from the corresponding pixel coordinate system of each side camera to the world coordinate system, a second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera, and a third transformation matrix from the corresponding pixel coordinate system of the central projection to the world coordinate system. The specific confirmation process of the first, second, and third transformation matrices can be referred to the descriptions in steps S131-S134. Considering that the images acquired by the side cameras and the central camera are both based on their corresponding pixel coordinate systems, the pixel coordinate systems used to describe the information content in the images acquired by the side cameras and the central camera respectively are also called image coordinate systems.

[0075] Step S131: Determine the homography matrix between the pixel coordinate system and the world coordinate system of each side camera based on the first image, and determine the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each side camera based on the homography matrix.

[0076] In the specific implementation, the purpose of calibration is to obtain multiple transformation matrices. Each transformation matrix specifically describes the transformation relationship between coordinates in two different coordinate systems. The first transformation matrix describes the transformation relationship of each side camera from its corresponding pixel coordinate system to the world coordinate system. Under the first image of the side camera, the coordinates of the center point of the calibration point are extracted to obtain the coordinates of the center point in the pixel coordinate system of the side camera. Simultaneously, using the plane where the calibration board is located as the xy plane of the world coordinate system, and the center of the first calibration point as the origin (0, 0, 0), the world coordinates of each calibration point can be recorded according to the layout relationship of each calibration point when drawing the calibration points. During the specific transformation process, through the operation of the first transformation matrix and the world coordinate system, the first transformation matrix of each side camera from its corresponding pixel coordinate system to the world coordinate system can be obtained. Specifically, the intrinsic and extrinsic parameters of each side camera can be obtained by Zhang Zhengyou calibration for the first image corresponding to each side camera; based on the product of the intrinsic and extrinsic parameters corresponding to each side camera, the first transformation matrix of each side camera from the corresponding pixel coordinate system to the world coordinate system can be obtained. This method of confirming the intrinsic and extrinsic parameters can be Zhang Zhengyou calibration or other camera calibration methods based on calibration objects.

[0077] Step S132: Using the second image acquired by each side camera, calculate the phase of the center point coordinates of each calibration point in the first image, and transform the center point coordinates of each pose state to the pixel coordinate system corresponding to the center projection to obtain the corresponding projection point coordinates based on the phase.

[0078] In the specific implementation process, the phase of the center point coordinates of each calibration point in the first image is calculated using the second image captured by each side camera. This includes: confirming the wrapping phase of each side camera at each frequency using the second image corresponding to the sinusoidal stripes captured by each side camera at each frequency; and unfolding the wrapping phase corresponding to the target frequency to obtain the unfolded phase of the sinusoidal stripes within one cycle, where the target frequency is the frequency with the largest value. That is, after the side cameras have captured all target images corresponding to the five-frequency four-step sinusoidal stripes, the unfolded phase of the five-frequency four-step sinusoidal stripes within one cycle can be determined based on the target images captured by the side cameras. The unfolded phase refers to recovering the original phase value from the wrapping phase. In one embodiment, the wrapping phase of each frequency can be determined based on the target image captured by each side camera corresponding to each step of each frequency. Then, the wrapping phase of the frequency with the largest value is unfolded to obtain the unfolded phase of the five-frequency four-step sinusoidal stripes within one cycle.

[0079] The "wrapping phase" refers to the phase information obtained through phase-shifting technology. This phase is constrained between -π and π, hence the term "wrapping." Specifically, for the target image captured by each side camera corresponding to four steps at each frequency, the wrapping phase at each frequency can be determined using the following formula:

[0080]

[0081] in This represents the coordinates of a pixel in the target image. Represents the coordinates in the target image In the i The corresponding grayscale values ​​at each step size Then it is the coordinates After obtaining the wrapped phase, since the wrapped phase is between [-π, π], it needs to be further normalized to the range [0, 1]. In one optional expansion method, the multi-frequency heterodyne principle is used to expand the low-frequency wrapped phase to assist the high-frequency wrapped phase.

[0082] Then, by combining the target image captured by each side camera with the wrapping phase corresponding to each frequency of the sine stripe, the wrapping phase corresponding to each frequency in the sine stripe can be determined.

[0083] After determining the wrapping phase of each frequency of the sinusoidal fringe, it is necessary to further expand the wrapping phase of the target frequency. The target frequency is the frequency with the largest numerical value among all frequencies. The reason for choosing the frequency with the largest numerical value is that the frequency with the largest numerical value has the highest accuracy. The five frequencies in the five-frequency four-step sinusoidal fringe are decreasing in order, so the first frequency can be taken as the target frequency. When expanding the wrapping phase of the first frequency, multi-frequency heterodyne can be used to expand the wrapping phase. After expansion, an unambiguous one-cycle phase diagram is obtained. The specific formula is as follows:

[0084]

[0085] in, For the expanded phase of the first frequency, floor For the floor function, It is the difference between the wrapping phase of the first frequency and the wrapping phase of the second frequency. If the difference is greater than 0, it is retained; if it is less than 0, it is incremented by 1. This is the value of the first frequency. This represents the difference between the first and second frequencies; if the difference is less than 0, the absolute value is taken. This is the wrap-around phase of the first frequency.

[0086] The embodiments of the present invention calculate the wrapping phase of the sinusoidal fringes at each frequency, thereby limiting the phase value to a specific range, eliminating phase discontinuities, and making the phase value easier to process and analyze, so that the unfolding phase of the sinusoidal fringes within one period can be accurately and effectively determined subsequently.

[0087] Step S133: Based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system, determine the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera.

[0088] Among them, the second transformation matrix is ​​one of the transformation matrices that need to be confirmed. In this embodiment, the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera is confirmed by the least squares method.

[0089] Step S134: Based on the coordinates of the projection points and the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system, confirm the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system.

[0090] Furthermore, this embodiment confirms the third transformation matrix from the corresponding pixel coordinate system to the world coordinate system through a telecentric lens imaging model; that is, it confirms the last required transformation matrix. It should be noted that the embodiments of this application describe the confirmation of the first, second, and third transformation matrices sequentially. This is merely a textual order and does not imply a limitation on the specific processing order. The matrices can be determined in any order, or, if the processing power of the electronic device is sufficient, they can be processed in parallel. The finally confirmed first, second, and third transformation matrices can be recorded separately and used to transform the image coordinates sequentially during actual detection; alternatively, the first, second, and third transformation matrices can be multiplied in the correct transformation order and fused to obtain a final transformation matrix. During actual detection, the image coordinates can be directly transformed using this fused final transformation matrix.

[0091] In implementing steps S133 and S134, based on necessary prior processing steps, the center point coordinates Pc in the side camera pixel coordinate system, Pcc in the center camera pixel coordinate system, Pp in the projected pixel coordinate system, and Pw in the world coordinate system are obtained for each pose state. The homography matrix H transformation relationship between the above point sets is calculated, including the transformation relationship Hcp between the center camera and the center projection, and the transformation relationship Hpw between the center projection and the world coordinate system. Simultaneously, using the telecentric lens imaging model, the pixel size of the telecentric lens is calculated, and the above coordinates are substituted into the telecentric lens imaging model as follows:

[0092]

[0093] In the telecentric lens imaging model, m represents the telecentric lens magnification, s is the lens distortion factor, (u0, v0) is the image coordinate origin, and r and t are the rotation and translation matrices in the extrinsic parameters, respectively. In the experiment, the initial value of m, i.e., the telecentric lens magnification, can be estimated as 1 (telecentric lens parameter information), the initial value of s is set to 0, and the initial value of the camera origin is set to the pixel center point. The extrinsic parameters are obtained by simultaneously solving the known initialization intrinsic parameters and the normalization matrix; here, we assume... h ij This represents the number in the i-th row and j-th column of the normalized matrix H. Based on the above formula, the normalized matrix formula can be obtained. Compared to the normalized matrix of the pinhole model, this matrix does not have a normalized matrix on the Z-axis.

[0094]

[0095] This normalization function can be directly solved using the DLT (Direct Linear Transformation) method to obtain H. Since R is an orthogonal identity matrix, we can then obtain:

[0096]

[0097] Combining the above formulas, we get (assuming m / du = m / dv = a):

[0098]

[0099] Then, based on the rotation matrix constraints... r ij ^2= h ijSince ^2 / a^2 <= 1, we can obtain the solution for 'a', where 'a' represents the pixel size of the central projection. This pixel size can be obtained from the pixel size of the projection DMD and the magnification of the telecentric lens (this data can be provided by the manufacturer at the time of manufacture). Therefore, we can compare the factory-set pixel size with the calibrated pixel size to determine whether the calibration result conforms to the actual physical imaging model, thus obtaining the intrinsic parameter matrix of the central projection. At this point, we have obtained the first transformation matrix from the pixel coordinate system of the side camera to the world coordinate system, the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera, and the third transformation matrix from the pixel coordinate system corresponding to the central projection to the world coordinate system (the third transformation matrix contains the intrinsic parameter matrix of the central projection). This completes the entire calibration process.

[0100] Based on the calibration results above, the object to be tested, such as a circuit board or component, can be placed in the test area. Automatic optical inspection is then completed by projecting coded sinusoidal fringes through central projection and acquiring images through front and side cameras. The basic workflow in 3D AOI involves image acquisition and processing based on existing calibration, which will not be detailed here.

[0101] Figure 2 This is a schematic diagram of the structure of a calibration device for an automated optical inspection apparatus provided in an embodiment of this application. The automated optical inspection apparatus includes a central camera and a central projector sharing a single lens, as well as multiple side cameras arranged around the central camera, such as... Figure 2 As shown, the calibration device of the automatic optical inspection equipment includes a stripe coding unit 210, an image acquisition unit 220, and a matrix confirmation unit 230.

[0102] Among them, the stripe coding unit 210 is used to encode multiple sinusoidal stripes based on a preset frequency and a preset number of steps. The multiple sinusoidal stripes are projected sequentially onto the calibration plate in each pose state through central projection. Multiple calibration points are drawn on the surface of the calibration plate. The calibration plate in each pose state faces the central camera or different side cameras respectively.

[0103] The image acquisition unit 220 is used to acquire a first image of the calibration plate under light illumination by the central camera and the side camera in each pose state, and to acquire a second image of the calibration plate under sinusoidal fringe projection each time by the side camera.

[0104] The matrix verification unit 230 is used to verify the transformation matrix from the pixel coordinate system of the side camera, the center camera, and the center projection to the corresponding target coordinate system based on the associated images of the side camera, the center camera, and the center projection in the image set, and to use the transformation matrix as the calibration result of the automatic optical inspection device; the image set includes the first image, the second image, and the projection content of the center projection.

[0105] Based on the above embodiments, the matrix verification unit 230 includes:

[0106] The first transformation confirmation module is used to confirm the homography matrix between the pixel coordinate system and the world coordinate system of each side camera based on the first image, and to confirm the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each side camera based on the homography matrix.

[0107] The coordinate confirmation module is used to calculate the phase of the center point coordinates of each calibration point in the first image through the second image acquired by each side camera, and to transform the center point coordinates of each circle in each pose state to the pixel coordinate system corresponding to the center projection to obtain the corresponding projection point coordinates based on the phase.

[0108] The second transformation confirmation module is used to confirm the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera, based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system.

[0109] The third transformation confirmation module is used to confirm the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system based on the coordinates of the projection point and the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system.

[0110] Based on the above embodiments, the first conversion confirmation module includes:

[0111] The parameter confirmation submodule is used to obtain the intrinsic and extrinsic parameters of each side camera by calibrating the first image corresponding to each side camera using Zhang Zhengyou's calibration.

[0112] The product confirmation submodule is used to obtain the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system for each side camera based on the product of the intrinsic and extrinsic parameters corresponding to each side camera.

[0113] Based on the above embodiments, the second conversion confirmation module includes:

[0114] The least squares processing submodule is used to determine the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera by using the least squares method, based on the pixel coordinates and projection coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system.

[0115] Based on the above embodiments, the coordinate confirmation module includes:

[0116] The first phase confirmation submodule is used to confirm the wrapping phase of each side camera at each frequency by using the second image corresponding to the sinusoidal stripe acquired by each side camera at each frequency.

[0117] The second phase confirmation submodule is used to expand the wrapping phase corresponding to the target frequency to obtain the expanded phase of the sine stripes within one period, where the target frequency is the frequency with the largest value.

[0118] Based on the above embodiments, the third conversion confirmation module includes:

[0119] The imaging model application submodule is used to confirm the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system through the telecentric lens imaging model.

[0120] The calibration device for the automatic optical inspection equipment provided in this application embodiment is included in an electronic device and can be used to execute the calibration method for the corresponding automatic optical inspection equipment provided in the above embodiment, and has corresponding functions and beneficial effects.

[0121] It is worth noting that in the embodiments of the calibration device of the above-mentioned automatic optical inspection equipment, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0122] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device includes a processor 310 and a memory 320, and may also include an input device 330, an output device 340, and a communication device 350; the number of processors 310 in the electronic device may be one or more. Figure 3 Taking a processor 310 as an example; the processor 310, memory 320, input device 330, output device 340, and communication device 350 in the electronic device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0123] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the calibration method of the automatic optical inspection equipment in the embodiments of this application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby realizing the above-mentioned calibration method of the automatic optical inspection equipment.

[0124] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0125] Input device 330 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 340 may include display devices such as a display screen.

[0126] The aforementioned electronic device includes a calibration device for an automatic optical inspection device, which can be used to perform calibration methods for any automatic optical inspection device and has corresponding functions and beneficial effects.

[0127] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program performs relevant operations in the calibration method of the automatic optical inspection device provided in any embodiment of this application, and has corresponding functions and beneficial effects.

[0128] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0129] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0131] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A calibration method for an automatic optical inspection device, characterized in that, The automated optical inspection device includes a central camera and a central projector sharing a single lens, as well as multiple side cameras arranged around the central camera. The calibration method for the automated optical inspection device includes: Multiple sinusoidal stripes are encoded based on a preset frequency and a preset number of steps. The multiple sinusoidal stripes are sequentially projected onto a calibration plate in each pose state through the central projection. Multiple calibration points are drawn on the surface of the calibration plate. Each calibration plate in each pose state is fixed to a calibration bracket and faces the central camera or different side cameras respectively. The calibration bracket is used to carry the calibration plate into the detection area sequentially during the calibration process. In each pose state, the calibration plate is captured by the central camera and the side camera under the illumination of the light source, and the calibration plate is captured by the side camera under the projection of the sinusoidal stripes each time. Based on the associated images of the side camera, the center camera, and the center projection in the image set, the transformation matrix from the pixel coordinate system of the side camera, the center camera, and the center projection to the corresponding target coordinate system is determined, and the transformation matrix is ​​used as the calibration result of the automatic optical detection device; the image set includes the first image, the second image, and the projection content of the center projection; The step of determining the transformation matrix from the pixel coordinate system to the corresponding target coordinate system of each of the side camera, the center camera, and the center projection based on their respective associated images in the image set includes: Based on the first image captured by the side camera, determine the homography matrix between the pixel coordinate system and the world coordinate system of each side camera, and based on the homography matrix, determine the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each side camera; By using the second image captured by each of the side cameras, the phase of the center point coordinates of each calibration point in the first image is calculated, and the center point coordinates of each pose state are transformed to the pixel coordinate system corresponding to the center projection to obtain the corresponding projection point coordinates based on the phase. Based on the pixel coordinates of each calibration point in the first image captured by the central camera in the corresponding pixel coordinate system and the coordinates of the projection point, a second transformation matrix is ​​determined from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera. Based on the coordinates of the projection points and the pixel coordinates of each calibration point in the first image captured by the central camera in the corresponding pixel coordinate system, the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system is determined.

2. The calibration method for the automatic optical inspection equipment according to claim 1, characterized in that, Based on the first image, determine the homography matrix between the pixel coordinate system and the world coordinate system of each of the side cameras, and based on the homography matrix, determine the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each of the side cameras, including: By calibrating the first image corresponding to each side camera using Zhang Zhengyou, the intrinsic and extrinsic parameters of each side camera are obtained; Based on the product of the intrinsic and extrinsic parameters corresponding to each side camera, a first transformation matrix from the corresponding pixel coordinate system to the world coordinate system is obtained for each side camera.

3. The calibration method for the automatic optical inspection equipment according to claim 1, characterized in that, The step of determining the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera, based on the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system and the coordinates of the projection point, includes: Based on the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system and the coordinates of the projection point, the second transformation matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera is determined by the least squares method.

4. The calibration method for the automatic optical inspection equipment according to claim 1, characterized in that, The step of calculating the phase of the center point coordinates of each calibration point in the first image from the second image acquired by each of the side cameras includes: The wrapping phase of each side camera at each frequency is confirmed by using the second image corresponding to the sinusoidal stripe acquired by each side camera at each frequency. The wrapped phase corresponding to the target frequency is unfolded to obtain the unfolded phase of the sinusoidal stripe within one period, where the target frequency is the frequency with the largest value.

5. The calibration method for the automatic optical inspection equipment according to claim 1, characterized in that, The step of determining the third transformation matrix from the corresponding pixel coordinate system to the world coordinate system of the central projection based on the coordinates of the projection points and the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system includes: The third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system was confirmed using the telecentric lens imaging model.

6. A calibration device for an automatic optical inspection equipment, characterized in that, The automated optical inspection device includes a central camera and a central projector sharing a single lens, as well as multiple side cameras arranged around the central camera. The calibration device of the automated optical inspection device includes: A stripe encoding unit is used to encode multiple sinusoidal stripes based on a preset frequency and a preset number of steps. The multiple sinusoidal stripes are sequentially projected onto a calibration plate in each pose state through the central projection. Multiple calibration points are drawn on the surface of the calibration plate. Each calibration plate in each pose state is fixed to a calibration bracket and faces the central camera or different side cameras respectively. The calibration bracket is used to carry the calibration plate into the detection area sequentially during the calibration process. The image acquisition unit is used to acquire a first image of the calibration plate under light source illumination through the central camera and the side camera in each pose state, and to acquire a second image of the calibration plate under sinusoidal fringe projection each time through the side camera; The matrix verification unit is used to verify the transformation matrix from the pixel coordinate system to the corresponding target coordinate system of the side camera, the center camera, and the center projection, respectively, based on the associated images of the side camera, the center camera, and the center projection in the image set, and to use the transformation matrix as the calibration result of the automatic optical inspection device; the image set includes the first image, the second image, and the projection content of the center projection; The matrix confirmation unit includes: The first transformation confirmation module is used to confirm the homography matrix between the pixel coordinate system and the world coordinate system of each side camera based on the first image captured by the side camera, and to confirm the first transformation matrix from the corresponding pixel coordinate system to the world coordinate system of each side camera based on the homography matrix. The coordinate confirmation module is used to calculate the phase of the center point coordinates of each calibration point in the first image through the second image acquired by each of the side cameras, and to convert the center point coordinates of each circle in each pose state to the pixel coordinate system corresponding to the center projection to obtain the corresponding projection point coordinates based on the phase. The second conversion confirmation module is used to confirm the second conversion matrix from the pixel coordinate system corresponding to the central projection to the pixel coordinate system corresponding to the central camera, based on the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system and the coordinates of the projection point; The third transformation confirmation module is used to confirm the third transformation matrix of the central projection from the corresponding pixel coordinate system to the world coordinate system based on the coordinates of the projection point and the pixel coordinates of each calibration point in the first image acquired by the central camera in the corresponding pixel coordinate system.

7. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the calibration method for the automatic optical inspection device as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the calibration method for the automated optical inspection device as described in any one of claims 1-5.

Images