3D automatic optical inspection method and system

Abstract

The invention discloses a 3D automatic optical inspection method and system, and relates to the optical inspection technology, and the method comprises the steps: employing a camera and an optical machine to form a 3D imaging system for system calibration; controlling an optical machine to sequentially project grating pictures of one full white, one full black, M complementary Gray code black and white stripes and N sine phase shift stripes, and acquiring the pictures by using a camera; for any ray machine, calculating a wrapped phase value by using N sine phase shift stripe gratings collected by a camera; calculating a phase period code for the Gray code black and white fringes; unwrapping the wrapped phase along the reference plane phase diagram by using system geometric constraints to obtain an unwrapped phase diagram; calculating to obtain an object point cloud by using a triangulation principle; and repeating the process of any optical machine, and averaging the point cloud depths to obtain a final point cloud. According to the invention, high-precision three-dimensional measurement is realized by using a small number of grating pictures by adopting complementary Gray codes, high-frequency phase shift grating images and a phase calculation algorithm based on geometric constraints.

Description

Technical Field

[0001] The present application relates to the field of optical inspection technology, and in particular to a 3D automatic optical inspection method and system. Background Art

[0002] Automated optical inspection (AOI) is an automated visual inspection technology widely used in the electronics manufacturing industry. It relies on high-resolution cameras, light sources, and image processing software to quickly analyze object features and identify defects. Traditional 2D AOI systems have been widely used in the field of electronics manufacturing and are usually composed of one or more high-resolution industrial cameras and light sources. However, 2D AOI can only detect defects in two-dimensional planes and has difficulty identifying defects in the height direction of objects. It has gradually failed to meet the increasingly stringent quality control requirements.

[0003] In contrast, 3D AOI technology is a more advanced automatic optical inspection technology that uses three-dimensional imaging technology to simultaneously detect two-dimensional plane and height defects of objects. 3D AOI usually uses a 3D camera module to work in collaboration with a light source. Three-dimensional imaging technology is mainly divided into two categories: line laser scanning and structured light three-dimensional reconstruction. Among them, the structured light three-dimensional reconstruction solution is more widely used due to its faster imaging speed.

[0004] Current 3D AOI systems are mainly used for circuit board component inspection in the field of electronic manufacturing, with micron-level high accuracy and a small field of view (usually 60mm). 2 For larger objects, the system usually needs to capture and merge images in different areas, which results in a longer overall inspection time. Although some industrial 3D cameras can provide a larger measurement field of view, their measurement accuracy is far lower than the requirements of 3D AOI systems and cannot meet the requirements of high-precision three-dimensional defect detection.

[0005] Existing industrial 3D cameras usually consist of an optical machine and one or two cameras. The optical machine projects a grating pattern onto the surface of the object to be measured. The modulation of the grating pattern on the surface of the object causes deformation. The camera captures the deformed image, generates a phase map of the surface contour of the object through image processing, and reconstructs the three-dimensional point cloud using the system parameters calibrated by the 3D camera. However, due to the angular difference between the optical axes of the optical machine and the camera, the focal points of the two are not on the same plane, which may cause the grating or the captured image to be out of focus, thereby reducing the measurement precision and detection accuracy. In addition, existing industrial 3D cameras are usually not equipped with additional light sources and rely solely on optical machines as light sources, which can easily lead to poor uniformity of the captured 2D images and overexposure of the central area, thereby affecting the accuracy of 2D plane detection.

[0006] At present, the structured light 3D reconstruction technology of industrial 3D cameras mainly uses a combination of Gray code and phase-shift grating, or a multi-frequency phase-shift grating to calculate the phase. Although the multi-frequency phase-shift grating method can achieve high-precision phase calculation through high-frequency grating, it is more sensitive to image noise and prone to calculation errors, thereby reducing the stability of detection. The phase calculation accuracy of the Gray code and phase-shift grating combination method is higher, but the Gray code encoding level corresponding to the high-frequency phase-shift grating will increase, resulting in an increase in the total number of grating images, thereby extending the image acquisition time and reducing the detection speed. Summary of the invention

[0007] The embodiments of the present application provide a 3D automatic optical inspection method and system, which adopts complementary Gray code and high-frequency phase-shifted grating images, and a phase calculation algorithm based on geometric constraints to achieve high-precision three-dimensional measurement with a small number of grating images.

[0008] The present application provides a 3D automatic optical inspection method, including:

[0009] Use cameras and optical machines to calibrate the 3D imaging system to obtain the internal and external parameters of each device;

[0010] Control the optical machine to sequentially project 1 all-white image, 1 all-black image, M complementary Gray code black and white stripes, and N sinusoidal phase-shifted stripes grating images, and use the camera to collect the images;

[0011] For any optical machine, the wrapped phase value is calculated using N sinusoidal phase-shifted fringe gratings captured by the camera;

[0012] For any M complementary Gray code black and white stripes of an optical machine, use the complementary Gray code decoding algorithm to calculate the phase period code; and,

[0013] For any optical machine, the measurement plane where the object is located is taken as the reference plane, and the three-dimensional coordinates of the points on the reference plane in the camera coordinate system are calculated;

[0014] According to the calculated phase period encoding and utilizing the system geometric constraints, the wrapped phase is unfolded along the reference plane phase diagram to obtain an unfolded phase diagram;

[0015] Using the principle of triangulation, the point cloud of the object is calculated;

[0016] Repeat any optical-mechanical processing process, take the average of the point cloud depth, and get the final point cloud.

[0017] Optionally, it also includes pre-connecting system equipment, wherein the system equipment includes a 3D imaging system consisting of a camera and two optical machines, a processing device, and a control device:

[0018] Connect the camera to the processing device;

[0019] Connect the processing device to the control device to control the optical machine and camera;

[0020] Connecting the control device to the two optical machines to control the two optical machines to project gratings, and connecting the control device to the camera to use the trigger signal returned by the optical machine to trigger the camera;

[0021] Connect the camera to a light source.

[0022] Optionally, a 3D imaging system composed of a camera and an optical machine is used for system calibration. The internal parameters of each device obtained include:

[0023] Focal length (f x ,f y ), principal point coordinates (x0, y0) and lens distortion parameters

[0024] External parameters include: the device's rotation matrix R 3×3 and the translation vector t 3×1 .

[0025] Optionally, the optical machine is controlled to sequentially project 1 all-white image, 1 all-black image, M complementary Gray code black and white stripes, and N sinusoidal phase-shifted stripe grating images, and the camera is used to collect images including:

[0026] When the optical machine is symmetrical on the left and right sides of the camera, the grating stripes are oriented longitudinally;

[0027] When the optical machine is symmetrical above and below the camera, the direction of the grating stripes is horizontal.

[0028] Optionally, for any optical machine, the N sinusoidal phase-shifted fringe gratings captured by the camera satisfy:

[0029]

[0030] Where A(x,y) represents the average grayscale of the grating, B(x,y) represents the grayscale of the grating modulation, n is the image number n = 0, 1, 2, ..., N-1, is the wrap phase value, calculated satisfy:

[0031]

[0032] N grating images of sinusoidal phase-shifted fringes are analyzed to obtain a wrapped phase map.

[0033] Optionally, for any one optical machine, M complementary Gray code black and white stripes are decoded using a complementary Gray code, and the phase period encoding is calculated including:

[0034] For any M complementary Gray code black and white stripes of an optical machine, the complementary Gray code decoding algorithm is used to calculate two phase period codes K1(x, y) and K2(x, y), where the phase period code K1(x, y) is obtained by decoding the first M-1 Gray codes, and the phase period code K2(x, y) is calculated by the Mth complementary Gray code;

[0035] Expand the wrapped phase diagram to satisfy:

[0036]

[0037] The number of phase cycles after expansion is 2 M-1 ω pixels, ω represents the period of the sinusoidal phase shift fringe, 2 M-1 ω <Res p ,Res p is the resolution of the optomechanical grating fringe direction.

[0038] Optionally, for any optical machine, the measurement plane where the object is located is used as the reference plane, and the three-dimensional coordinates (X c ,Y c ,Z c )satisfy:

[0039]

[0040] Where H represents the height of the measurement plane of the 3D AOI from the camera imaging center. is the camera focal length, is the camera principal point coordinate;

[0041] Optical-mechanical coordinate system 3D coordinates (X p ,Y p ,Z p )satisfy:

[0042]

[0043] in and are the camera external parameters, and is the external parameter of the optical machine;

[0044] The reference phase diagram of the calculated reference plane satisfies:

[0045]

[0046] Optionally, according to the calculated phase period encoding, the wrapped phase is unfolded along the reference plane phase diagram using the system geometric constraints, and the unfolded phase diagram Φ(x,y) satisfies:

[0047]

[0048] According to the principle of triangulation, the depth coordinate Z of the object in the world coordinate system is calculated using the fully expanded phase image Φ(x,y) and the internal and external parameters of the camera and optical machine. w (x,y).

[0049] An embodiment of the present application also proposes a 3D automatic optical inspection system, including a 3D imaging system consisting of a camera and two optical machines, a processing device, and a control device. A computer program is stored on the processing device, and when the computer program is executed by the processor, the steps of the aforementioned 3D automatic optical inspection method are implemented.

[0050] The method of the embodiment of the present application adopts complementary Gray code and high-frequency phase-shifted grating images, and a phase calculation algorithm based on geometric constraints to achieve high-precision three-dimensional measurement with a small number of grating images.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0053] Figure 1 The basic process diagram of the 3D automatic optical inspection method according to an embodiment of the present application is shown;

[0054] Figure 2 The optical inspection system of the 3D automatic optical inspection method according to the embodiment of the present application is composed of. DETAILED DESCRIPTION

[0055] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0056] The embodiment of the present application provides a 3D automatic optical inspection method. The method of the embodiment of the present application adopts complementary Gray code and high-frequency phase-shifted grating images, and a phase calculation algorithm based on geometric constraints to achieve high-precision three-dimensional measurement with a small number of grating images. The 3D automatic optical inspection device includes a high-resolution camera, a lens, two axis-shift optical machines, a control device, a light source and a processing device, which can simultaneously realize large-viewing-field, high-precision 3D and 2D detection data acquisition and analysis work, such as Figure 1 The method of the present application comprises the following steps:

[0057] In step S101, the system equipment is pre-connected, wherein the system equipment includes a 3D imaging system composed of a camera and two optical machines, a processing device, and a control device:

[0058] Connect the camera to the processing device and transfer data;

[0059] Connect the processing device to the control device to control the optical machine and camera;

[0060] Connecting the control device to the two optical machines to control the two optical machines to project gratings, and connecting the control device to the camera to use the trigger signal returned by the optical machine to trigger the camera;

[0061] Connect the camera to a light source, and the camera can selectively trigger the light source when taking pictures.

[0062] In a specific example, Figure 2 As shown, the system consists of a high-resolution camera 2, a lens, two shift optical machines, the first optical machine 1 and the second optical machine 3; a control device 6, a light source 5 and a processing device 7. Camera: placed in the center, with the optical axis facing the object to be measured vertically downward. Lens 4: installed in front of the camera, matching the resolution of the camera and meeting a larger field of view. Shift optical machine: placed symmetrically on both sides of the camera (either left-right symmetrical or top-down symmetrical), the two optical machines face the object to be measured at a certain angle relative to the camera, and the optical axes intersect with the optical axis of the camera, and the two optical machines present a centrally symmetrical posture relative to the optical axis of the camera. The field of view of the camera and the two optical machines coincides with the plane where the object to be measured is located. The optical machine adopts a shift optical design to ensure that the two optical machines focus on the same plane. Light source 5: installed below the camera 2 and the optical machine, with a hollow ring design to avoid the field of view of the camera and the optical machine. Control device 6: connected to the camera, optical machine, light source and processing device 7 respectively, used to realize the control of camera 2, optical machine and light source 5.

[0063] In step S102, a 3D imaging system is calibrated using a camera and an optical machine to obtain internal and external parameters of each device. In some embodiments, the internal parameters of each device include: focal length (f x ,f y ), principal point coordinates (x0, y0) and lens distortion parameters External parameters include: the device's rotation matrix R 3×3 and the translation vector t 3×1 .

[0064] In step S103, the optical machine is controlled to sequentially project 1 all-white, 1 all-black, M complementary Gray code black and white stripes, and N grating images of sinusoidal phase-shifted stripes, and the camera is used to collect the images. In some embodiments, the optical machine is controlled to sequentially project 1 all-white, 1 all-black, M complementary Gray code black and white stripes, and N grating images of sinusoidal phase-shifted stripes (N≥7), and the camera is used to collect the images, including:

[0065] When the optical machine is symmetrical on the left and right sides of the camera, the grating stripes are oriented longitudinally;

[0066] When the optical machine is symmetrical above and below the camera, the direction of the grating stripes is horizontal.

[0067] In step S104, for any optical machine, the wrapped phase value is calculated using the N sinusoidal phase-shifted fringe gratings captured by the camera.

[0068] In step S105, for any one of the M complementary Gray code black and white stripes of the optical machine, a complementary Gray code decoding algorithm is used to calculate the phase period code; and,

[0069] In step S106 , for any optical machine, the measurement plane where the object is located is taken as a reference plane, and the three-dimensional coordinates of the points on the reference plane in the camera coordinate system are calculated.

[0070] In step S107, according to the calculated phase period code and utilizing the system geometric constraints, the wrapped phase is unfolded along the reference plane phase diagram to obtain an unfolded phase diagram.

[0071] In step S108, the object point cloud is calculated using the triangulation principle.

[0072] In step S109, any optical-mechanical processing process is repeated, and the point cloud depth is averaged to obtain the final point cloud.

[0073] In some embodiments, for any optical machine, the N sinusoidal phase-shifted fringe gratings captured by the camera satisfy:

[0074]

[0075] Where A(x,y) represents the average grayscale of the grating, B(x,y) represents the grayscale of the grating modulation, n is the image number n = 0, 1, 2, ..., N-1, is the wrap phase value, calculated satisfy:

[0076]

[0077] N grating images of sinusoidal phase-shifted fringes are analyzed to obtain a wrapped phase map.

[0078] In some embodiments, for any one optical machine, using a complementary Gray code decoding algorithm, calculating the phase period encoding includes:

[0079] For any M complementary Gray code black and white stripes of an optical machine, the complementary Gray code decoding algorithm is used to calculate two phase period codes K1(x, y) and K2(x, y), where the phase period code K1(x, y) is obtained by decoding the first M-1 Gray codes, and the phase period code K2(x, y) is calculated by the Mth complementary Gray code;

[0080] Expand the wrapped phase diagram to satisfy:

[0081]

[0082] The number of phase cycles after expansion is 2 M-1 ω pixels, ω represents the period of the sinusoidal phase-shifted fringes.

[0083] We use high-frequency phase-shifted fringes (small ω) to improve the accuracy of 3D reconstruction. A smaller M is used to reduce the number of complementary Gray code gratings. Setting 2 M-1 ω <Res p , where Res p The resolution of the optical machine grating stripe direction is improved, thereby improving the accuracy of the optical machine projection grating, and the resolution of the optical machine is higher. Only partially expanded, still wrapped in 2 M-1 ω pixel period.

[0084] In some embodiments, for any optical machine, the measurement plane where the object is located is used as the reference plane, and the three-dimensional coordinates (X c ,Y c ,Z c )satisfy:

[0085]

[0086] Where H represents the known height of the measurement plane of the 3D AOI from the camera imaging center. is the camera focal length, is the camera principal point coordinate;

[0087] Optical-mechanical coordinate system 3D coordinates (X p ,Y p ,Z p )satisfy:

[0088]

[0089] in and are the camera external parameters, and is the external parameter of the optical machine;

[0090] The reference phase diagram of the calculated reference plane satisfies:

[0091]

[0092] In some embodiments, the wrapped phase image is unfolded according to the calculated phase period encoding, and the wrapped phase is unfolded along the reference plane phase image using the system geometric constraints to obtain an unfolded phase image Φ(x, y) satisfying:

[0093]

[0094] According to the principle of triangulation, the depth coordinate Z of the object in the world coordinate system is calculated using the fully expanded phase image Φ(x,y) and the internal and external parameters of the camera and optical machine. w (x,y).

[0095] For two optical machines, the M complementary Gray code black and white stripes of any optical machine are repeated and the phase period encoding is calculated using the complementary Gray code decoding algorithm to obtain the depth coordinates Z w1 (x,y),Z w2 (x, y). Then take the average of the two sets of depth coordinates to get the depth coordinate Z w (x,y) is the final object point cloud.

[0096] The complementary Gray code encoding scheme adopted by the method of the embodiment of the present application can achieve a higher decoding accuracy compared to the ordinary Gray code encoding scheme, improve the accuracy of point cloud computing, and thus improve the accuracy of automatic optical detection. In addition, the present application adopts a smaller number of gratings than normal complementary Gray code encoding, and uses the geometric parameters calibrated by the 3D AOI system to establish a reference plane based on the measurement plane, and uses the reference plane to solve the problem that a small number of complementary Gray code encoded gratings cannot fully unfold the phase. The method of the embodiment of the present application only needs to use a smaller number of gratings than normal complementary Gray code encoding, which reduces the time of image acquisition and improves the efficiency of automatic optical detection.

[0097] An embodiment of the present application also proposes a 3D automatic optical inspection system, including a 3D imaging system consisting of a camera and two optical machines, a processing device, and a control device. A computer program is stored on the processing device, and when the computer program is executed by the processor, the steps of the aforementioned 3D automatic optical inspection method are implemented.

[0098] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0099] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0101] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A 3D automatic optical inspection method, characterized in that: include: Use cameras and optical machines to calibrate the 3D imaging system to obtain the internal and external parameters of each device; Control the optical machine to sequentially project 1 all-white image, 1 all-black image, M complementary Gray code black and white stripes, and N sinusoidal phase-shifted stripes grating images, and use the camera to collect the images; For any optical machine, the wrapped phase value is calculated using N sinusoidal phase-shifted fringe gratings captured by the camera; For any M complementary Gray code black and white stripes of an optical machine, the phase period code is calculated using the complementary Gray code decoding algorithm; as well as, For any optical machine, the measurement plane where the object is located is taken as the reference plane, and the three-dimensional coordinates of the points on the reference plane in the camera coordinate system are calculated; According to the calculated phase period encoding and utilizing the system geometric constraints, the wrapped phase is unfolded along the reference plane phase diagram to obtain an unfolded phase diagram; Using the principle of triangulation, the point cloud of the object is calculated; Repeat any optical-mechanical processing process, take the average of the point cloud depth, and get the final point cloud.

2. The 3D automatic optical inspection method according to claim 1, characterized in that: It also includes pre-connected system equipment, where the system equipment includes a 3D imaging system consisting of a camera and two optical machines, a processing device, and a control device: Connect the camera to the processing device; Connect the processing device to the control device to control the optical machine and camera; Connecting the control device to the two optical machines to control the two optical machines to project gratings, and connecting the control device to the camera to use the trigger signal returned by the optical machine to trigger the camera; Connect the camera to a light source.

3. The 3D automatic optical inspection method according to claim 2, characterized in that: The camera and the optical machine form a 3D imaging system for system calibration. The internal parameters of each device obtained include: Focal length (f x ,f y ), principal point coordinates (x0, y0) and lens distortion parameters External parameters include: the device's rotation matrix R 3×3 and the translation vector t 3×1 .

4. The 3D automatic optical inspection method according to claim 1, characterized in that: The optical machine is controlled to sequentially project 1 all-white, 1 all-black, M complementary Gray code black and white stripes, and N sinusoidal phase-shifted stripes grating images, and the camera is used to collect the following images: When the optical machine is symmetrical on the left and right sides of the camera, the grating stripes are oriented longitudinally; When the optical machine is symmetrical above and below the camera, the direction of the grating stripes is horizontal.

5. The 3D automatic optical inspection method according to claim 2, characterized in that: For any optical machine, the N sinusoidal phase-shifted fringe gratings collected by the camera satisfy: Where A(x,y) represents the average grayscale of the grating, B(x,y) represents the grayscale of the grating modulation, n is the image number n = 0, 1, 2, ..., N-1, is the wrap phase value, calculated satisfy: N grating images of sinusoidal phase-shifted fringes are analyzed to obtain a wrapped phase map.

6. The 3D automatic optical inspection method according to claim 5, characterized in that: For any M complementary Gray code black and white stripes of an optical machine, the complementary Gray code decoding algorithm is used to calculate the phase period code including: For any M complementary Gray code black and white stripes of an optical machine, the complementary Gray code decoding algorithm is used to calculate two phase period codes K1(x, y) and K2(x, y), where the phase period code K1(x, y) is obtained by decoding the first M-1 Gray codes, and the phase period code K2(x, y) is calculated by the Mth complementary Gray code; Expand the wrapped phase diagram to satisfy: The number of phase cycles after expansion is 2 M-1 ω pixels, ω represents the period of the sinusoidal phase shift fringe, 2 M-1 ω <Res p ,Res p is the resolution of the optomechanical grating fringe direction.

7. The 3D automatic optical inspection method according to claim 6, characterized in that: For any optical machine, the measurement plane where the object is located is taken as the reference plane, and the three-dimensional coordinates (X c ,Y c ,Z c )satisfy: Where H represents the height of the measurement plane of the 3D AOI from the camera imaging center. is the camera focal length, is the coordinate of the camera principal point; Optical-mechanical coordinate system 3D coordinates (X p ,Y p ,Z p )satisfy: in and are the camera external parameters, and is the external parameter of the optical machine; The reference phase diagram of the calculated reference plane satisfies:

8. The 3D automatic optical inspection method according to claim 7, characterized in that: According to the calculated phase cycle encoding, the wrapped phase is unfolded along the reference plane phase diagram using the system geometric constraints, and the unfolded phase diagram Φ(x,y) satisfies: According to the principle of triangulation, the depth coordinate Z of the object in the world coordinate system is calculated using the fully expanded phase image Φ(x,y) and the internal and external parameters of the camera and optical machine. w (x,y).

9. A 3D automatic optical inspection system, characterized in that: A 3D imaging system comprising a camera and two optical machines, a processing device, and a control device. A computer program is stored on the processing device. When the computer program is executed by a processor, the steps of the 3D automatic optical inspection method as described in any one of claims 1 to 8 are implemented.

Images