A method and device applied to AOI detection

Abstract

The present application relates to the field of visual detection technology of optical module industry, and provides a method and device applied to AOI detection. A low-power microscopic imaging module of a fixed shot visual subsystem is controlled to acquire a global image of a detection object on an object platform, and each flying shot point coordinate is located from the global image; according to the each flying shot point coordinate, a motion shaft is controlled to drive the flying shot visual subsystem to move, and a plurality of local high-definition images are acquired by performing flying shot on each flying shot point; the acquired plurality of local high-definition images are spliced into a complete to-be-detected image with the global image as a background; each regional image is obtained by performing regional cutting on the to-be-detected image according to the category of the to-be-detected object; and the defect category of each regional image is determined according to a defect feature, and a defect detection result is generated. The present application solves the problem that large-area defects are not fully extracted and are incomplete, and ensures the comprehensiveness and accuracy of detection.

Description

A method and apparatus for AOI inspection Technical Field

[0001] This invention relates to the field of visual inspection technology in the optical module industry, and in particular to a method and apparatus for AOI inspection. Background Technology

[0002] The ever-increasing demand for data transmission rates from artificial intelligence (AI) applications is directly driving the optical module industry into a phase of rapid evolution, with both production scale and quality requirements for optical modules rising simultaneously. Currently, in the optical module production process, while there are numerous high-definition image acquisition devices available for 3D measurement (such as the accuracy of component placement / angle, the flatness of the chip after placement, and the height of adhesive) and defect detection (such as dirt, scratches, chipping, presence or absence of adhesive, and presence or absence of gold wires on component surfaces), manual operation remains the primary method. This manual inspection method not only suffers from large fluctuations in accuracy and low efficiency but also struggles to meet the quality control requirements of large-scale mass production.

[0003] Optical modules are core products in the optical communication industry. The objects and images inspected vary significantly across different process stages. Images differ considerably in color, resolution, and clarity, as these data come from images taken by different optical modules. Furthermore, defect data is relatively random. A typical example, as shown in Figure 1, is the under-segmentation of large-area defects, resulting in incomplete or inaccurate extraction of these defects.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem this invention aims to solve is that optical modules are core products in the optical communication industry. The inspection objects and images vary significantly across different process stages, with substantial differences in color, resolution, and clarity. This data comes from images taken by different optical modules, and defect data is relatively random. A typical example, as shown in Figure 1, is that large-area defects are not fully extracted, resulting in incomplete data.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for AOI detection, comprising:

[0008] The low-magnification microscopic imaging module of the fixed-shot vision subsystem captures a global image of the object to be detected on the platform and uses the global image to locate the coordinates of each shooting point.

[0009] Based on the coordinates of each aerial photography point, the motion axis is controlled to drive the aerial photography vision subsystem to move, and aerial photography is performed at each aerial photography point to acquire multiple local high-definition images.

[0010] Using the global image as a background, multiple acquired local high-definition images are stitched together to form a complete image to be detected; wherein, the image to be detected also serves as the image to be measured in 3D measurement, and the corresponding measurement is completed;

[0011] Based on the category of the object to be detected, the region of the image to be detected is cropped to obtain images of each region; and the defect category of each region image is determined according to the defect characteristics to generate defect detection results.

[0012] Preferably, when acquiring a global image of the object being detected on the platform and locating the coordinates of each camera point from the global image, the method further includes:

[0013] Based on the coordinates of the first round of aerial photography points, the contour grayscale values ​​of the regions in the global image to be stitched into local high-definition images are identified.

[0014] If the outline grayscale value of the corresponding first local high-definition image shows a continuous break, the region of the first local high-definition image is weighted and magnified to update the coordinates of the first round of aerial photography points to obtain the updated coordinates of the second round of aerial photography points. This ensures that the first local high-definition images captured around the coordinates of the second round of aerial photography points can cover the image content at a preset distance in the direction of the continuous break.

[0015] Preferably, when the first local high-definition image belongs to the same type of chip, and it is necessary to capture other local high-definition images of the same type that are independent of the first local high-definition image to obtain the coordinates of multiple aerial shooting points, the continuity breaking specifically includes:

[0016] Based on the coordinates of the first round of aerial photography points, the boundary contours used to replace the first local high-definition image in the global image are determined. A contour gray value array is constructed using the gray values ​​of the contour pixels. If the difference between consecutive values ​​in the contour gray value array is less than a preset value, and the gray value at the corresponding position of the boundary contour array of other similar local high-definition images differs from the gray value at a preset threshold, then it is determined that the contour gray value array of the corresponding first local high-definition image has a continuous break; wherein, the position of the break is the contour position corresponding to the gray value abnormality.

[0017] Preferably, if multiple local high-definition images of the same type as the first local high-definition image are arranged in an array, and the contour region of the continuous breaking circle is located on the first boundary contour opposite to the adjacent first local high-definition image and the second local high-definition image in the array, the weighted amplification of the region of the first local high-definition image specifically includes:

[0018] The boundary of the first local high-definition image is weighted and magnified so that the relative contours of adjacent first local high-definition images and second local high-definition images in the array, after being captured by the weighted first local high-definition image, cover the area between the relative first boundary contours of the first local high-definition image and the second local high-definition image by the weighted captured first local high-definition image.

[0019] Preferably, if multiple local high-definition images of the same type as the first local high-definition image are discretely distributed, the weighted magnification of the region of the first local high-definition image specifically includes:

[0020] The boundary of the first local high-definition image is weighted and magnified so that the boundary of the first local high-definition image is adjacent to the boundary of another type of local high-definition image; or, the boundary of the first local high-definition image is weighted and magnified so that the boundary of the first local high-definition image reaches a preset maximum boundary value.

[0021] Preferably, when acquiring a global image of the object being detected on the platform and locating the coordinates of each camera point from the global image, the method further includes:

[0022] If it is confirmed that the continuous breaking of the circle includes at least the first local high-definition image, the second local high-definition image and the third local high-definition image, and the three are located in the array of the same type of chip object with discrete distribution, the low magnification microscopic imaging module of the fixed-shot vision subsystem is controlled to take fixed shots of the array area composed of the first local high-definition image on the object platform to obtain array supplementary images.

[0023] Based on the array re-captured images, the contour bounding images of the regions between the local high-definition images in the array containing the first local high-definition image are obtained; and with the global image as the background, multiple local high-definition images are superimposed on the global image, and the contour bounding images are used to cover the background between the local high-definition images in the array region, thereby stitching together a complete image to be detected.

[0024] Secondly, the present invention also provides a device for AOI inspection, used to implement the method described in the first aspect. The device includes a flying camera vision subsystem, a fixed camera vision subsystem, a height measurement subsystem, an X-axis motion mechanism, a Y-axis motion mechanism, and a Z-axis motion mechanism. Specifically:

[0025] The aerial photography vision subsystem achieves high-precision detection of localized minute defects through aerial photography mosaicking.

[0026] The fixed-shot vision subsystem achieves global large field-of-view coverage detection of the object to be detected through fixed-shot detection.

[0027] The height measurement subsystem achieves accurate height measurement of the object being measured and detected through physical or optical principles. Depending on the actual accuracy and / or range requirements, one or more height measurement schemes can be selected, such as contact height measurement, laser triangulation, time-of-flight (ToF), ultrasonic height measurement, and visual stereo height measurement.

[0028] The X-axis motion mechanism, Y-axis motion mechanism, and Z-axis motion mechanism are used to drive the three-dimensional movement of the vision subsystem. One of the following is selected: a lead screw, a linear motor, and an air-bearing motion platform, depending on the accuracy requirements.

[0029] Preferably, the aerial photography vision subsystem further includes a high-magnification microscopic imaging module, a telecentric lens, a high-frequency LED light source, and an aerial photography controller, specifically:

[0030] The high-magnification microscopic imaging module uses a global shutter industrial camera as a vision sensor to achieve high-precision detection of localized minute defects;

[0031] A telecentric lens, in conjunction with the high-magnification microscopic imaging module, achieves pixel-level precision.

[0032] High-frequency LED light source, synchronized with camera exposure, avoids uneven image brightness caused by light source flicker during high-speed movement;

[0033] The high-magnification microscopic imaging module's shutter speed, focal length, and aperture parameters are controlled, and high-speed strobe control and synchronous capture are achieved.

[0034] Preferably, the fixed-shot vision subsystem further includes a low-magnification imaging module, a telecentric lens, and a coaxial ring-shaped combined light source, specifically:

[0035] The low-magnification imaging module uses a high-resolution area array camera to achieve a large global field of view coverage of the detected object;

[0036] The telecentric lens supports adjustable magnification and, in conjunction with the low-magnification imaging module, achieves pixel-level precision.

[0037] The coaxial ring light source combines the vertical uniform illumination of coaxial light with the multi-angle edge enhancement of ring light through optical design. By selecting different colors of coaxial light and ring light, accurate detection can be achieved in highly reflective and complex defect scenarios.

[0038] Preferably, the device further includes a feeding buffer unit, a discharging buffer unit, a material detection unit, and a shock-absorbing platform, specifically:

[0039] The feeding buffer unit and the unloading buffer unit are used to buffer the material to be tested and the material that has already been tested;

[0040] The shockproof platform adopts a pneumatic support structure to achieve precise positioning and stable placement of the product, reducing vibration interference during the testing process.

[0041] Thirdly, the present invention also provides an apparatus for AOI detection, used to implement the method for AOI detection described in the first aspect, the apparatus comprising:

[0042] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method for AOI detection as described in the first aspect.

[0043] Fourthly, the present invention also provides a method and apparatus for AOI detection using a non-volatile computer storage medium, wherein the computer storage medium stores computer-executable instructions that are executed by one or more processors to perform the method for AOI detection described in the first aspect.

[0044] This invention achieves high-precision detection of localized minute defects through a high-magnification microscopic imaging module and achieves global large-field coverage of the detected object through a low-magnification microscopic imaging module, successfully solving the problem of incomplete or incomplete extraction of large-area defects and ensuring the comprehensiveness and accuracy of the detection. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0046] Figure 1 is a schematic diagram of a defect provided in an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of a device for AOI detection provided in an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a device and an intelligent handling robot applied to AOI inspection provided in an embodiment of the present invention;

[0049] Figure 4 is a flowchart illustrating a method for AOI detection provided in an embodiment of the present invention;

[0050] Figure 5 is a flowchart illustrating another method for AOI detection provided in an embodiment of the present invention;

[0051] Figure 6 is a schematic diagram of a chip image in a chip array scenario provided by an embodiment of the present invention;

[0052] Figure 7 is a schematic diagram of a local high-definition image provided by an embodiment of the present invention;

[0053] Figure 8 is a schematic diagram of the double-layer contour of a local high-definition image provided by an embodiment of the present invention;

[0054] Figure 9 is a schematic diagram of a mirror image of a first local high-definition image provided in an embodiment of the present invention;

[0055] Figure 10 is a schematic diagram of a first local high-definition image being rotated using a corner of the edge containing the continuous broken circle region as a rotation point, according to an embodiment of the present invention.

[0056] Figure 11 is a flowchart illustrating another method for AOI detection provided in an embodiment of the present invention;

[0057] Figure 12 is a schematic diagram of an array-based image re-capture provided by an embodiment of the present invention;

[0058] Figure 13 is a schematic diagram of a single-product inspection process provided in an embodiment of the present invention;

[0059] Figure 14 is a schematic diagram of a batch product testing process provided in an embodiment of the present invention;

[0060] Figure 15 is a schematic diagram of the architecture of an AOI detection device provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0062] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0063] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0064] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0065] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0066] Example 1:

[0067] This embodiment provides a device for AOI (Automated Optical Inspection) detection, as shown in Figure 2. The device includes a flying camera vision subsystem, a fixed-camera vision subsystem, a height measurement subsystem, an X-axis motion mechanism, a Y-axis motion mechanism, and a Z-axis motion mechanism. Specifically:

[0068] The aerial photography vision subsystem achieves high-precision detection of localized minute defects through aerial photography mosaicking.

[0069] The fixed-shot vision subsystem achieves global large field-of-view coverage detection of the object to be detected through fixed-shot detection.

[0070] The height measurement subsystem achieves accurate height measurement of the object being measured and detected through physical or optical principles. Depending on the actual accuracy and / or range requirements, one or more height measurement schemes can be selected, such as contact height measurement, laser triangulation, time-of-flight (ToF), ultrasonic height measurement, and visual stereo height measurement.

[0071] The X-axis motion mechanism, Y-axis motion mechanism, and Z-axis motion mechanism are used to drive the three-dimensional movement of the vision subsystem. One of the following is selected: a lead screw, a linear motor, and an air-bearing motion platform, depending on the accuracy requirements.

[0072] This invention integrates multiple functional modules, including a flying camera vision subsystem, a fixed camera vision subsystem, and a height measurement subsystem, into one system. It achieves high-precision detection of localized minute defects through a high-magnification microscopic imaging module and achieves global wide-field coverage of the detected object through a low-magnification microscopic imaging module. This successfully solves the problem of balancing detection accuracy and field of view, ensuring the comprehensiveness and accuracy of the detection.

[0073] In conjunction with embodiments of the present invention, there is a preferred implementation in which the aerial photography vision subsystem further includes a high-magnification microscopic imaging module, a telecentric lens, a high-frequency LED light source, and an aerial photography controller, specifically:

[0074] The high-magnification microscopic imaging module uses a global shutter industrial camera as a vision sensor to achieve high-precision detection of localized minute defects;

[0075] A telecentric lens, in conjunction with the high-magnification microscopic imaging module, achieves pixel-level precision.

[0076] High-frequency LED light source, synchronized with camera exposure, avoids uneven image brightness caused by light source flicker during high-speed movement;

[0077] The high-magnification microscopic imaging module's shutter speed, focal length, and aperture parameters are controlled, and high-speed strobe control and synchronous capture are achieved.

[0078] In conjunction with embodiments of the present invention, there is a preferred implementation in which the fixed-shot vision subsystem further includes a low-magnification imaging module, a telecentric lens, and a coaxial ring-shaped combined light source, specifically:

[0079] The low-magnification imaging module uses a high-resolution area array camera to achieve a large global field of view coverage of the detected object;

[0080] The telecentric lens supports adjustable magnification and, in conjunction with the low-magnification imaging module, achieves pixel-level precision.

[0081] The coaxial ring light source combines the vertical uniform illumination of coaxial light with the multi-angle edge enhancement of ring light through optical design. By selecting different colors of coaxial light and ring light, accurate detection can be achieved in highly reflective and complex defect scenarios.

[0082] In conjunction with the embodiments of the present invention, there is also an optional implementation, in which the device further includes a feeding buffer unit, a discharging buffer unit, a material detection unit, and a shock-absorbing platform, specifically:

[0083] The feeding buffer unit and the unloading buffer unit are used to buffer the materials to be tested and the materials that have already been tested;

[0084] The shockproof platform adopts a pneumatic support structure to achieve precise positioning and stable placement of the product, reducing vibration interference during the testing process.

[0085] As shown in Figure 3, the device of the present invention can also have a detection device (i.e., the material detection unit in Figure 2), which adopts an open mechanical structure and is combined with an intelligent handling robot and a robotic arm to realize automatic loading of products to be tested and automatic unloading of products that have been tested. Moreover, when loading / unloading the detection device, it does not affect the normal operation of the detection device. This solution can be compatible with the co-production testing needs of various types of products and improve production flexibility.

[0086] As shown in Figure 2, in a specific implementation, the entire device can also be placed on a workbench as shown in Figure 2. The workbench is mainly used to support the device applied to AOI inspection, and to provide the device with movable casters and corresponding storage functions. Additionally, the material detection unit in this embodiment of the invention is mainly used to assist the material transfer system in detecting whether the material detected by the corresponding aerial vision subsystem and fixed-position vision subsystem is located in the target area.

[0087] Example 2:

[0088] This invention proposes a method for AOI detection. This method can be implemented using the device for AOI detection described in Embodiment 1, or it can be implemented using other key fixed-shot vision subsystems and flying-shot vision subsystems. It should be emphasized that the innovation of this invention lies more in the shooting processing strategy and corresponding method flow based on two vision subsystems shown in the extended implementation, as shown in Figure 4. The method includes:

[0089] In step 201, the low-magnification microscopic imaging module of the fixed-shot vision subsystem is controlled to capture a global image of the object to be detected on the platform, and the coordinates of each shooting point are located from the global image.

[0090] Specifically, based on the global image, the camera locations that require local high-precision detection are located using image analysis algorithms, and the X and Y coordinates of each camera location are determined.

[0091] In step 202, based on the coordinates of each aerial photography point, the motion axis is controlled to move the aerial photography vision subsystem, and aerial photography is performed at each aerial photography point to acquire multiple local high-definition images.

[0092] This includes controlling the altitude measurement subsystem to move to each aerial photography location, measuring and recording the Z-axis altitude coordinates of each location, providing a basis for adjusting the focal length of the aerial photography image.

[0093] During operation, the system works as follows: based on the X and Y coordinates of each shooting point, the X and Y axes are controlled to move the shooting vision subsystem. The focal length of the shooting vision subsystem is adjusted in combination with the Z-axis height coordinate to perform shooting operations on each shooting point and acquire local high-definition images.

[0094] In step 203, using the global image as a background, multiple acquired local high-definition images are stitched together to form a complete image to be detected.

[0095] The image to be detected also serves as the image to be measured in 3D measurement, and the corresponding measurement is performed.

[0096] It should be further explained that the corresponding image stitching algorithm in this embodiment of the invention is to replace the pixel parameter values ​​of the corresponding area in the global image according to the above-mentioned aerial shooting points in the local high-definition image. In the actual implementation process, the subdivision steps involved also include pre-adapting the global image pixel resolution to the local high-definition image by enlarging it. These processes can all be completed with existing image processing technology. The image stitching algorithm here is not an innovative improvement of this invention, so it will not be elaborated on further.

[0097] In step 204, the image to be detected is cropped according to the category of the object to be detected to obtain images of each region; and the defect category of each region image is determined according to the defect features to generate defect detection results.

[0098] This invention achieves high-precision detection of localized minute defects through a high-magnification microscopic imaging module and achieves global large-field coverage of the detected object through a low-magnification microscopic imaging module, successfully solving the problem of balancing detection accuracy and field of view, and ensuring the comprehensiveness and accuracy of the detection.

[0099] In conjunction with the embodiments of the present invention, considering the undersegmentation problem of large-area defects, resulting in incomplete and inaccurate extraction of large-area defects, the method of the embodiments of the present invention, in addition to achieving global large-field coverage of the detected object through a low-magnification microscopic imaging module, successfully solving the problem of incomplete and inaccurate extraction of large-area defects, also provides a more efficient processing method for this problem in the extended implementation scheme of the embodiments of the present invention. When taking fixed shots of the detected object on the platform to obtain a global image, and locating the coordinates of each shooting point from the global image, as shown in Figure 5, the method further includes:

[0100] In step 2011, based on the coordinates of the first round of aerial photography points, the contour grayscale values ​​of the regions in the global image to be stitched into local high-definition images are identified.

[0101] This means that after completing the global image capture and the coordinates of each aerial shooting point, the regional outline of the local high-definition image to be stitched together is already clear. In the optional example scheme of this invention, the gray value of the outline can be a single outline. As shown in Figure 7, for each module of local high-definition image, the single outline shown in the dashed box is used for gray value recognition.

[0102] In addition, in an optional embodiment of the present invention, a dual-layer contour analysis method is also provided. In this method, the dual-layer contour uses the dashed frame that overlaps with the boundary of the local high-definition image shown in Figure 7 as the first layer contour, and the second layer contour is formed by the four sides separated by a preset number of pixels. Thus, the corresponding outward extension of the damage direction trend can be further established. For example, it can be understood that the contour grayscale values ​​of the first local high-definition image are respectively composed of the dashed frames shown in Figure 7 and Figure 8 to form the dual-layer contour analysis.

[0103] In step 2012, if the outline grayscale value of the corresponding first local high-definition image shows a continuous break, the region of the first local high-definition image is weighted and magnified to update the coordinates of the first round of aerial photography points to obtain the updated coordinates of the second round of aerial photography points, so that the first local high-definition images taken around the coordinates of the second round of aerial photography points can cover the image content at a preset distance in the direction of the continuous break.

[0104] Regarding the aforementioned concept of continuity breaking, the following definitions and explanations are provided for different situations in conjunction with the embodiments of the present invention.

[0105] Scenario 1:

[0106] When the first local high-definition image belongs to the same type of chip, and it is necessary to capture other local high-definition images of the same type that are independent of the first local high-definition image to obtain the coordinates of multiple aerial shooting points, the "continuous breakthrough" specifically includes:

[0107] Based on the coordinates of the first round of aerial photography points, the boundary contours used to replace the first local high-definition image in the global image are determined. A contour gray value array is constructed using the gray values ​​of the contour pixels. If the difference between consecutive values ​​in the contour gray value array is less than a preset value, and the gray value at the corresponding position of the boundary contour array of other similar local high-definition images differs from the gray value at a preset threshold, then it is determined that the contour gray value array of the corresponding first local high-definition image has a continuous break; wherein, the position of the break is the contour position corresponding to the gray value abnormality.

[0108] The preset value is a relatively small value and can be determined based on the actual situation. If the difference between consecutive values ​​in the outline grayscale value array is less than the preset value, it means that the consecutive values ​​in the outline grayscale value array are similar.

[0109] The preset threshold can be determined according to the actual situation, and no specific limit is made here.

[0110] Scenario 2:

[0111] When the first local high-definition image does not have a class classification and exists as a unique file in each single board material, the continuity breaking specifically includes:

[0112] Based on the coordinates of the first round of aerial photography points, the boundary contours used to replace the first local high-definition image in the global image are determined. A contour gray value array is constructed using the gray values ​​of the contour pixels. If the difference between consecutive values ​​in the contour gray value array is less than a preset value, and the boundary is more than a preset value away from other values ​​in the contour gray value array, then the contour gray value changes of the boundary obtained by further shooting with high-frequency LED light source switching angle supplementary light lack consistency. In this case, it is determined that the contour gray value array of the corresponding first local high-definition image has a continuity break.

[0113] The analysis mode for scenario two can effectively address defects such as scratches on the surface of individual chips on the motherboard and damage caused by accidental cutting during the splitting operation.

[0114] For the present invention, the application scenario of the method is mainly in the field of optical modules. In the corresponding field of optical modules, most chips are based on arrays. For individual chips, testing is usually done during the procurement process. Therefore, the above-mentioned second scenario will not be described as an important and necessary feature in the implementation of the embodiments of the present invention.

[0115] Regarding the weighted magnification operation involved in step 2012 in the above extended implementation, and considering the possible situations in the embodiments of the present invention, a preferred implementation is also provided. If multiple local high-definition images of the same type as the first local high-definition image are in an array relationship, and the contour region of the continuous breaking circle is located on the first boundary contour of the adjacent first local high-definition image and the second local high-definition image in the array, the weighted magnification of the region of the first local high-definition image specifically includes: weighted magnification of the boundary of the first local high-definition image, such that after the relative contours of the adjacent first local high-definition image and the second local high-definition image in the array are captured by the weighted first local high-definition image, the weighted captured first local high-definition image covers the region between the relative first boundary contours of the first local high-definition image and the second local high-definition image. Taking the chip array scene shown in Figure 6 as an example, the similar local high-definition images it exhibits are shown in the dashed boxes in Figure 7. Only the first to fourth local high-definition images are labeled in a representative way. It can be understood that the four dashed boxes of the same type located to the right of the fourth local high-definition image in Figure 7 should be labeled as the fifth to eighth local high-definition images from left to right, respectively. However, for the sake of the simplicity of the attached figure, no corresponding labeling is made.

[0116] Taking the relative contours of the first and second local high-definition images in the figure as an example, if, according to the analysis method of this invention, the contour on the global image used to replace the first local high-definition image shows the results of the above analysis, then the coordinates of the corresponding shooting point of the first local high-definition image are weighted and amplified, so that the contour line of the finally captured first local high-definition image coincides with the contour line of the adjacent second local high-definition image on the relative boundary contour, as shown in Figure 8. Here, the right contour line of the contour line used for subsequent stitching of the first local high-definition image and located on the global image coincides with the left contour line of the second local high-definition image. Thus, the problem of incomplete extraction of defects as shown in Figure 1 can be effectively addressed. It is still necessary to further explain that, in the example of the implementation scheme of this invention, since the corresponding continuity break occurs between adjacent local high-definition images of the same type, the second local high-definition image, which is adjacent to the first local high-definition image, has essentially largely connected the incompletely extracted defect pattern portion through the operation results shown in Figure 8. Furthermore, according to the implementation of the present invention, if the second local high-definition image has the same continuity breaking situation as the first local high-definition image on the global image, then by further applying the process of steps 2011-2012 of the present invention, the overall characteristics of the defect can be further improved.

[0117] In conjunction with the embodiments of the present invention, in addition to the above-described cases one and two, there is also a case three. In case three, if multiple local high-definition images of the same type as the first local high-definition image are discretely distributed (essentially, the discrete distribution in case three is similar to the single-entity existence in case two), the weighted magnification of the region of the first local high-definition image specifically includes:

[0118] The boundary of the first local high-definition image is weighted and magnified so that the boundary of the first local high-definition image is adjacent to the boundary of another type of local high-definition image; or, the boundary of the first local high-definition image is weighted and magnified so that the boundary of the first local high-definition image reaches a preset maximum boundary value.

[0119] The preset maximum boundary value is usually determined based on factors such as the camera resolution, the size of the lens capture area, the theoretical maximum chip scratch value, and the contour distance between the captured first local high-definition image and another type of local high-definition image in the surrounding area.

[0120] It should be noted that Case 3 here also applies to Case 1. The boundary contour of the corresponding continuous broken circle is not the relative boundary contour of the adjacent local high-definition image. Taking Figure 7 as an example, when the corresponding continuous broken circle is located above the boundary contour marked by the dashed line of the first local high-definition image, directly applying the extension method after step 2012 (i.e., after being weighted and captured by the first local high-definition image, the weighted and captured first local high-definition image covers the area between the first boundary contour of the first local high-definition image and the second local high-definition image) cannot effectively cover this problem. In this regard, the present invention also provides a more lightweight method, that is, taking the boundary of the continuous broken circle in the first local high-definition image as the edge, further capturing a mirror image of the first local high-definition image that acts as a mirror relative to the boundary, as shown in Figure 9. The image located above the dashed frame of the first local high-definition image is the so-called mirror image. The reason for this is that in the actual implementation process, the continuity of the outline of each type of local high-definition image can be naturally completed after the coordinates of the shooting point are determined in step 201. Therefore, the acquisition range and focus parameters of the corresponding local high-definition image are still retained. At this time, they are directly used to complete the acquisition of the mirror image of the first local high-definition image. The cost-effectiveness of adjustment is the highest, which is equivalent to just adding a point coordinate at the end of the shooting point coordinate queue.

[0121] Combining the double-layer contour analysis method described in step 2011 above, by analyzing the offset of the second layer contour corresponding to the continuous broken area in the first layer contour, the mirror image rotation angle of the dashed frame shown in Figure 9 is generated, thereby making the content of the damaged extension area obtained by the image more accurate and richer.

[0122] As shown in Figure 10, the rotation method typically uses one corner of the edge containing the continuous breaking area as the rotation point. Then, the rotation angle is obtained by using the offset of the second contour corresponding to the continuous breaking area in the first contour in the calculated double-layer contour as a parallel guide. The rotation point is the corner point on the left side of the extended guide calculated with the corresponding offset.

[0123] In the implementation of this invention, in addition to the three situations and corresponding solutions provided above, this invention also provides a fourth situation and a corresponding solution derived therefrom. When taking fixed-shot images of the detection object on the platform to obtain a global image, and locating the coordinates of each shooting point from the global image, as shown in Figure 11, the method further includes:

[0124] In step 301, if it is confirmed that the continuous break-through includes at least the first local high-definition image, the second local high-definition image, and the third local high-definition image, and the three are located in the array of the same type of chip object with discrete distribution, the low-magnification microscopic imaging module of the fixed-shot vision subsystem is controlled to take fixed shots of the array area composed of the first local high-definition image on the object platform to obtain array supplementary images.

[0125] In step 302, based on the array re-captured images, the outline bounding box images of the regions between the local high-definition images in the array where each first local high-definition image is located are obtained.

[0126] As shown in Figure 12, the white background located below each local high-definition image represents the array's re-captured image. The resolution of the image in the white background area is higher than that of the global image. Figure 12 is only for illustration. For ease of understanding and increased recognizability, its content is only expressed with a white background. In actual implementation, its content will be presented with a higher resolution than the global image.

[0127] In step 303, with the global image as the background, multiple local high-definition images are superimposed on the global image, and the outline image is used to cover the background between the local high-definition images in the array area, thereby stitching them together into a complete image to be detected.

[0128] As shown in Figure 12, it can be understood that all the contents contained in the outer frame of the white substrate constitute the array re-enhanced image, which has been overlaid on the global image. The eight local high-definition images on the array re-enhanced image of the white substrate shown in Figure 12 are the effect of further overlaying the array re-enhanced image. Thus, it ensures the display of defects in the area between the corresponding local high-definition images, the clarity of the local high-definition images, and the highlighting of key information in the global image.

[0129] Example 3:

[0130] In this embodiment of the invention, compared with the method proposed in Embodiment 2, this embodiment of the invention can be understood as the application of the method of Embodiment 2 in a specific architectural environment. Therefore, the description of this embodiment of the invention will focus on the description of the framework process, while the implementation of the specific steps involving the innovative content of this invention can be referred to the relevant technical details and extended content in Embodiment 2, which will not be repeated in this embodiment of the invention.

[0131] In this embodiment of the invention, an AI-driven defect detection algorithm is introduced: a two-level AI algorithm architecture is adopted. The first level uses a target detection model (or segmentation model) to segment different effective detection areas according to the preset categories of the detection objects (such as lasers, detectors, driver ICs, etc.). The second level uses an image segmentation model to perform defect identification and defect category determination (such as cracks, foreign objects, missing materials, etc.) on each effective detection area.

[0132] In this embodiment of the invention, a client / server (C / S) architecture for computing resource configuration is introduced: a user interface (UI) is deployed on the client side, allowing operators to set parameters, control detection, and view results, supporting remote control; the image detection algorithm module (including an AI model) is deployed on the server side, equipped with a high-performance GPU card to perform image processing and data storage; one server can serve multiple clients, and the server and clients are connected via high-speed Ethernet to ensure real-time data transmission and response. The C / S architecture enables centralized management and efficient scheduling of computing resources, reducing the hardware configuration requirements of a single device.

[0133] The specific workflow adopted in this invention is as follows:

[0134] The single-item product inspection steps, as shown in Figure 13, include:

[0135] In step 401, global image acquisition is performed by controlling the low-magnification microscopic imaging module of the fixed-shot vision subsystem (adjusting the magnification according to the size of the object being detected) to perform a fixed-shot operation on the object being detected on the platform to acquire a global image.

[0136] In step 402, the aerial photography point location is determined based on the global image obtained in step 401. The aerial photography points that need to be detected locally with high precision are located using an image analysis algorithm, and the X and Y coordinates of each aerial photography point are determined.

[0137] In step 403, altitude information is measured by controlling the altitude measurement subsystem to move to each aerial photography location, measuring and recording the Z-axis altitude coordinates of each location, providing a basis for adjusting the focal length of the aerial photography image.

[0138] In step 404, local aerial imaging is performed. Based on the X / Y coordinates obtained in step 402, the X and Y axes of motion are controlled to move the aerial imaging vision subsystem. Combined with the Z-axis height coordinates obtained in step 403, the focal length of the aerial imaging vision subsystem is adjusted. A aerial imaging operation is performed on each aerial imaging point to obtain local high-definition images.

[0139] In step 405, image stitching is performed (as shown in Figure 6, which is the effect after stitching). After the aerial photography operation is completed, the multiple local high-definition images obtained in step 404 are stitched together into a complete image to be detected, using the global image obtained in step 401 as the background.

[0140] In step 406, 3D measurement calculation is performed. Based on preset measurement rules (such as height, distance, angle, area, volume, etc.) on the image to be measured after being stitched together in step 405, the 3D measurement is calculated and output to complete the 3D measurement.

[0141] In step 407, image data is transmitted via network, transferring the global image from step 401 and the stitched detection image from step 405 from the client to the server on which the AI ​​detection model is deployed.

[0142] In step 408, effective region cropping is performed. The server uses an AI object detection model (or segmentation model) to perform region cropping on the stitched detection image from step 405 based on the category of the detected object, removing irrelevant background information and retaining the effective detection region image.

[0143] In step 409, defect segmentation and classification are performed. The server uses an AI image segmentation model to perform defect segmentation on each valid detection region image after cropping in step 408, and determines the defect category based on defect features to generate defect detection results.

[0144] In step 410, the test results are fed back. The server sends the 3D measurement data from step 406 and the defect detection results from step 409 back to the local client. The client displays and archives the test results, completing the single-product test.

[0145] In conjunction with the relevant structure in Embodiment 1, this embodiment of the invention also provides a batch product testing step, as shown in Figure 14, including:

[0146] In step 501, the batch raw material is transferred. The intelligent handling robot and the robotic arm work together to transfer the batch of raw material to be tested (i.e., a batch of products to be tested) to the loading buffer unit of the testing equipment.

[0147] In step 502, the raw material is automatically fed, and the detection equipment automatically transfers the batch of raw material to be detected from the feeding buffer unit to the material detection unit through its own conveying mechanism.

[0148] In step 503, batch testing and data archiving are performed. The equipment performs testing operations on each product to be tested in the material testing unit in sequence according to the above "single product testing steps". Simultaneously, a production defect mapping diagram (including defect location, category, size and other information) is generated for each product, and the mapping diagram and measurement data are archived and stored.

[0149] In step 504, the clinker is automatically fed into the material feeding unit. The testing equipment automatically transfers the batch of clinker (i.e., a batch of tested products) that has been tested by the material testing unit to the feeding buffer unit through its own conveying mechanism.

[0150] In step 505, the batch of clinker is transferred. The intelligent handling robot and the robotic arm work together to take the batch of clinker that has been inspected out from the feeding buffer unit and transfer it to the sorting equipment.

[0151] In step 506, during the batch sorting operation, the sorting equipment reads the archived defect mapping diagram and performs automatic sorting of the inspected products within the batch according to preset sorting rules (such as defective products, good products, re-inspected products, etc.).

[0152] Compared with the prior art, the present invention has the following advantages:

[0153] By integrating multiple functional modules such as the aerial photography vision subsystem, the fixed-photography vision subsystem, and the altitude measurement subsystem, the system achieves simultaneous execution of 3D measurement and AOI inspection, shortening the inspection cycle, improving production efficiency, and reducing equipment procurement, operation and maintenance, and site occupancy costs.

[0154] By using a high-magnification microscopic imaging module to achieve high-precision detection of localized minute defects, and a low-magnification microscopic imaging module to achieve global large-field coverage of the detected object, the problem of balancing detection accuracy and field of view has been successfully solved, ensuring the comprehensiveness and accuracy of the detection.

[0155] Through a two-level AI algorithm architecture, intelligent detection and defect identification of different areas are achieved, improving the generalization and adaptability of the algorithm. It can automatically adapt to changes in the color, dimensional tolerance, surface texture and other characteristics of the detected object without additional programming.

[0156] By adopting a C / S architecture for computing resource configuration, the hardware configuration requirements of a single device are reduced by centrally managing and scheduling computing resources on the server side, which significantly reduces the cost of batch deployment of devices, while improving the scalability and maintainability of the devices.

[0157] Through the open mechanical structure and the collaborative design of "intelligent handling robot + robotic arm", the automatic feeding of products to be tested and the automatic unloading of tested products are realized, which improves the flexibility and automation level of the production line and adapts to the flexible production needs of multiple varieties and small batches.

[0158] Example 4:

[0159] Figure 15 shows a schematic diagram of the architecture of an AOI detection device according to an embodiment of the present invention. The AOI detection device in this embodiment includes one or more processors 21 and a memory 22. Figure 15 illustrates one processor 21 as an example.

[0160] The processor 21 and the memory 22 can be connected via a bus or other means. Figure 4 shows an example of a connection via a bus.

[0161] The memory 22, as a non-volatile computer-readable storage medium used in AOI inspection methods and apparatus, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the AOI inspection method in Embodiment 2. The processor 21 executes the AOI inspection method by running the non-volatile software program and instructions stored in the memory 22.

[0162] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0163] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the method for AOI detection described in Embodiment 1 above, for example, performing the steps shown in Figures 4, 5, and 11 above.

[0164] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0165] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for AOI detection, characterized in that, include: The low-magnification microscopic imaging module of the fixed-shot vision subsystem acquires a global image of the object to be detected on the platform, and locates the coordinates of each shooting point from the global image; when acquiring a global image of the object to be detected on the platform and locating the coordinates of each shooting point from the global image, the method further includes: based on the coordinates of the shooting points in the first round, performing contour grayscale value recognition on the region of the local high-definition image to be stitched in the global image. If the grayscale value of the outline corresponding to the first local high-definition image shows a continuous break, the region of the first local high-definition image is weighted and magnified to update the coordinates of the first round of aerial photography points to obtain the updated coordinates of the second round of aerial photography points. This ensures that the first local high-definition images captured around the coordinates of the second round of aerial photography points can cover the image content at a preset distance in the direction of the continuous break. Based on the coordinates of each aerial photography point, the motion axis is controlled to move the aerial photography vision subsystem to perform aerial photography at each aerial photography point to acquire multiple local high-definition images. Using the global image as the background, the acquired multiple local high-definition images are stitched together to form a complete image to be detected. Based on the category of the object to be detected, the region of the image to be detected is cropped to obtain images of each region. The defect category of each region image is determined based on the defect features to generate a defect detection result.

2. The method for AOI detection according to claim 1, characterized in that, When the first local high-definition image belongs to a type of chip, and it is necessary to capture other local high-definition images of the same type that are independent of the first local high-definition image to obtain multiple aerial shooting point coordinates, the continuity breaking specifically includes: determining the boundary contour in the global image used to replace the first local high-definition image based on the aerial shooting point coordinates of the first round, constructing a contour gray value array with the gray values ​​of the contour pixels, and if the difference between consecutive values ​​in the contour gray value array is less than a preset value, while the gray value at the corresponding position of the boundary contour array of other local high-definition images of the same type differs from the gray value at a preset threshold, then it is determined that the contour gray value array of the corresponding first local high-definition image has a continuity breaking; wherein, the breaking position is the contour position corresponding to the gray value abnormality.

3. The method for AOI detection according to claim 1, characterized in that, If multiple local high-definition images of the same type as the first local high-definition image are arranged in an array, and the contour region of the continuous breaking circle is located on the first boundary contour of the adjacent first local high-definition image and the second local high-definition image in the array, the weighted amplification of the region of the first local high-definition image specifically includes: weighted amplification of the boundary of the first local high-definition image, so that after the relative contours of the adjacent first local high-definition image and the second local high-definition image in the array are captured by the weighted first local high-definition image, the weighted captured first local high-definition image covers the region between the first boundary contours of the first local high-definition image and the second local high-definition image.

4. The method for AOI detection according to claim 1, characterized in that, If multiple local high-definition images of the same type as the first local high-definition image are discretely distributed, the weighted amplification of the region of the first local high-definition image specifically includes: weighted amplification of the boundary of the first local high-definition image so that the boundary of the first local high-definition image is adjacent to the boundary of another type of local high-definition image; or, weighted amplification of the boundary of the first local high-definition image so that the boundary of the first local high-definition image reaches a preset maximum boundary value.

5. The method for AOI detection according to claim 1, characterized in that, When acquiring a global image of the object to be detected on the platform and locating the coordinates of each camera point from the global image, the method further includes: if it is confirmed that at least three local high-definition images—a first local high-definition image, a second local high-definition image, and a third local high-definition image—have occurred in a continuous loop, and these three images are discretely distributed in the array of images of the same type of chip object, the method controls the low-magnification microscopic imaging module of the fixed-shot vision subsystem to acquire an array supplementary image of the array area formed by the first local high-definition image on the platform; based on the array supplementary image, the method acquires the outline frame image of the region between each local high-definition image in the array containing the first local high-definition image; and with the global image as the background, multiple local high-definition images are superimposed on the global image, and the outline frame image is used to cover the background between each local high-definition image in the array area, thereby stitching together a complete image to be detected.

6. A device for AOI inspection, characterized in that, To implement the method described in any one of claims 1-5, the apparatus includes a flying camera vision subsystem, a fixed-shot vision subsystem, an altitude measurement subsystem, an X-axis motion mechanism, a Y-axis motion mechanism, and a Z-axis motion mechanism. Specifically: the flying camera vision subsystem achieves high-precision detection of localized minute defects through flying camera mosaicking; the fixed-shot vision subsystem achieves global large-field-of-view coverage detection of the object to be detected through fixed-shot imaging; the altitude measurement subsystem achieves accurate measurement of the height of the object to be measured through physical or optical principles, and selects one or more altitude measurement schemes such as contact altimetry, laser triangulation, time-of-flight (ToF), ultrasonic altimetry, and visual stereoscopic altimetry according to actual accuracy and / or range requirements; the X-axis motion mechanism, Y-axis motion mechanism, and Z-axis motion mechanism are used to drive the three-dimensional movement of the vision subsystem, and select one of a lead screw, a linear motor, and an air-bearing motion platform according to accuracy requirements.

7. The apparatus for AOI inspection according to claim 6, characterized in that, The aerial photography vision subsystem also includes a high-magnification microscopic imaging module, a telecentric lens, a high-frequency LED light source, and an aerial photography controller. Specifically: the high-magnification microscopic imaging module uses a global shutter industrial camera as a vision sensor to achieve high-precision detection of localized minute defects; the telecentric lens works in conjunction with the high-magnification microscopic imaging module to achieve pixel-level accuracy; the high-frequency LED light source is synchronized with the camera exposure to avoid uneven image brightness caused by light source flicker during high-speed movement; and the aerial photography controller is used to control the shutter, focal length, and aperture parameters of the high-magnification microscopic imaging module, and to achieve high-speed strobe control and synchronous capture.

8. The apparatus for AOI inspection according to claim 6, characterized in that, The fixed-shot vision subsystem also includes a low-magnification imaging module, a telecentric lens, and a coaxial ring combined light source. Specifically: the low-magnification imaging module uses a high-resolution area array camera to achieve a large global field of view coverage of the object to be detected; the telecentric lens supports adjustable magnification and works with the low-magnification imaging module to achieve pixel-level accuracy; the coaxial ring combined light source, through optical design, combines the vertical uniform illumination of coaxial light with the multi-angle edge enhancement of ring light, and selects different colors of coaxial light and ring light to achieve accurate detection in highly reflective and complex defect scenarios.

9. The apparatus for AOI inspection according to claim 6, characterized in that, The device also includes a feeding buffer unit, a discharging buffer unit, a material detection unit, and a shock-absorbing platform. Specifically, the feeding buffer unit and the discharging buffer unit are used to buffer the material to be tested and the material that has already been tested; the shock-absorbing platform adopts a pneumatic support structure to achieve accurate positioning and stable placement of the product, and reduce vibration interference during the testing process.

Images