Appearance inspection method, 3D appearance inspection device, and 3D appearance inspection system
The three-dimensional appearance inspection method integrates AI inference and rule-based size detection to address uneven shape determination and false detections, enhancing defect detection accuracy and efficiency in manufacturing.
Patent Information
- Application Number
- JP2024037734
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-03-12
- Publication Date
- 2025-09-26
AI Technical Summary
Existing visual inspection methods struggle with determining uneven shapes and require extensive expert tuning, leading to low robustness and frequent false detections, especially in manufacturing environments with low defective product rates.
A three-dimensional appearance inspection method combining AI inference for two-dimensional defect judgment with rule-based size detection on both two-dimensional and three-dimensional shape information to accurately identify defects.
Enhances defect detection accuracy by leveraging AI for initial rough detection and rule-based size detection on three-dimensional shape information, reducing false positives and improving overall inspection efficiency.
Smart Images

Figure 2025139029000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to an appearance inspection method, a three-dimensional appearance inspection device, and a three-dimensional appearance inspection system for detecting defective products from an inspection object. [Background technology]
[0002] Traditionally, in the fields of manufacturing and quality control, visual inspection, which checks for scratches and chips on the surface of products on production lines, has relied on a large number of manual tasks and is a skilled job requiring a high level of skill.In recent years, as the working-age population has declined and the labor shortage has become more serious, automation of visual inspection has been promoted by applying artificial intelligence (AI) technologies such as deep learning and rule-based expert systems, which process images of the object to be inspected and measure and evaluate quantitative parameters such as color, shape, texture, and surface characteristics (see, for example, Patent Documents 1 to 4). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-190911 [Patent Document 2] Japanese Patent Application Publication No. 2019-204321 [Patent Document 3] Japanese Patent Publication No. 2021-174456 [Patent Document 4] Patent No. 7416071 [Patent Document 5] Patent No. 7353644 [Patent Document 6] Patent No. 7385919 Summary of the Invention [Problem to be solved by the invention]
[0004] Although visual inspection using an image camera has been common up until now, there is a problem in that it is not possible to determine the uneven shape because the inspection is based on a two-dimensional image.
[0005] In the fields of manufacturing and quality control, automation of visual inspections is being promoted using AI and rules. However, in visual inspections that are automated using AI or rules, learning is carried out through the judgment of experts, with the mechanical pass / fail judgment results being reviewed after the fact, which creates the problem that it is difficult to put the system into operation until a certain level of learning has progressed.
[0006] Rule-based systems require extremely delicate tuning by experts, but they still suffer from low robustness and frequent over-detection and false detection.
[0007] Furthermore, on manufacturing lines with a low rate of defective products, there are few samples of defective products to learn from, which means that the accuracy of the AI does not improve, resulting in frequent overdetection and false detection of defective products.
[0008] On the other hand, AI excels at detecting unknown defects and identifying defect types. Rule-based systems excel at quantifying acquired data. Image cameras excel at capturing images at high speed. 3D sensors excel at 3D quantification.
[0009] Therefore, the present invention has been devised in consideration of the problems and advantages of each of the conventional technologies described above, and its purpose is to provide an appearance inspection method, a three-dimensional appearance inspection device, and a three-dimensional appearance inspection system that can efficiently, reliably, and quantitatively detect defective products from an object to be inspected in a short period of time.
[0010] Other objects of the present invention and specific advantages obtained by the present invention will become more apparent from the following description of the embodiments. [Means for solving the problem]
[0011] The present invention is an appearance inspection method comprising a two-dimensional image acquisition step of acquiring a two-dimensional image of an inspection area of an object to be inspected, a first defect judgment step of performing defect judgment using AI inference on the two-dimensional image acquired in the two-dimensional image acquisition step, a three-dimensional shape information acquisition step of acquiring three-dimensional shape information of a defect candidate area in the inspection area of the object to be inspected that corresponds to a defect area in the two-dimensional image that has been judged to be defective in the first defect judgment step, a size detection step of performing rule-based size detection on the three-dimensional shape information of the defect candidate area to obtain size information, and a final judgment step of performing defect judgment on the defect candidate area using the size information obtained in the size detection step, wherein the defect candidate area judged to be defective in the final judgment step is determined to be a defective area.
[0012] In the appearance inspection method of the present invention, the size detection process can include a first size detection process that performs rule-based size detection on a defective area of the two-dimensional image that has been determined to be defective in the first defect determination process to obtain size information in a two-dimensional plane, and a second size detection process that performs rule-based size detection on three-dimensional shape information of the defect candidate area to obtain size information in the height direction.
[0013] In the appearance inspection method according to the present invention, the size detection step may acquire size information including the size and height of the defect candidate region in a two-dimensional plane.
[0014] The present invention also provides a three-dimensional appearance inspection apparatus, comprising: two-dimensional image acquisition means for acquiring a two-dimensional image of an inspection area of an inspection object; three-dimensional shape information acquisition means for acquiring three-dimensional shape information of a defect candidate area in the inspection area of the inspection object; relative movement means for relatively moving the inspection object and the three-dimensional shape information acquisition means; and information processing means for controlling operations of the two-dimensional image acquisition means, the three-dimensional shape information acquisition means, and the relative movement means, and for detecting a defect area in the inspection area of the inspection object using the two-dimensional image acquired by the two-dimensional image acquisition means and the three-dimensional shape information acquired by the three-dimensional shape information acquisition means, The means is characterized in that it drives the two-dimensional image acquisition means to acquire a two-dimensional image of the inspection area of the inspection object, performs a defect judgment using AI inference on the acquired two-dimensional image, and designates a defect area of the two-dimensional image that is judged to be defective as a defect candidate area, drives the relative movement means and the three-dimensional shape information acquisition means to acquire three-dimensional shape information of the defect candidate area, performs rule-based size detection on the acquired three-dimensional shape information of the defect candidate area to obtain size information, uses the size information to make a final defect judgment on the defect candidate area, and designates the defect candidate area that is judged to be defective in the final defect judgment as a defect area.
[0015] In the three-dimensional appearance inspection device of the present invention, the information processing means can perform rule-based size detection on defective areas of the two-dimensional image that have been determined to be defective by the defect determination using AI inference, to obtain size information on a two-dimensional plane, and can also perform rule-based size detection on three-dimensional shape information of the defect candidate area, to obtain size information in the height direction.
[0016] Furthermore, in the three-dimensional appearance inspection apparatus according to the present invention, the information processing means can perform rule-based size detection on the three-dimensional shape information of the defect candidate area, and obtain size information including the size and height of the defect candidate area in a two-dimensional plane.
[0017] In the three-dimensional appearance inspection apparatus according to the present invention, the relative movement means may be an XYZ stage that moves the inspection object or the three-dimensional shape information acquisition means in three-dimensional directions.
[0018] Furthermore, in the three-dimensional appearance inspection apparatus according to the present invention, the relative movement means may be an articulated robot that moves the three-dimensional shape information acquisition means in three-dimensional directions.
[0019] Furthermore, in the three-dimensional appearance inspection apparatus according to the present invention, the three-dimensional shape information acquisition means may be an optical three-dimensional shape measurement apparatus that acquires three-dimensional shape information by scanning a defect candidate area within the inspection area of the object to be inspected with measurement light.
[0020] The present invention is a 3D visual inspection system that is installed on a transport line that transports inspection objects by a transport device, and is characterized by comprising: an AI visual inspection device that performs a qualitative inspection in which the inspection area of the inspection object is imaged to obtain a 2D image, and the acquired 2D image is subjected to a defect judgment using AI inference, and a rule-based size detection is performed on a defect area of the 2D image that is judged to be defective to obtain size information; and a 3D visual inspection device that, for an inspection object judged to be defective by the AI visual inspection device, acquires 3D shape information of a defect candidate area within the inspection area of the inspection object that corresponds to the defect area of the 2D image, and performs rule-based size detection on the acquired 3D shape information of the defect candidate area to obtain height size information, and uses the size information obtained in the qualitative inspection process and the height size information to make a final defect judgment of the defect candidate area, and a quantitative inspection in which the defect candidate area judged to be defective in the final judgment process is regarded as a defect area.
[0021] In the three-dimensional appearance inspection system of the present invention, the three-dimensional appearance inspection device may be equipped with an optical three-dimensional shape measurement means that scans defect candidate areas within the inspection area of the inspection object with measurement light to obtain three-dimensional shape information. [Effects of the Invention]
[0022] In the present invention, even if a two-dimensional image of the inspection area of an object to be inspected is acquired, and a defect judgment based on AI inference is performed on the acquired two-dimensional image, and there is a large amount of overdetection of defective products including defective areas in the inspection area of the object to be inspected, three-dimensional shape information of defect candidate areas in the inspection area of the object to be inspected that correspond to the defect areas in the two-dimensional image that are judged to be defective is acquired, rule-based size detection is performed on the acquired three-dimensional shape information of the defect candidate areas, and the obtained size information can be used to make a final defect judgment of the defect candidate areas with high accuracy, thereby improving the yield of appearance defect detection of objects to be inspected.
[0023] That is, in the present invention, rough detection of visual defects of an inspection object is performed by defect judgment through AI inference using a two-dimensional image, and for inspection objects with visual defects that are judged to be defective in this rough detection, rule-based size detection is performed using three-dimensional shape information of a defect candidate area in the inspection area of the inspection object that corresponds to the defect area in the two-dimensional image to detect size information including height direction information, and visual defects are re-detected, that is, a final defect judgment is made on the defect candidate area, thereby making it possible to efficiently and reliably detect defective products from inspection objects.
[0024] Therefore, according to the present invention, it is possible to provide a visual inspection method, a three-dimensional visual inspection device, and a three-dimensional visual inspection system that can efficiently and reliably detect defective products from an object to be inspected. [Brief explanation of the drawings]
[0025] [Figure 1] FIG. 1 is an external view of a three-dimensional inspection device to which the present invention is applied. [Figure 2] FIG. 2 is a block diagram showing the configuration of the above three-dimensional inspection device. [Figure 3] FIG. 3 is a flowchart showing the procedure of the three-dimensional inspection process executed by the three-dimensional inspection device. [Figure 4]Figure 4 (A), (B), and (C) show various defect areas detected on a two-dimensional image taken by the above-mentioned three-dimensional inspection device. (A) shows a defect area that has occurred on the side edge of a circuit board on which components are mounted, (B) shows a defect area that has occurred on the circuit pattern of the above-mentioned circuit board, and (C) shows a defective soldering area that has occurred between the component mounted on the circuit board and the circuit pattern. [Figure 5] (A), (B), and (C) in Figure 5 are figures showing images displayed on a display device connected to the information processing unit, showing various judgment states of defective areas during the process of performing an appearance inspection process on the printed circuit board using the above-mentioned three-dimensional inspection device. (A) shows an image of a defective area judged as a soldering defect area on the circuit board after the AI inference defect judgment processing unit of the information processing unit performs an AI inference defect judgment on the two-dimensional image of the circuit pattern area. (B) shows an image of a defective area in which the rule-based defect judgment processing unit of the information processing unit performs XY size detection of the soldering defect area in the two-dimensional image of the circuit pattern area and quantifies the defective area. (C) shows an image of a defective area in which the rule-based size detection is performed using three-dimensional shape information acquired by the three-dimensional shape measurement unit to obtain size information (Z) and quantify the defective area in three dimensions for a soldering defect area in the inspection area of the object to be inspected 5 corresponding to the defective area in the two-dimensional image judged to be a defective area by the rule-based defect judgment processing. [Figure 6] Figures 6(A), (B), and (C) show various defect areas on the machined surface of an aluminum cast part detected on a two-dimensional image taken by the above-mentioned three-dimensional inspection device. (A) shows a defect area occurring on the side edge of the aluminum part, (B) shows a defect area occurring on the upper surface of the machined surface of the above-mentioned aluminum cast part, and (C) shows a defect area occurring on the periphery of the hole taper part of the above-mentioned aluminum cast part. [Figure 7]7A, 7B, and 7C are diagrams showing images displayed on a display device connected to the information processing unit, showing various judgment states of defect areas during the process of performing an appearance inspection process on the machined surface of the aluminum cast part using the above-mentioned 3D inspection device. (A) shows an image of a defect area judged to be a defect area caused by a dent on the side edge of the machined surface of the aluminum cast part, after the AI inference defect judgment processing unit of the information processing unit has performed defect judgment using AI inference on the 2D image of the machined surface of the aluminum cast part. (B) shows an image of a defect area in which the rule-based defect judgment processing unit of the information processing unit has performed XY size detection on the defect area in the 2D image of the machined surface of the aluminum cast part and quantified the defect area. (C) shows an image of a defect area in which the defect area is quantified in three dimensions, after rule-based size detection is performed using the 3D shape information acquired by the 3D shape measurement unit to obtain size information (Z) for a defect candidate area in the inspection area of the inspection object 5 corresponding to the defect area in the 2D image that was judged to be a defect area by the rule-based defect judgment processing. [Figure 8] FIG. 8 is a flowchart showing another procedure of the three-dimensional inspection process executed by the three-dimensional inspection device. [Figure 9] FIG. 9 is a flowchart showing still another procedure of the three-dimensional inspection process executed by the three-dimensional inspection device. [Figure 10] FIG. 10 is an external view of another example of the configuration of a three-dimensional inspection device to which the present invention is applied. [Figure 11] FIG. 11 is a block diagram showing the configuration of the three-dimensional inspection apparatus according to the other configuration example. [Figure 12] FIG. 12 is a block diagram showing an example of the configuration of a three-dimensional inspection system according to the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Common components will be described by using common reference numerals in the drawings. Furthermore, the present invention is not limited to the following examples, and can be modified as desired without departing from the spirit of the present invention.
[0027] The present invention is carried out by a three-dimensional appearance inspection apparatus 100 having a configuration as shown in Figures 1 and 2. Figure 1 is an external view of the three-dimensional appearance inspection apparatus 100 to which the present invention is applied, and Figure 2 is a block diagram showing the configuration of the three-dimensional appearance inspection apparatus 100.
[0028] As shown in Figures 1 and 2, this three-dimensional appearance inspection device 100 comprises an illumination unit 10 that illuminates the inspection area of the inspection object 5 placed on the measurement table 1, a two-dimensional imaging unit 20 that images the inspection area of the inspection object 5, a three-dimensional shape measurement unit 30 that measures the three-dimensional shape of a candidate defect area within the inspection area of the inspection object 5, an XYZ stage 40 that moves the inspection object 5 and the three-dimensional shape measurement unit 30 relatively, and an information processing unit 50 that controls the operations of these units and performs information processing to detect a defect area in the inspection area of the inspection object 5 using the two-dimensional image acquired by the two-dimensional imaging unit 20 and the three-dimensional shape information acquired by the three-dimensional shape measurement unit 30.
[0029] The lighting unit 10 is composed of a lighting device using an LED lamp or the like, and is turned on when lighting power is supplied from the lighting power supply device 11, thereby illuminating the inspection area of the inspection object 5 placed on the measurement table 1.
[0030] The two-dimensional imaging unit 20 is a two-dimensional camera such as a CCD camera, and functions as a two-dimensional image acquisition means for acquiring a two-dimensional image of the inspection area of the inspection object.
[0031] The three-dimensional shape measuring unit 30 is an optical three-dimensional shape measuring device that performs three-dimensional shape measurement by directing measurement light at the inspection object 5 and detecting the light reflected by the inspection object 5, and functions as a three-dimensional shape information acquisition means that acquires three-dimensional shape information of defect candidate areas within the inspection area of the inspection object 5.
[0032] Here, the three-dimensional shape measurement unit 30 can be any of a variety of optical three-dimensional shape measurement devices, including those that use multiple cameras to photograph an object from multiple angles, extract feature points and patterns on the object's surface based on the images obtained, and measure the three-dimensional shape from the positional information of those points; triangulation-based systems that measure the three-dimensional shape of an object by irradiating the object's surface with laser light, detecting the reflected pattern with a sensor, and using triangulation to calculate the position of each point on the object's surface with a computer; and laser scanning-based systems that scan the object's surface with laser light, obtain information such as the angle of reflection and the position of reception of the light by scanning the object's surface with the laser light, and receive the laser light reflected by the object's surface with a sensor or camera, and then calculate information such as the position and height of each point on the object's surface from the angle of reflection and the position of reception of the light using a computer to reconstruct the three-dimensional shape of the object's surface, thereby measuring the three-dimensional shape and dimensions of the object.
[0033] The three-dimensional shape measuring unit 30 in this three-dimensional appearance inspection device 100 is equipped with an optical comb three-dimensional shape sensor 33 to which reference light and measurement light using an optical comb that are coherent with each other are input from an optical comb light source 31 via an interferometer 32, and the optical comb three-dimensional shape measuring device functions as a three-dimensional shape information acquisition means that acquires three-dimensional shape information of a defect candidate area within the inspection area of the inspection object 5 by irradiating the inspection object 5 with measurement light using the optical comb and scanning the inspection object 5 with the optical comb, for example, an optical three-dimensional shape measuring device (see Patent Documents 5 and 6) previously proposed in the present application that measures the three-dimensional shape of an object non-contact by scanning the measurement light irradiated onto the measurement object from an optical comb rangefinder is adopted for the three-dimensional shape measuring unit 30.
[0034] The optical comb three-dimensional shape sensor 33 operates when sensor power is supplied from the sensor power supply device 33, scans the inspection object 5 with measurement light using the optical comb, and obtains an interference light signal by detecting the interference light between the measurement light reflected and returned by the inspection object 5 and the reference light using the optical comb with a sensor, and outputs a three-dimensional shape measurement signal including three-dimensional shape information of the defect candidate area within the inspection area of the inspection object 5 via the signal processing unit 34.
[0035] The XYZ stage 40 in this three-dimensional appearance inspection device 100 is equipped with an X-axis direction moving section 43X, a Y-axis direction moving section 43Y, and a Z-axis direction moving section 43Z that move along three-axis (X, Y, Z) direction running paths 42X, 42Y, 42Z provided on a base 41, and the running path 42Z is provided on the Y-axis direction moving section 43Y that moves along the running path 42Y.
[0036] The measurement table 1 on which the inspection object 5 is placed is provided on the X-axis direction moving section 43X, and the illumination section 10, two-dimensional imaging section 20, and three-dimensional shape measuring section 30 are arranged on the Z-axis direction moving section 43Z.
[0037] This XYZ stage 44 moves the measurement table 1 provided on the X-axis moving section 43X together with the inspection object 5 placed on the measurement table 1 in the X-axis direction along the running path 42X, and can also move the illumination section 10, the two-dimensional imaging section 20, and the three-dimensional shape measuring section 30 arranged on the Z-axis moving section 43Z in the axial direction along the running path 42Y and in the Z-axis direction along the running path 42Z, and functions as a relative moving means for moving the inspection object 5 placed on the measurement table 1 relative to the three-dimensional shape measuring section 30, i.e., the three-dimensional shape information acquisition means.
[0038] The information processing unit 50 in the three-dimensional appearance inspection device 100 is composed of a control personal computer (PC) having functions such as an AI inference defect judgment processing unit 51, a rule-based judgment processing unit 52, an optical comb control unit 53, an integrated processing unit 54, and an XYZ stage control unit 55.
[0039] This information processing unit 50 has the AI inference failure judgment processing unit 51, rule-based judgment processing unit 52, optical comb control unit 53, and XYZ stage control unit 55 internally connected to an integrated processing unit 54, and is equipped with a camera port 56 internally connected to the AI inference failure judgment processing unit 51 and rule-based judgment processing unit 52, an optical comb port 57 internally connected to the rule-based judgment processing unit 52 and optical comb control unit 53, and a GPIO port 58 internally connected to the integrated processing unit 54 and XYZ stage control unit 55.
[0040] In this information processing unit 50, the AI inference failure judgment processing unit 51 and the rule-based judgment processing unit 52 are connected to the external imaging unit 20 via the camera port 56, the rule-based judgment processing unit 52 is connected to the external signal processing unit 34 via the optical comb port 57, the optical comb control unit 53 is connected to the external optical comb power supply unit 31 and sensor power supply unit 33 via the optical comb port 56, and further, the integrated processing unit 54 is connected to the external lighting power supply unit 11 via the GPIO port 58, and the XYZ stage control unit 55 is connected to the external XYZ stage 44 via the GPIO port 58.
[0041] An input device 61 and a display device 62 are connected to the information processing unit 50.
[0042] The information processing unit 50 functions as an information processing means that controls the operation of the illumination unit 10, 2D imaging unit 20, 3D shape measurement unit 30, and XYZ stage 40 using the optical comb control unit 53, integrated processing unit 54, and XYZ stage control unit 55 in accordance with the procedure shown in the flowchart of Figure 3, and detects defective areas in the inspection area of the inspection object 5 using the 2D image acquired by the 2D imaging unit 20 and the 3D shape information acquired by the 3D shape measurement unit 30 using the AI inference defect judgment processing unit 51, rule-based judgment processing unit 52, and integrated processing unit 54.
[0043] In this three-dimensional appearance inspection device 100, the information processing section 50 performs information processing in the three-dimensional appearance inspection mode according to the procedure shown in the flowchart of FIG. 3, thereby implementing the appearance inspection method according to the present invention.
[0044] In the information processing in the three-dimensional appearance inspection mode by the information processing unit 50, the integrated processing unit 54 first performs a two-dimensional image acquisition step S to acquire a two-dimensional image of the inspection area of the inspection object 5. A Processing is performed.
[0045] That is, when an inspection start instruction to start the inspection is input from the input device 61, the information processing unit 50 starts the appearance inspection (step S1), loads the work, i.e., places the inspection object 5 on the measurement table 1 on the XYZ stage 44, turns on the illumination, i.e., supplies illumination power from the illumination power supply device 11 to the illumination unit 10, turns on the illumination unit 10, irradiates the inspection object 5 on the measurement table 1 with illumination light, and images the inspection area of the inspection object 5 on the measurement table 1 irradiated with the illumination light using the 2D imaging unit 20 to obtain a 2D image (step S4).
[0046] Next, the integrated processing unit 54 performs a first defect determination step S in which a defect determination is performed by AI inference on the two-dimensional image acquired in the two-dimensional image acquisition step. B Processing is performed.
[0047] That is, the AI inference defect judgment processing unit 51 performs defect judgment using AI inference on the two-dimensional image information of the inspection area of the inspection object 5 obtained by imaging using the two-dimensional imaging unit 20 in step S4 (step S5).
[0048] Next, the integrated processing unit 54 performs the first defect determination process S B A first size detection step S2 performs rule-based size detection on the defective area of the two-dimensional image determined to be defective in the step S1 to obtain size information (X, Y). C Processing is performed.
[0049] That is, for the defective area in the two-dimensional image that has been determined as NG in step S5, the rule-based determination processing unit 52 performs rule-based size detection to obtain size information (X, Y) (step S6).
[0050] Next, the integration processing unit 54 performs a three-dimensional shape information acquisition step S to acquire three-dimensional shape information of a defect candidate area in the inspection area of the inspection object corresponding to the defective area in the two-dimensional image. D Processing is performed.
[0051] That is, the XYZ stage control unit 55 controls the XYZ stage 44 to move the three-dimensional shape measuring unit 30 to a position where three-dimensional shape information of the defect candidate area in the inspection area of the object to be inspected 5 corresponding to the defective area in the two-dimensional image judged as "NG" in step S5 can be acquired (step S7), and the three-dimensional shape measuring unit 30 acquires three-dimensional shape information of the defect candidate area in the object to be inspected 5 (step S8).
[0052] Furthermore, the integrated processing unit 54 performs the three-dimensional shape information acquisition step S D A second size detection step S2 performs rule-based size detection on the three-dimensional shape information of the defect candidate area obtained in step S1 to obtain size information (Z). E Processing is performed.
[0053] That is, the rule-based judgment processing unit 52 performs rule-based size detection on the three-dimensional shape information of the defect candidate area of the inspection object 5 acquired by the three-dimensional shape measuring unit 30 in step S8 to obtain size information (Z) (step S9).
[0054] Then, the integrated processing unit 54 performs the first size detection process S C The size information (X, Y) obtained in the second size detection step S E The final determination step S is carried out to determine whether the defect candidate area is defective using the size information (Z) obtained in the step S. F This final judgment step S EThe defect candidate area determined to be defective in step 1 is determined to be a defective area.
[0055] That is, it is determined whether or not the Z size detection of step S9 has been performed for all defect candidate areas within the inspection area of the inspection object 5 (step S10). If the determination result is "NO", that is, if there are unprocessed defect candidate areas, the process returns to step S6 and repeats the processes from step S6 to step S10. If the determination result in step S9 is "YES", that is, if detection of size information (X, Y) and size information (Z) for all defect candidate areas has been completed, the size information (Z) is obtained using the size information (X, Y) and size information (Z) for all detected defect candidate areas within the inspection area of the inspection object 5 (step S9).
[0056] Then, in the final judgment of step S11, if the judgment is "NG", i.e., the defect candidate area judged to be defective is determined to be a defective area, an "NG" judgment is output by the display device 62 (step S12), and the NG work is removed, i.e., the inspection object 5 including the defect area is removed from the measurement table 1 on the XYZ stage 44 (step S13). Also, even if the judgment of the final judgment of step S11 is "OK", i.e., even if the inspection object 5 includes a defect candidate area in the two-dimensional image that was judged to be "NG" in the defective judgment by AI inference in step S5, if the judgment result of the final judgment of the defect candidate area using the size information (X, Y) and the size information (Z) of all the defect candidate areas within the inspection area is "OK", an "OK" judgment is output by the display device 62 (step S14), and the OK work is removed, i.e., the inspection object 5 not including a defect area is removed from the measurement table 1 on the XYZ stage 44 (step S15), and then termination processing is performed.
[0057] In the post-termination process, an inspection termination determination process is performed to determine whether an instruction to terminate the inspection has been given via the input device 61 (step S16).If the determination result is "NO", i.e., if there is no instruction to terminate the inspection, the process returns to step S2, and the next object 5 to be inspected is placed on the measuring table 1 on the XYZ stage 44, and the appearance inspection process from step S2 to step S13 is repeated.If the determination result of step S16 is "YES", i.e., if there is an instruction to terminate the inspection, the appearance inspection is terminated.
[0058] That is, the three-dimensional appearance inspection device 100 includes a two-dimensional image acquisition step S for acquiring a two-dimensional image of the inspection area of the inspection object 5. A and the above-mentioned two-dimensional image acquisition process S A The first defect judgment process S performs defect judgment using AI inference on the 2D images acquired in B and the first defect determination step S B A first size detection step S2 performs rule-based size detection on the defective area of the two-dimensional image determined to be defective in the step S1 to obtain size information (X, Y). C and a three-dimensional shape information acquisition step S for acquiring three-dimensional shape information of a defect candidate area in the inspection area of the object to be inspected, the defect area corresponding to the defect area in the two-dimensional image. D and the above-mentioned three-dimensional shape information acquisition process S D A second size detection step S2 performs rule-based size detection on the three-dimensional shape information of the defect candidate area obtained in step S1 to obtain size information (Z). E and the first size detection step S C The size information (X, Y) obtained in the second size detection step S E The final determination step S is carried out to determine whether the defect candidate area is defective using the size information (Z) obtained in the step S. F The final determination step S F The appearance inspection method according to the present invention is characterized in that the defect candidate area determined to be defective in the step (1) is set as a defective area.
[0059] Here, the three-dimensional inspection device 100 performs an appearance inspection process on a printed circuit board as the inspection object 5, and images of various defect areas detected on a two-dimensional image of the printed circuit board displayed on the display device 62 connected to the information processing unit 50 are shown in (A), (B), and (C) of Figure 4. In Figure 4, (A) shows an image of a defect area occurring on the side edge of a circuit board on which components are mounted, (B) shows a defect area occurring on the circuit pattern of the circuit board, and (C) shows a soldering defect area occurring between the component mounted on the circuit board and the circuit pattern.
[0060] 5A, 5B, and 5C show images displayed on the display device 62 connected to the information processing unit 50, showing various determination states of defective areas during the visual inspection process of the printed circuit board by the three-dimensional inspection apparatus 100. In Fig. 5, (A) shows an image of a defective area determined to be a defective soldering area on the circuit board after the AI inference defect determination processing unit of the information processing unit performs defect determination using AI inference on the two-dimensional image of the circuit pattern area, (B) shows an image of a defective area obtained by quantifying the defective area after performing XY size detection of the defective soldering area in the two-dimensional image of the circuit pattern area using the rule-based defect determination processing unit of the information processing unit, and (C) shows an image of a defective area obtained by quantifying the defective area in three dimensions after performing rule-based size detection using three-dimensional shape information acquired by the three-dimensional shape measurement unit on a defective soldering area in the inspection area of the inspection object 5 corresponding to the defective area in the two-dimensional image determined to be a defective area by the rule-based defect determination processing, thereby obtaining size information (Z).
[0061] That is, in the 3D appearance inspection apparatus 100, a printed circuit board is placed as the inspection target 5 on the measurement table 1 on the XYZ stage 44, and the circuit pattern area on the upper surface of the printed circuit board is imaged as the inspection target area by the imaging unit 20 to obtain a 2D image of the circuit pattern area. The AI inference defect judgment processing unit 51 of the information processing unit 50 performs defect judgment by AI inference on the 2D image of the circuit pattern area, thereby roughly identifying the defect location, defect type, etc., as shown in Fig. 5A. Next, for defect locations that have been judged as NG by the AI inference, the rule-based defect judgment processing unit 52 of the information processing unit 50 detects the XY size of the defect area in the 2D image of the circuit pattern area, and the size of the defect location in the 2D image is quantified as shown in Fig. 5B. Furthermore, for defect candidate areas within the inspection area of the object to be inspected 5 corresponding to defect areas within the two-dimensional image that have been determined to be defect areas by the rule-based defect determination process, three-dimensional shape information is acquired by the three-dimensional shape measurement unit 30, and this three-dimensional shape information is used by the rule-based defect determination processing unit 52 of the information processing unit 50 to detect the Z size of the defect candidate areas, and the unevenness of the defect area is quantified in three dimensions, as shown in (C) of Figure 5.
[0062] Furthermore, here, a printed circuit board is used as the object of inspection 5, and defects in a printed circuit board with components mounted thereon and in the circuit pattern on the surface of the printed circuit board are detected by visual inspection using the three-dimensional visual inspection device 100. However, it is also possible to simply place a printed circuit board as the object of inspection 5 on the measurement table 1 and detect defects by three-dimensional visual inspection. The object of inspection 5 that is placed on the measurement table 1 and visually inspected by the three-dimensional visual inspection device 100 can be anything, not just a printed circuit board, as long as its upper surface is the inspection area. For example, as shown in Figures 6 and 7, an aluminum cast part can be placed on the measurement table 1 as the object of inspection 5, and defects in the processed surface can be detected by three-dimensional visual inspection.
[0063] Figures 6 (A), (B), and (C) show various defect areas on the machined surface of an aluminum cast part detected on a two-dimensional image by the above-mentioned three-dimensional inspection device 100, where (A) shows a defect area occurring on the side edge of the aluminum part, (B) shows a defect area occurring on the upper surface of the machined surface of the above-mentioned aluminum cast part, and (C) shows a defect area occurring on the periphery of the hole taper part of the above-mentioned aluminum cast part. 7A, 7B, and 7C are diagrams showing images displayed on a display device connected to the information processing unit, showing various judgment states of defect areas during the process of performing the visual inspection process of the machined surface of the aluminum cast part by the 3D inspection device 100. (A) shows an image of a defect area judged to be a defect area caused by a dent on the side edge of the machined surface of the aluminum cast part, after the AI inference defect judgment processing unit of the information processing unit has performed defect judgment using AI inference on the 2D image of the machined surface of the aluminum cast part. (B) shows an image of a defect area in which the rule-based defect judgment processing unit of the information processing unit has performed XY size detection on the defect area in the 2D image of the machined surface of the aluminum cast part and quantified the defect area. (C) shows an image of a defect area in which the defect area is quantified in three dimensions, after rule-based size detection is performed using the 3D shape information acquired by the 3D shape measurement unit to obtain size information (Z) for a defect candidate area in the inspection area of the inspection object 5 corresponding to the defect area in the 2D image that was judged to be a defect area by the rule-based defect judgment processing.
[0064] Furthermore, in this three-dimensional appearance inspection device 100, the optical comb three-dimensional shape measurement device adopted in the three-dimensional shape measurement unit 30 uses an optical comb as the measurement light, and through coaxial measurement at a large working distance, can accurately measure the three-dimensional shape of even an inspection object 5 with a complex shape without any blind spots, thereby obtaining three-dimensional shape information.
[0065] Here, in the appearance inspection process executed according to the procedure shown in the flowchart of FIG. 3, the final judgment step S F In the first size detection step S C The size information (X, Y) obtained in the second size detection step S EThe size information (Z) obtained in the first size detection step S is used to make a final defect judgment on the defect candidate area. C A second defect determination step S in which the defect of the defect candidate area is determined using the size information (X, Y) obtained in the step S C ' is provided, and the final judgment process S F Then, the second determination step S C The defect candidate area determined to be defective in the second size detection step S C The size information (Z) obtained in step (2) can be used to determine whether the defect candidate area is defective.
[0066] FIG. 8 is a flowchart showing another procedure of the three-dimensional inspection process executed by the three-dimensional inspection apparatus 100.
[0067] That is, the three-dimensional visual inspection apparatus 100 can be configured to execute the three-dimensional visual inspection method of the present invention in accordance with the procedure shown in the flowchart of FIG.
[0068] In the appearance processing according to the procedure shown in the flowchart of FIG. 8, the integrated processing unit 54 of the three-dimensional appearance inspection apparatus 100 performs the first defect determination step S in the appearance inspection processing executed according to the procedure shown in the flowchart of FIG. B A first size detection step S2 performs rule-based size detection on the defective area of the two-dimensional image determined to be defective in the step S1 to obtain size information (X, Y). C After the process, the second defect determination process S C In the first size detection step S C Using the size information (X, Y) obtained in step S2, a defect determination is performed on the defect candidate area whose size has been quantified. C If there is a defect candidate area that is determined to be defective in step S ′, the 3D shape information acquisition step S D , the second size detection step S E , final judgment step S F ' and the above-mentioned final judgment step S E The defect candidate area determined to be defective in step 1 is determined to be a defective area.
[0069] In the appearance processing according to the procedure shown in the flowchart of Figure 8, the same processing steps as in the appearance inspection processing executed according to the procedure shown in the flowchart of Figure 3 above are given the same reference numerals, and detailed explanations of those steps will be omitted.
[0070] That is, for the defective areas in the two-dimensional image that were judged as NG in step S5, the integrated processing unit 54 of the three-dimensional appearance inspection device 100 judges the quantified size of the defect candidate areas as defective using the size information (X, Y) obtained by rule-based size detection by the rule-based judgment processing unit 52 in step S6 (step S6'), and if the judgment result is "OK", an "OK" judgment is output by the display device 62 (step S14), and the OK work is removed, i.e., the inspection object 5 that does not contain a defective area is removed from the measuring table 1 on the XYZ stage 44 (step S15), and then termination processing is performed.
[0071] Then, if the judgment result in step S6' is "NG", that is, if there is a defect candidate area that is judged to be defective even in the defective judgment of the defect candidate area whose size has been quantified, the integrated processing unit 54 of the 3D appearance inspection device 100 proceeds to the relative movement of step S7, performs each process from step S7 to step S11', and determines the defect candidate area that is judged to be "NG" in the final judgment process of step S11' as a defective area.
[0072] Here, the final judgment step S F ', that is, in the final determination process of step S11', a determination process is performed to determine that the defect candidate area is a defective area using only the size information (Z) obtained in the size detection process of step S9.
[0073] In addition, in the appearance inspection process performed according to the procedure shown in the flowchart of FIG. 3, a final judgment step S F In the first size detection step S C The size information (X, Y) obtained in the second size detection process S EThe size information (Z) obtained in the first size detection step S is used to make a final defect judgment of the defect candidate area. C and skip the second size detection step S E In ', the three-dimensional shape information acquisition process S D Alternatively, size information (X, YZ) may be detected using the three-dimensional shape information of the defect candidate area obtained in step (2).
[0074] FIG. 9 is a flowchart showing still another procedure of the three-dimensional inspection process executed by the three-dimensional inspection apparatus 100.
[0075] That is, the three-dimensional visual inspection apparatus 100 can execute the three-dimensional visual inspection method of the present invention in accordance with the procedure shown in the flowchart of FIG.
[0076] The three-dimensional inspection method executed according to the procedure shown in the flowchart of FIG. 9 is the first size detection step S in the three-dimensional inspection method executed according to the procedure shown in the flowchart of FIG. C The same processing steps as those described above are denoted by the same reference numerals in FIG. 7, and detailed explanations thereof will be omitted.
[0077] Furthermore, the three-dimensional appearance inspection apparatus 100 is provided with the XYZ stage 44 which functions as a relative movement means for moving the inspection object 5 placed on the measurement table 1 and the three-dimensional shape measuring unit 30, i.e., the three-dimensional shape information acquisition means, relatively; however, in the three-dimensional appearance inspection apparatus of the present invention, the relative movement means can be, instead of the XYZ stage 44, a multi-joint robot 80 which moves the three-dimensional shape measuring unit 30, i.e., the three-dimensional shape information acquisition means, in three-dimensional directions, as in the three-dimensional appearance inspection apparatus 100A shown in Figures 8 and 9.
[0078] FIG. 10 is an external view of a three-dimensional visual inspection apparatus 100A, which is another example of the configuration of a three-dimensional visual inspection apparatus to which the present invention is applied, and FIG. 11 is a block diagram showing the configuration of the three-dimensional visual inspection apparatus 100A.
[0079] This three-dimensional visual inspection device 100A replaces the XYZ stage 44 in the three-dimensional visual inspection device 100 shown in Figures 1 and 2 with an articulated robot 80, and components that are identical to those of the three-dimensional visual inspection device 100 are shown with the same symbols in Figures 10 and 11, and detailed descriptions of them will be omitted.
[0080] That is, this three-dimensional inspection device 100A includes the above-mentioned articulated robot 80 that is driven by a robot driving unit 81 that is externally connected to a robot control unit 55A of an information processing unit 50A via a GPIO board 58, and the illumination unit 10, the two-dimensional imaging unit 20, and the three-dimensional shape measuring unit 30 are disposed at the tip of a robot arm 80A of the above-mentioned articulated robot 80.
[0081] The lighting unit 10, the two-dimensional imaging unit 20, and the three-dimensional shape measuring unit 30, which are arranged at the tip of the robot arm 80A, can freely change the posture facing the inspection object 5 placed on the stand 70 by driving the articulated robot 80 by the robot driving unit 81, and the two-dimensional imaging unit 20 can obtain two-dimensional images of the inspection area of each surface of the surface inspection object 5 except for the bottom surface side facing the top surface of the stand 70, and the three-dimensional shape measuring unit 30 can obtain three-dimensional shape information.
[0082] In this three-dimensional inspection apparatus 100A, the relative movement in step S7 in the appearance processing according to the procedures shown in the flowcharts of Figures 3, 6, and 7 is controlled by the robot control unit 55A, instead of the XYZ stage 44 being controlled by the XYZ stage control unit 55 in the three-dimensional inspection apparatus 100 shown in Figures 1 and 3, and the appearance inspection method of the present invention can be carried out according to the procedures shown in the flowcharts of Figures 3, 6, and 7.
[0083] Furthermore, in the above three-dimensional inspection devices 100, 100A, both the imaging unit 20, which images the inspection area of the inspection object 5 to obtain a two-dimensional image, and the three-dimensional shape measuring unit 30, which obtains three-dimensional shape information of candidate defect areas within the inspection area of the inspection object 5, are moved relative to the inspection object 5 by relative movement means, i.e., the XYZ stage 44 and the articulated robot 80. However, for example, a three-dimensional inspection system may be constructed which includes an AI visual inspection device on a conveying line where the inspection objects are conveyed by a conveying device that obtains two-dimensional image information of the inspection area of the inspection object by an imaging means and performs visual inspection processing by an information processing means, and a three-dimensional visual inspection device that obtains three-dimensional shape information by a three-dimensional shape measuring means for candidate defect areas of the inspection object determined to be defective by the AI visual inspection device and performs a final visual inspection by an information processing means.
[0084] FIG. 12 is a block diagram showing an example of the configuration of a three-dimensional inspection system 200 according to the present invention.
[0085] This three-dimensional visual inspection system 200 comprises an AI visual inspection device 20A that acquires two-dimensional image information of the inspection area of the inspection object on a conveying line where the inspection object is conveyed by a conveying device and performs visual inspection processing, and a three-dimensional visual inspection device 30A that acquires three-dimensional shape information for the defect candidate area of the inspection object that has been determined to be defective by the AI visual inspection device 20A and performs a final visual inspection.
[0086] This three-dimensional visual inspection system 200 can be configured, for example, by using multiple visual inspection devices with visual inspection functions through qualitative inspection processes and qualitative judgment processing, which are installed in an existing production line in which inspection objects processed by a processing machine 201 are transported by a transport device, as AI visual inspection devices 20A, 20B, and by providing the three-dimensional visual inspection device 30A that performs a final visual inspection of the defect candidate areas of the inspection object through a quantitative inspection process and quantitative judgment processing using three-dimensional shape information after the qualitative inspection process and qualitative judgment processing by the AI visual inspection devices 20A, 20B.
[0087] In this 3D visual inspection system 200, the AI visual inspection devices 20A and 20B perform visual inspection through a qualitative inspection process and qualitative judgment process using 2D image information by executing processes from steps S1 to S6' and steps S14 to S15 in the processing flow shown in Figure 3, respectively, and the 3D visual inspection device 30A performs processes from steps S7 to S15, thereby performing a final visual inspection through a quantitative inspection process and quantitative judgment process using 3D shape information for defect candidate areas of the inspection object that have been judged to be defective by the AI visual inspection devices 20A and 20B.
[0088] In this three-dimensional visual inspection system 200, in the final visual inspection by the three-dimensional visual inspection device 30A, the inspection object whose size information (X, Y, Z) of the defect candidate area is equal to or greater than the inspection threshold is determined to be an OK product, and the inspection object whose size information (X, Y, Z) of the defect candidate area is equal to or greater than the inspection threshold is determined to be an NG product.
[0089] Here, in the above-mentioned three-dimensional inspection devices 100 and 100A, a two-dimensional image of the inspection area of the inspection object 5 is acquired by the imaging unit 20, and in the above-mentioned three-dimensional inspection system 200, two-dimensional image information of the inspection area of the inspection object 5 is acquired by the inspection AI appearance inspection device 20A, rule-based size detection is performed on the defective area of the above-mentioned two-dimensional image to obtain size information (X, Y), and defect judgment is made on the size information (X, Y). However, the size information used for defect judgment is not limited to size information (X, Y) in the X and Y directions which are orthogonal to each other in the defective area of the above-mentioned two-dimensional image, but may be size information in a different direction. [Explanation of symbols]
[0090] 1 measurement table, 5 inspection object, 10 lighting unit, 20 2D imaging unit, 20A, 20B AI appearance inspection device, 30A 3D appearance inspection device, 30 3D shape measurement unit, 31 optical comb light source, 32 interferometer, 33 optical comb 3D shape sensor, 40 XYZ stage, 41 base (base), 42X, 42Y, 42Z travel path, 43X X-axis movement unit, 43Y Y-axis movement unit, 43Z Z-axis movement unit, 50, 50A information processing unit, 51 AI inference defect judgment processing unit, 52 rule-based judgment processing unit, 53 optical comb control unit, 54 integration processing unit, 55 XYZ stage control unit 55A robot control unit, 55 camera board, 56 optical comb board, 57 GPIO board, 61 input device, 62 display device, 70 stand, 80 articulated robot, 80A robot arm, 81 Robot drive unit, 100, 100A 3D inspection device, 200 3D inspection system, 210 Processing machine
Claims
1. a two-dimensional image acquisition step of acquiring a two-dimensional image of an inspection area of an inspection object; a first defect determination step of performing defect determination by AI inference on the two-dimensional image acquired in the two-dimensional image acquisition step; a three-dimensional shape information acquisition step of acquiring three-dimensional shape information of a defect candidate region within an inspection region of the inspection object corresponding to a defect region of the two-dimensional image determined to be defective in the first defect determination step; a size detection step of performing rule-based size detection on the three-dimensional shape information of the defect candidate area to obtain size information; a final determination step of determining whether the defect candidate area is defective using the size information obtained in the size detection step; and a defect candidate area determined to be defective in the final determination step, the defect candidate area being determined to be defective;
2. The size detection step includes: a first size detection step of performing rule-based size detection on the defect area of the two-dimensional image determined to be defective in the first defect determination step to obtain size information on a two-dimensional plane; a second size detection step of performing rule-based size detection on the three-dimensional shape information of the defect candidate area to acquire size information in the height direction; 2. The visual inspection method according to claim 1, further comprising:
3. 2. The visual inspection method according to claim 1, wherein said size detection step acquires size information including the size and height of said defect candidate region in a two-dimensional plane.
4. a two-dimensional image acquisition means for acquiring a two-dimensional image of an inspection area of an inspection object; a three-dimensional shape information acquisition means for acquiring three-dimensional shape information of a defect candidate region within an inspection region of the inspection object; a relative movement means for moving the inspection object and the three-dimensional shape information acquisition means relative to each other; an information processing means for controlling the operations of the two-dimensional image acquiring means, the three-dimensional shape information acquiring means, and the relative moving means, and for detecting a defect area in the inspection area of the object to be inspected by using the two-dimensional image acquired by the two-dimensional image acquiring means and the three-dimensional shape information acquired by the three-dimensional shape information acquiring means; Equipped with the information processing means drives the two-dimensional image acquisition means to acquire a two-dimensional image of the inspection area of the inspection object, performs a defect judgment using AI inference on the acquired two-dimensional image, and designates a defect area of the two-dimensional image that is judged to be defective as a defect candidate area; drives the relative movement means and the three-dimensional shape information acquisition means to acquire three-dimensional shape information of the defect candidate area; performs rule-based size detection on the acquired three-dimensional shape information of the defect candidate area to obtain size information; uses the size information to make a final defect judgment on the defect candidate area; and designates the defect candidate area that is judged to be defective in the final defect judgment as a defect area.
5. The three-dimensional appearance inspection device according to claim 4, characterized in that the information processing means performs rule-based size detection on defective areas of the two-dimensional image that have been determined to be defective by the defect determination using AI inference to obtain size information on a two-dimensional plane, and performs rule-based size detection on three-dimensional shape information of the defect candidate area to obtain size information in the height direction.
6. The three-dimensional appearance inspection device described in claim 4, characterized in that the information processing means performs rule-based size detection on the three-dimensional shape information of the defect candidate area to obtain size information including the size and height of the defect candidate area in a two-dimensional plane.
7. 7. The three-dimensional appearance inspection apparatus according to claim 4, wherein the relative movement means is an XYZ stage that moves the inspection object or the three-dimensional shape information acquisition means in three dimensions.
8. 7. The three-dimensional appearance inspection apparatus according to claim 4, wherein the relative movement means is an articulated robot that moves the three-dimensional shape information acquisition means in three dimensions.
9. 7. The three-dimensional appearance inspection device according to claim 4, wherein the three-dimensional shape information acquisition means is an optical three-dimensional shape measurement device that acquires three-dimensional shape information by scanning a defect candidate area within the inspection area of the object to be inspected with measurement light.
10. an AI appearance inspection device that is provided on a conveyance line that conveys inspection objects by a conveyance device, captures an inspection area of the inspection object to obtain a two-dimensional image, performs a defect judgment on the obtained two-dimensional image by AI inference, and performs rule-based size detection on a defect area of the two-dimensional image that is judged to be defective to obtain size information; and a three-dimensional appearance inspection device that acquires, for the inspection object determined to be defective by the AI appearance inspection device, three-dimensional shape information of a defect candidate area within an inspection area of the inspection object that corresponds to the defect area of the two-dimensional image, performs rule-based size detection on the acquired three-dimensional shape information of the defect candidate area to obtain size information in a height direction, makes a final defect judgment of the defect candidate area using the size information acquired in the qualitative inspection step and the size information in the height direction, and performs a quantitative inspection in which the defect candidate area determined to be defective in the final judgment step is determined to be a defect area; A three-dimensional appearance inspection system comprising:
11. The three-dimensional appearance inspection system according to claim 10, characterized in that the three-dimensional appearance inspection device is provided with an optical three-dimensional shape measurement means for scanning a defect candidate area within the inspection area of the object to be inspected with measurement light to acquire three-dimensional shape information.
Citation Information
Patent Citations
Inspection device
JP2019190911A
Information processing method, information processor, and program
JP2019204321A
Abnormality determination method and abnormality determination device
JP2021174456A
Optical scanner calibration method, optical scanner, and optical three-dimensional shape measuring device
JP7353644B2
Distance measurement method, optical comb distance meter, and optical three-dimensional shape measurement device
JP7385919B2