Computer-implemented method and system for automatic optical inspection

Abstract

The invention relates to a computer-implemented method for automatic optical inspection, comprising: capturing (S1) a time series of image data (BD) of at least one individual image (10) of a plurality of products (12) produced on a production line (24); providing (S2) a graphical user interface (14) for displaying the at least one individual image (10) of the plurality of products (12) produced on the production line (24), the graphical user interface (14) having a control (16), in particular a slide control, which is configured such that, when actuated, the control scrolls through the time series of image data (BD); applying (S3) a machine learning algorithm (A) for automatic optical inspection to the image data (BD) of the at least one individual image (10) of the plurality of products (12) produced on the production line (24); and, if an anomaly is detected, outputting (S4) at least one individual image (10) having the anomaly on the graphical user interface (14) and positioning the control (16) at a position of the individual image (10) having the anomaly along the time series of image data (BD). The invention further relates to a system (1) for automatic optical inspection.

Description

[0001] Description

[0002] Title for automatic

[0003] The invention relates to a computer-implemented method for automatic optical inspection.

[0004] Furthermore, the invention relates to a system for automatic optical inspection, a production line for manufacturing a workpiece and a workpiece or product produced by the production line.

[0005] State of the art

[0006] Automatic optical inspection is primarily used to detect anomalies in industrial manufacturing processes and the resulting defective products.

[0007] A manufacturing process or industrial process is generally understood to be a standardized workflow in which a product is manufactured using predetermined manufacturing processes, work tools and resources through mechanical and / or manual processing of raw materials or intermediate products.

[0008] Anomaly detection is a data analysis method that identifies specific instances or patterns that deviate from the norm or statistically expected behavior. The goal is to determine, through data analysis, whether a data point deviates from the normal or expected pattern or behavior. To do this, a reference must first be established to define what behavior can be considered normal. A variety of methods are used for anomaly detection, all of which have in common the ability to identify outliers in large data sets. These methods typically use historical data from a machine or an entire system.

[0009] There are currently two approaches for detecting anomalies in repeatable time series data, such as that collected during production or the machining of industrial workpieces.

[0010] In rule-based approaches, experts manually set thresholds, either for the time series itself or for an extracted feature, with observations above this threshold being classified as anomalous.

[0011] Tabulation involves extracting tabular features from the time series. An anomaly detection algorithm for tabular data is applied to the resulting features to detect anomalies or changes in state. Machine learning algorithms are often used for this purpose.

[0012] However, in order to monitor the performance of the algorithm used, domain knowledge of a process expert is always required.

[0013] Consequently, there is a need to prepare the results determined during an automatic optical inspection for the process expert in such a way that they can be efficiently evaluated.

[0014] The invention is therefore based on the object of finding an approach with which the results of an automatic optical inspection can be evaluated in an efficient manner.

[0015] Disclosure of the invention The object is achieved according to the invention by a computer-implemented method for automatic optical inspection having the features of patent claim 1.

[0016] Furthermore, the object is achieved according to the invention by a computer program having the features of patent claim 10 and a computer-readable data carrier having the features of patent claim 11.

[0017] The object is further achieved according to the invention by a system for automatic optical inspection having the features of patent claim 12.

[0018] Furthermore, the object is achieved by a production line for producing a workpiece having the features of patent claim 13 and a workpiece produced by the production line having the features of patent claim 14.

[0019] The invention relates to a computer-implemented method for automatic optical inspection.

[0020] The method comprises capturing a time series of image data of at least one individual image of a plurality of products produced on a production line.

[0021] Furthermore, the method comprises providing a graphical user interface for displaying the at least one individual image of the plurality of products produced on the production line, wherein the graphical user interface has a controller, in particular a slider, which, when actuated, is configured to scroll through the time series of image data.

[0022] The method further comprises applying a machine learning algorithm for automatic optical inspection to the image data of the at least one individual image of the plurality of products produced on the production line, and upon detection of an anomaly, outputting at least one individual image exhibiting the anomaly on the graphical user interface and positioning the controller at a position of the individual image exhibiting the anomaly along the time series of image data.

[0023] The invention further relates to a computer program with program code for carrying out the method according to the invention when the computer program is executed on a computer.

[0024] Furthermore, the invention relates to a computer-readable data carrier with program code of a computer program in order to carry out the method according to the invention when the computer program is executed on a computer.

[0025] The invention further relates to a system for automatic optical inspection, wherein the system is configured to carry out the method according to the invention.

[0026] Furthermore, the invention relates to a production line for manufacturing a workpiece, wherein the workpiece is manufactured using an industrial process, wherein the production line is designed to monitor the industrial process using the method according to the invention.

[0027] The invention further relates to a workpiece produced by the production line, wherein the production line produces the workpiece using the method according to the invention.

[0028] An idea of ​​the present invention is to enable efficient evaluation of the results of the automatic optical inspection by positioning the controller at a position of the individual image exhibiting the anomaly along the time series of image data.

[0029] The slider is set or adjusted at a position of a first individual image exhibiting an anomaly. Furthermore, each anomaly along the time series of image data can be marked. This allows for easy jumping between detected anomalies along the time series of image data. Jumping can be performed, for example, using a dedicated forward and / or back button on the graphical user interface.

[0030] Further embodiments of the present invention are the subject of the further subclaims and the following description with reference to the figures.

[0031] According to a preferred development of the invention, it is provided that the machine learning algorithm carries out a classification of the at least one individual image of the plurality of products produced on a production line into a first standard class and a second anomaly class.

[0032] The standard class refers to a normal condition of the produced product, whereas the anomaly class represents a defective condition of the produced product.

[0033] According to a further preferred development of the invention, it is provided that the time series of image data comprises a plurality of individual images of spatial sub-regions of the product.

[0034] The spatial subregions of the product can, for example, be the bonding points of a component. Thus, all spatial subregions of the product can be output in a single representation.

[0035] According to a further preferred development of the invention, it is provided that the step of outputting the at least one individual image having the anomaly on the graphical user interface comprises an automatic annotation of the spatial sub-region of the product having the anomaly.

[0036] Thus, the user can see, without further verification, which of the plurality of spatial subregions of the product exhibits the anomaly. According to a further preferred development of the invention, before applying the machine learning algorithm for automatic optical inspection to the image data, a temporal subregion of the time series of image data is selected, wherein the selected temporal subregion defines a standard range of the produced products.

[0037] In order to define a data basis for the standard range of manufactured products for the machine learning algorithm, it is advantageous for a process expert to select the desired temporal sub-range. Alternatively, the selection of the temporal sub-range can be automated according to a predefined rule. The rule can, for example, require that certain image attributes, such as the orientation or alignment of an object, remain within a predefined definition range. Other image attributes include brightness, a pattern, etc.

[0038] According to a further preferred development of the invention, the machine learning algorithm determines a deviation score from the defined normal range for at least one individual image. The determined deviation score can then be output to the user in combination with the correspondingly classified anomaly individual image.

[0039] According to a further preferred development of the invention, a feature vector is generated for each spatial sub-region of the product, and the deviation score value is determined for each spatial sub-region of the product from the respective feature vector using the machine learning algorithm. The use of the feature vector enables a measurement of a vector distance from a predetermined standard range or centroid.

[0040] According to a further preferred development of the invention, in response to the detection of the anomaly, a directed action of a technical system, in particular the production line, is triggered. Thus, the anomaly can be responded to advantageously, for example, by making a corresponding adjustment to the production line.

[0041] According to a further preferred development of the invention, it is provided that the time series of image data of the at least one individual image of the plurality of products produced on the production line is captured by a camera image sensor.

[0042] The features of the computer-implemented method for automatic optical inspection described herein are also applicable to the system for automatic optical inspection and vice versa.

[0043] Short description of the drawings

[0044] For a better understanding of the present invention and its advantages, reference is now made to the following description in conjunction with the accompanying drawings.

[0045] The invention is explained in more detail below using exemplary embodiments which are shown in the schematic illustrations of the drawings.

[0046] They show:

[0047] Fig.1 is a flowchart of the computer-implemented method for automatic optical inspection according to a preferred embodiment of the invention; and

[0048] Fig. 2 is a schematic representation of the automatic optical inspection system according to the preferred embodiment of the invention. Unless otherwise indicated, like reference numerals refer to like elements in the drawings.

[0049] Detailed description of the embodiments

[0050] The computer-implemented method for automatic optical inspection shown in Fig. 1 comprises capturing S1 a time series of image data BD of at least one individual image 10 of a plurality of products 12 produced on a production line 24.

[0051] Furthermore, the method comprises providing S2 a graphical user interface 14 for displaying the at least one individual image 10 of the plurality of products 12 produced on the production line 24, wherein the graphical user interface 14 has a controller 16, in particular a slider, which, when actuated, is configured to scroll through the time series of image data BD.

[0052] The method further comprises applying S3 a machine learning algorithm A for automatic optical inspection to the image data BD of the at least one individual image 10 of the plurality of products 12 produced on the production line 24.

[0053] Furthermore, the method comprises detecting an anomaly, outputting S4 at least one individual image 10 exhibiting the anomaly on the graphical user interface 14 and positioning the controller 16 at a position of the individual image 10 exhibiting the anomaly along the time series of image data BD.

[0054] The machine learning algorithm A performs a classification of the at least one individual image 10 of the plurality of products 12 produced on a production line 24 into a first standard class K1 and a second anomaly class K2.

[0055] The time series of image data BD comprises a plurality of individual images of spatial subregions 18 of the product 12. The step of outputting the at least one individual image 10 exhibiting the anomaly on the graphical user interface 14 comprises automatically annotating the spatial subregion 18 of the product 12 exhibiting the anomaly.

[0056] Before applying the machine learning algorithm A for automatic optical inspection to the image data BD, a selection of a temporal sub-range 20 of the time series of image data BD is carried out, wherein the selected temporal sub-range 20 defines a standard range of the produced products 12.

[0057] The machine learning algorithm A determines a deviation score 22 from the defined standard range for each at least one individual image 10. Furthermore, a feature vector is generated for each spatial sub-area 18 of the product 12, and the deviation score 22 is determined for each spatial sub-area 18 of the product 12 from the respective feature vector using the machine learning algorithm A.

[0058] In response to the detection of the anomaly, a directed action of a technical system 1, in particular of the production line 24, is also triggered. The time series of image data BD of the at least one individual image 10 of the plurality of products 12 produced on the production line 24 is further captured by a camera image sensor 25.

[0059] Fig. 2 shows a schematic representation of the system 1 for automatic optical inspection according to the preferred embodiment of the invention.

[0060] The system 1 for automatic optical inspection shown in Fig. 2 comprises first means 24 for acquiring a time series of image data BD of at least one individual image 10 of a plurality of products 12 produced on a production line 26.

[0061] Furthermore, the system 1 comprises second means 28 for providing a graphical user interface 14 for displaying the at least one individual image 10 of the plurality of products 12 produced on the production line 24, wherein the graphical user interface 14 has a controller 16, in particular a slider, which, when actuated, is configured to scroll through the time series of image data BD.

[0062] The system 1 further comprises third means 30 for applying a machine learning algorithm A for automatic optical inspection to the image data BD of the at least one individual image 10 of the plurality of products 12 produced on the production line 24.

[0063] Furthermore, upon detection of an anomaly, fourth means 32 for outputting at least one individual image 10 exhibiting the anomaly on the graphic

[0064] User interface 14 and positioning the controller 16 at a position of the individual image 10 exhibiting the anomaly along the time series of image data BD.

Claims

Claims 1 . A computer-implemented method for automatic optical inspection, the method comprising the following steps: capturing (S1) a time series of image data (BD) of at least one individual image (10) of a plurality of products (12) produced on a production line (24); Providing (S2) a graphical user interface (14) for displaying the at least one individual image (10) of the plurality of products (12) produced on the production line (24), wherein the graphical user interface (14) has a controller (16), in particular a slider, which, when actuated, is configured to scroll through the time series of image data (BD); Applying (S3) a machine learning algorithm (A) for automatic optical inspection to the image data (BD) of the at least one individual image (10) of the plurality of products (12) produced on the production line (24); and upon detection of an anomaly, outputting (S4) at least one individual image (10) exhibiting the anomaly on the graphical user interface (14) and positioning the controller (16) at a position of the individual image (10) exhibiting the anomaly along the time series of image data (BD).

2. Computer-implemented method according to claim 1, wherein the machine learning algorithm (A) performs a classification of the at least one individual image (10) of the plurality of products (12) produced on a production line (24) into a first standard class (K1) and a second anomaly class (K2).

3. Computer-implemented method according to claim 1 or 2, wherein the time series of image data (BD) comprises a plurality of individual images of spatial subregions (18) of the product (12).

4. The computer-implemented method of claim 3, wherein the step of outputting the at least one individual image (10) having the anomaly on the graphical user interface (14) comprises automatically annotating the spatial sub-region (18) of the product (12) having the anomaly.

5. Computer-implemented method according to claim 3 or 4, wherein before applying (S3) the machine learning algorithm (A) for automatic optical inspection to the image data (BD), a selection of a temporal sub-range (20) of the time series of image data (BD) is carried out, wherein the selected temporal sub-range (20) defines a standard range of the produced products (12).

6. The computer-implemented method according to claim 5, wherein the machine learning algorithm (A) determines a deviation score value (22) from the defined normal range for each at least one individual image (10).

7. Computer-implemented method according to claim 6, wherein a feature vector is generated for each spatial sub-area (18) of the product (12), and wherein the deviation score value (22) is determined for each spatial sub-area (18) of the product (12) from the respective feature vector using the machine learning algorithm (A).

8. Computer-implemented method according to one of the preceding claims, wherein in response to the detection of the anomaly, a directed action of a technical system (1), in particular the production line (24), is triggered.

9. Computer-implemented method according to one of the preceding claims, wherein the time series of image data (BD) of the at least one A single image (10) of the plurality of products (12) produced on the production line (24) is captured by a camera image sensor (25).

10. A computer program comprising program code for carrying out a method according to any one of claims 1 to 9 when the computer program is executed on a computer.

11. A computer-readable data carrier containing program code of a computer program for carrying out a method according to any one of claims 1 to 9 when the computer program is executed on a computer.

12. System (1) for automatic optical inspection, wherein the system (1) is configured to carry out a method according to one of claims 1 to 9.

13. A production line (24) for manufacturing a workpiece, wherein the workpiece is manufactured using an industrial process, wherein the production line (24) is designed to monitor the industrial process using the method according to one of claims 1 to 9.

14. A workpiece produced by a production line (24) according to claim 13, wherein the production line (24) produces the workpiece using a computer-implemented method according to any one of claims 1 to 9.

Images