Automatic visual inspection device and visual inspection program

Abstract

To provide an automatic visual inspection device and a visual inspection program capable of allowing a worker to recognize what factors had a significant impact on the outcome when determining the appearance of protective glass using a machine learning model.SOLUTION: The automatic visual inspection device causes a computer to perform the following automatic visual inspection processing: generating numerical data regarding the number of detections of a predetermined object in part or in whole of protective glass based on inspection images acquired by taking images of the protective glass; acquiring a first determination result based on the numerical data using a machine learning model for numerical data; and displaying the first determination result and correlation data showing the degree of influence of multiple types of features on the first determination result simultaneously on a display unit.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a technique for making a judgment on an inspection image acquired by photographing a protective glass. [Background technology]

[0002] Patent Document 1 discloses an automatic visual inspection device that detects stains on an inspection image acquired by photographing an inspection object based on a data group obtained by machine learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-28564 Summary of the Invention [Problem to be solved by the invention]

[0004] The protective glass of the laser processing head used in a laser processing system may become contaminated with spatter and fumes. It is conceivable to use a machine learning model to judge the appearance of such protective glass based on an inspection image acquired by photographing it. In such cases, there is a need to enable the operator to recognize which factors significantly influenced the judgment result.

[0005] The present disclosure has been made in consideration of these points, and its purpose is to enable an operator to recognize which factors have a significant influence on the judgment results when making a judgment regarding the appearance of a protective glass using a machine learning model. [Means for solving the problem]

[0006] In order to achieve the above object, a first embodiment of the present disclosure is characterized in that a computer is caused to execute a process of generating numerical data regarding a specified detected object on a protective glass based on an inspection image obtained by photographing the protective glass, obtaining a first judgment result based on the numerical data using a machine learning model for numerical data, and simultaneously displaying the first judgment result and correlation data indicating the degree of influence of multiple types of feature amounts contained in the numerical data on the first judgment result on a display device.

[0007] This allows the first judgment result and the correlation data to be displayed simultaneously on the display device, allowing the operator to recognize which of the multiple types of feature amounts contained in the numerical data had a strong influence on the first judgment result. [Effects of the Invention]

[0008] According to the present disclosure, when a machine learning model is used to judge the appearance of a protective glass, an operator can recognize which factors have had a significant impact on the judgment result. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram of an automatic visual inspection device according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a cross-sectional view of the protective glass imaging device taken along line II-II in FIG. [Figure 3] FIG. 3 is a diagram illustrating an example of an inspection image. [Figure 4] FIG. 4 is a table illustrating the numerical data. [Figure 5] FIG. 5 is a flowchart illustrating the operation of the automatic visual inspection apparatus according to the embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates the first determination result, its prediction accuracy, and correlation data in a display format. [Figure 7] FIG. 7 illustrates the second determination result and its prediction accuracy in a display format. [Figure 8] FIG. 8 is a diagram illustrating a display screen of the display device. [Figure 9] FIG. 9 illustrates correlation data in a display format according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or uses in any way.

[0011] 1 is a schematic diagram of an automatic appearance inspection device 100 according to an embodiment of the present disclosure for inspecting the appearance of a protective glass 202. The automatic appearance inspection device 100 includes a protective glass imaging device 10, a computer 15, a display device 16, and an input device 17.

[0012] As shown in FIG. 2, the protective glass photographing device 10 has a box-shaped housing 11 that opens upward. A camera 12 is housed in the upper part of the housing 11 so as to photograph downward. The upper part of the camera 12 protrudes outside the housing 11. A circular light 13 is housed in the upper part of the housing 11 so as to surround the camera 12 and illuminate downward. A rectangular plate-shaped backlight 14 is placed on the bottom of the housing 11 so as to illuminate upward. A protective glass cartridge 201 with a protective glass 202 attached is fixed between the camera 12, the light 13, and the backlight 14 by fixing means (not shown). In this state, the protective glass 202 faces the camera 12. Therefore, by photographing the protective glass 202 with the camera 12 in this state, an inspection image such as that shown in FIG. 3 can be obtained.

[0013] The computer 15 includes a control unit 151 , an image analysis unit 152 , and a processing unit 153 .

[0014] The control unit 151 controls the timing of photographing by the camera 12, the timing of lighting by the light 13, etc. The functions of the control unit 151 are realized by causing a CPU (Central Processing Unit) to execute a program, for example.

[0015] The image analysis unit 152 generates numerical data relating to the number of detected stains and their maximum area as detected objects in the central region of the protective glass 202 and in the outer region other than the central region by image processing based on the inspection image acquired by the camera 12. In FIG. 3, the central region of the protective glass 202 is shown surrounded by a two-dot chain line. The numerical data illustrated in FIG. 4 shows that, for the protective glass 202, the number of detected stains in the central region is five, and the maximum area of ​​one stain in the central region is 13.0 (mm 2 ), the number of detected stains in the outer region is one, and the maximum area of ​​one stain in the outer region is 0.1 (mm 2 The function of the image analysis unit 152 is realized by, for example, causing a CPU to execute an image analysis program (software).

[0016] The processing unit 153 performs calculations using a machine learning model based on the numerical data generated by the image analysis unit 152 and the inspection image acquired by the camera 12, and displays the calculation results on the display device 16. The functions of the processing unit 153 are realized by having a GPU (Graphics Processing Unit) execute a program. Detailed processing by the processing unit 153 will be described later. The programs executed by the image analysis unit 152 and the processing unit 153 constitute an appearance inspection program.

[0017] The display device 16 displays a predetermined display screen based on the output of the processing unit 153 of the computer 15.

[0018] The input device 17 receives an input indicating the determination result from an operator. The input device 17 is, for example, a keyboard, a mouse, or the like.

[0019] The operation of the automatic visual inspection apparatus 100 configured as described above will be described with reference to the flowchart of FIG.

[0020] First, in step S101, the camera 12 captures an image of the protective glass 202 to obtain an inspection image such as that shown in FIG.

[0021] Next, in step S102, the image analysis unit 152 of the computer 15 generates the numerical data as shown in FIG. 4 by image processing based on the inspection image acquired in step S101.

[0022] Next, in step S103, the processing unit 153 of the computer 15 acquires a first judgment result and its prediction accuracy using a machine learning model for numerical data based on the numerical data generated in step S102. The processing unit 153 then stores the acquired first judgment result and its prediction accuracy. The first judgment result indicates whether the protective glass 202 is suitable for use. The table at the top of FIG. 6 exemplifies the first judgment result and its prediction accuracy in a display format on the display device 16. In this example, the first judgment result (Judge) indicates that the protective glass 202 is not suitable for use (NG (no good)), and its prediction accuracy (Score) is 0.94. As described above, the machine learning model for numerical data receives the numerical data as input and outputs the first judgment result. The machine learning model for numerical data is generated by machine learning using multiple sets of numerical data and the first judgment result as training data. The processing unit 153 also stores correlation data indicating the degree of influence of multiple types of feature values ​​included in the numerical data on the first judgment result. This correlation data can be generated based on parameters that specify a machine learning model for numerical data. An example of the correlation data is shown in the lower part of FIG. 6 in a display format displayed on the display device 16. In the correlation data illustrated in FIG. 6, the features are the number of detected stains in the central region (inner count), the maximum area of ​​stains in the outer region (outer max area), the maximum area of ​​stains in the central region (inner max area), and the number of detected stains in the outer region (outer count). The graph in FIG. 6 shows, from top to bottom, the degree of influence (Variable Importance) of the number of detected stains in the central region (inner count), the maximum area of ​​stains in the outer region (outer max area), the maximum area of ​​stains in the central region (inner max area), and the number of detected stains in the outer region (outer count) on the first determination result. This graph shows the degree of influence of multiple features on the first determination result, from largest to smallest.

[0023] Furthermore, in parallel with steps S102 and S103, in step S104, the processing unit 153 acquires and stores a second judgment result using the image machine learning model based on the inspection image acquired in step S101. The second judgment result also indicates whether the protective glass 202 is suitable for use. FIG. 7 illustrates the second judgment result and its prediction accuracy in a display format on the display device 16. In this example, the second judgment result (Judge) indicates that the protective glass 202 is suitable for use (OK), and its prediction accuracy (Score) is 0.60. In this way, the image machine learning model receives the inspection image as input and outputs the second judgment result. The image machine learning model is generated by machine learning using multiple sets of inspection images and the second judgment results as training data.

[0024] Then, in step S105, processing unit 153 obtains an overall judgment result based on the first judgment result and the second judgment result. Specifically, if both the first judgment result and the second judgment result indicate that the protective glass 202 is suitable for use, processing unit 153 judges that the protective glass 202 is suitable for use (◯). On the other hand, if both the first judgment result and the second judgment result indicate that the protective glass 202 is not suitable for use, processing unit 153 judges that the protective glass 202 is not suitable for use (×). Furthermore, if only one of the first judgment result and the second judgment result indicates that the protective glass 202 is suitable for use, processing unit 153 judges that it is neither suitable nor unsuitable (Δ).

[0025] Next, in step S106, the processing unit 153 selects one of the first and second judgment results based on the prediction accuracy of the first judgment result obtained in step S103, the prediction accuracy of the second judgment result obtained in step S104, a first match rate between past judgment results using the machine learning model for numerical data and judgment results by the operator, and a second match rate between past judgment results using the machine learning model for images and judgment results by the operator. Specifically, the processing unit 153 calculates a priority by adding values ​​indicating the prediction accuracy and the match rate as a percentage, and selects the judgment result with the higher priority. For example, if the prediction accuracy of the first judgment result is 0.94 (94%) and the first match rate is 50%, and the prediction accuracy of the second judgment result is 0.6 (60%) and the second match rate is 60%, the priority of the second judgment result is 144 points and the priority of the second judgment result is 120 points. Therefore, in this case, the processing unit 153 selects the first determination result.

[0026] Next, in step S107, processing unit 153 determines whether the determination result selected in step S106 is the first determination result. If the determination result selected in step S106 is the first determination result, processing proceeds to step S108. On the other hand, if the determination result selected in step S106 is the second determination result, processing proceeds to step S109.

[0027] In step S108, the processing unit 153 simultaneously displays the first determination result obtained in step S103, the correlation data, the second determination result obtained in step S104, and the overall determination result obtained in step S105 on the display device 16, as shown in Fig. 8. At this time, the processing unit 153 displays the first determination result selected in step S106 larger than the second determination result. In the example of Fig. 8, the first determination result by the machine learning model for numerical data is that the protective glass 202 is not suitable for use, the second determination result by the machine learning model for images is that the protective glass 202 is suitable for use, and the overall determination result is neutral (△).

[0028] In step S109, processing unit 153 also causes display device 16 to simultaneously display the first determination result obtained in step S103, the correlation data, the second determination result obtained in step S104, and the overall determination result obtained in step S105. At this time, processing unit 153 causes the second determination result selected in step S106 to be displayed larger than the first determination result. Furthermore, in step S109, processing unit 153 causes the correlation data to be displayed smaller than in step S108.

[0029] In step S110, the worker visually checks the inspection image acquired in step S101 to determine whether the protective glass 202 is suitable for use. Then, the worker inputs a determination result to the input device 17. The processing unit 153 stores the determination result received by the input device 17, i.e., the worker's determination result, in association with the inspection image acquired in step S101. Thereafter, at an appropriate timing, the processing unit 153 generates a new image machine learning model using the inspection image acquired in step S101 and the worker's determination result acquired in step S110 as training data. The processing unit 153 then updates the image machine learning model to be used to the generated image machine learning model. The image machine learning model is updated, for example, once a day.

[0030] In step S111, the first and second match rates are calculated and updated based on the first judgment result, the second judgment result, and the judgment result by the operator that have been stored up to that point.

[0031] As described above, in steps S108 and S109, the first judgment result and the correlation data are simultaneously displayed on the display device, so that the operator can recognize which of the multiple types of feature values ​​contained in the numerical data has had a strong influence on the first judgment result.

[0032] Furthermore, the judgment result with the higher priority out of the first judgment result and the second judgment result is displayed in a large size on the display device 16, so that the worker can recognize at a glance which of the first judgment result and the second judgment result has the higher priority.

[0033] Furthermore, since the processing unit 153 can update the image machine learning model using the most recent judgment result by the operator, the accuracy of the second judgment result can be improved. Also, it is possible to suppress a decrease in the accuracy of the second judgment result due to environmental changes such as changes in lighting brightness and temperature depending on the season.

[0034] In the above embodiment, the correlation data displayed on the display device 16 is a graph as shown in FIG. 6. However, the correlation data displayed on the display device 16 may also be a matrix as shown in FIG. 9. The correlation data shown in this matrix is ​​created using training data for creating a machine learning model for numerical data. The values ​​shown in the matrix in FIG. 9 indicate the correlation coefficients between two types of elements. In FIG. 9, the correlation coefficient between the number of detected stains in the central region (inner count) and the first judgment result (judge) is 0.63, the correlation coefficient between the maximum area of ​​stains in the central region (inner max area) and the first judgment result (judge) is 0.38, the correlation coefficient between the number of detected stains in the outer region (outer count) and the first judgment result (judge) is 0.15, and the correlation coefficient between the maximum area of ​​stains in the outer region (outer max area) and the first judgment result (judge) is 0.34.

[0035] In the above embodiment, the numerical data is data related to dirt detected on the protective glass 202. However, the numerical data may also be data related to detected objects other than dirt, such as scratches or dust, detected on the protective glass 202. Furthermore, the numerical data is the number of detections and the maximum area in the central region and the outer region. However, the numerical data may also be the number of detections and the maximum area for the entire protective glass 202 or for another divided region. Alternatively, the numerical data may be the number of detections of dirt whose area falls within a predetermined numerical range. For example, dirt may be classified according to which of multiple numerical ranges its area belongs to, and the number of dirt particles belonging to each numerical range may be used as numerical data. [Industrial Applicability]

[0036] The automatic visual inspection device and visual inspection program disclosed herein can allow an operator to recognize which factors strongly influenced the judgment results when making a judgment about the appearance of protective glass using a machine learning model, and is useful as a technology for making judgments about inspection images obtained by photographing protective glass. [Explanation of symbols]

[0037] 100 Automatic visual inspection equipment 202 Protective Glass 15. Computer 16 Display device 17 Input Devices

Claims

1. A display device; generating numerical data relating to a predetermined detected object on the protective glass based on an inspection image obtained by photographing the protective glass; obtaining a first determination result based on the numerical data using a machine learning model for numerical data; and a computer that simultaneously displays on the display device the first judgment result and correlation data indicating the degree of influence of multiple types of feature amounts included in the numerical data on the first judgment result.

2. 2. The automatic visual inspection apparatus according to claim 1, The computer obtaining a second determination result based on the inspection image using an image machine learning model; an automatic visual inspection device that displays the second judgment result on the display device simultaneously with the first judgment result and the correlation data;

3. 3. The automatic visual inspection apparatus according to claim 2, The computer obtaining a comprehensive determination result based on the first determination result and the second determination result; an automatic visual inspection apparatus that displays the first judgment result, the correlation data, and the second judgment result, together with the overall judgment result, on the display device;

4. 4. The automatic visual inspection apparatus according to claim 3, The computer selecting one of the first judgment result and the second judgment result based on a prediction accuracy of the first judgment result, a prediction accuracy of the second judgment result, a first matching rate between a past judgment result by the machine learning model for numerical data and a judgment result by an operator, and a second matching rate between a past judgment result by the machine learning model for images and a judgment result by an operator; an automatic visual inspection device, characterized in that when the first judgment result and the second judgment result are simultaneously displayed on the display device, one of the judgment results is displayed larger than the other judgment result;

5. 5. The automatic visual inspection apparatus according to claim 4, an input device that receives an input indicating a determination result by the worker; The automatic visual inspection apparatus is characterized in that the computer calculates the first and second match rates based on the input to the input device.

6. The automatic visual inspection apparatus according to any one of claims 1 to 5, an input device that receives an input indicating a determination result from an operator; The computer generates a new machine learning model using the inspection image and the judgment result indicated by the input to the input device as training data.

7. generating numerical data relating to a predetermined detected object on the protective glass based on an inspection image obtained by photographing the protective glass; obtaining a first determination result based on the numerical data using a machine learning model for numerical data; An appearance inspection program that causes a computer to execute a process of simultaneously displaying on a display device the first judgment result and correlation data that indicates the degree of influence of multiple types of feature amounts included in the numerical data on the first judgment result.

Images