Image inspection device, readjustment necessity / unnecessity presentation method and readjustment necessity / unnecessity presentation program

Abstract

To cause a user to understand necessity / unnecessity of imaging condition readjustment in appearance inspection using an imaged image of a product and a learning model.SOLUTION: A first acquisition unit inputs an imaged image, in which a product being a target of appearance inspection is imaged, into a learning model to acquire an inspection result based on the learning model. A second acquisition unit acquires a variation amount of an imaging condition of the product on the basis of the imaged image. A clustering unit generates a first cluster group based on clustering that includes the inspection result and the variation amount. A calculation unit calculates a centroid error that is an error of a centroid in a second cluster group based on clustering using teacher data of the learning model and each cluster of the first cluster group. A determination unit determines necessity / unnecessity of the imaging condition readjustment on the basis of the centroid error. A presentation unit presents the necessity / unnecessity of the readjustment which is determined by the determination unit, to a user.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image inspection device, a method for indicating whether readjustment is necessary, and a program for indicating whether readjustment is necessary. [Background technology]

[0002] Conventionally, in a product inspection process, a technique has been known in which a captured image of the product is input to a learning model trained by machine learning to determine whether the product is good or bad. It is also known that such a technique can maintain inspection accuracy by retraining the learning model in response to changes in trends in the inspection results during actual operation.

[0003] For example, in the technology disclosed in Patent Document 1, whether or not to re-learn is determined based on the amount of change over time in image features extracted by inputting captured images into a learning model during actual operation, with the time immediately after learning of the learning model as the reference point. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-086149 Summary of the Invention [Problem to be solved by the invention]

[0005] However, during actual operation, the imaging conditions of the product may change over time due to factors such as the installation state of the camera, lighting conditions, etc. The above-mentioned conventional technology does not take such changes in imaging conditions into consideration.

[0006] Therefore, when the above-described conventional technology is used, even in a situation where readjustment of the imaging conditions is required rather than re-training the learning model, there is a risk that re-training will be performed based on the amount of change in image features due to a change in the imaging conditions. Furthermore, if re-training is performed, the learning model will adapt to imaging conditions that are different from the original imaging conditions, which may result in a decrease in inspection accuracy.

[0007] The present disclosure provides a technology that allows a user to understand whether or not image capture conditions need to be readjusted during visual inspection using captured images of a product and a learning model. [Means for solving the problem]

[0008] An image inspection device according to one aspect of the present disclosure includes a first acquisition unit, a second acquisition unit, a clustering unit, a calculation unit, a determination unit, and a presentation unit. The first acquisition unit inputs captured images of a product to be visually inspected into a learning model to obtain inspection results from the learning model. The second acquisition unit acquires a change in the imaging conditions of the product based on the captured images. The clustering unit generates a first group of clusters by clustering including the inspection results and the change. The calculation unit calculates a second group of clusters by clustering using training data of the learning model, and a centroid error, which is an error in the center of gravity of each cluster in the first group of clusters. The determination unit determines whether readjustment of the imaging conditions is necessary based on the centroid error. The presentation unit presents the determination of whether readjustment is necessary, as determined by the determination unit, to a user. [Effects of the Invention]

[0009] According to the present disclosure, in a visual inspection using captured images of a product and a learning model, the user can be made to understand whether or not readjustment of the imaging conditions is necessary. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an outline of a readjustment necessity presenting method according to an embodiment. [Figure 2]FIG. 2 is a block diagram showing an example of the configuration of an image inspection device according to the embodiment. [Figure 3] FIG. 3 is a diagram showing an example of items of imaging conditions. [Figure 4] FIG. 4 is a diagram showing an example of obtaining the amount of change in the position and angle of the workpiece. [Figure 5] FIG. 5 is a diagram showing an example of presentation of the amount of change in the imaging conditions. [Figure 6] FIG. 6 is a diagram showing an example of a presentation layout presented by the image inspection device. [Figure 7] FIG. 7 is a diagram showing an example of the layout of the imaging condition change amount display section. [Figure 8] FIG. 8 is a flowchart showing a processing procedure executed by the image inspection device according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an outline of a method for presenting whether or not readjustment is necessary according to a modified example. [Figure 10] FIG. 10 is a flowchart showing a processing procedure executed by an image inspection device according to a modified example. [Figure 11] FIG. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of an image inspection device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0012] In the following, an example will be given in which an image inspection device 10 (see FIG. 2) according to an embodiment of the present disclosure (hereinafter referred to as "this embodiment" where appropriate) performs an appearance inspection of a thin plate-like product such as a circuit board. Note that this example does not particularly limit the shape of the product. Also, the readjustment necessity presentation method according to this embodiment is a readjustment necessity presentation method executed by this image inspection device 10. Also, in the following, the product to be inspected will be referred to as a "workpiece" where appropriate.

[0013] The present disclosure will be described in the following order. 1. Overview 2. Example of image inspection device configuration 3. Processing Procedure 4. Variations 5. Hardware Configuration 6. Conclusion

[0014] <<1. Overview>> First, an overview of the readjustment necessity presentation method according to this embodiment will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram illustrating the overview of the readjustment necessity presentation method according to this embodiment.

[0015] Existing technologies that use learning models to inspect products do not take into account changes in imaging conditions, such as camera installation status and lighting conditions, that occur during actual operation. Therefore, when using existing technologies, there is a risk that re-training will be performed based on changes in image features due to changes in imaging conditions, even in situations where readjustment of imaging conditions is more necessary than re-training the learning model. Furthermore, if re-training is performed, the learning model will adapt to imaging conditions that are different from the original imaging conditions, which could result in a decrease in inspection accuracy.

[0016] Therefore, in the embodiment, in the method for indicating whether readjustment is necessary, the image inspection device 10 inputs captured images of a product to be visually inspected into a learning model and obtains inspection results based on this learning model. The image inspection device 10 also obtains the amount of change in the product's imaging conditions based on the captured images. The image inspection device 10 also generates a first group of clusters by clustering including the inspection results and the amount of change. The image inspection device 10 also calculates a second group of clusters by clustering using training data from the learning model, and a centroid error, which is the error in the center of gravity for each cluster in the first group of clusters. The image inspection device 10 also determines whether readjustment of the imaging conditions is necessary based on the centroid error. The image inspection device 10 also presents the determined need for readjustment to the user.

[0017] Specifically, as shown in FIG. 1, in the readjustment necessity presenting method according to the embodiment, the image inspection device 10 generates a group of clusters based on the measurement data set by clustering. These are the "first cluster group." The image inspection device 10 also generates a group of clusters based on the teacher data set by clustering. These are the "second cluster group."

[0018] The first cluster group is generated by clustering using the measurement data set, i.e., the group of captured images taken during actual operation, as input to a learning model for determining pass / fail, along with the inspection results obtained by inputting the amount of change in imaging conditions obtained based on each captured image.

[0019] On the other hand, the second cluster group is generated by clustering using the training data set, i.e., the training data group used to train the learning model, as input, and each test result obtained by inputting the training data group into the learning model. Because the training data is images captured under the original imaging conditions that serve as a reference, each test result obtained by inputting the training data group into the learning model implicitly includes features for the reference values ​​of the imaging conditions.

[0020] The image inspection device 10 then calculates the centroid error for each corresponding cluster in the first cluster group and the second cluster group. This centroid error is based on clustering using each inspection result and the amount of change in each imaging condition as input, and therefore includes as its elements a component due to the variation in the inspection results and a component due to the amount of change in the imaging conditions (hereinafter referred to as "imaging condition components" as appropriate). Note that the centroid error in this embodiment indicates the distance between the first cluster group and the second cluster group. Therefore, the centroid error can be rephrased as the "distance between clusters."

[0021] The image inspection device 10 separates these components and determines that the imaging conditions need to be readjusted if the imaging condition component exceeds a predetermined threshold. Specifically, as shown in Figure 1, the image inspection device 10 first determines that the imaging conditions may need to be readjusted if the center of gravity error exceeds a predetermined first threshold.

[0022] If the centroid error exceeds the first threshold and the ratio (n%) of the change in the imaging conditions to the centroid error exceeds a predetermined second threshold, the image inspection device 10 determines that the imaging conditions need to be readjusted.The image inspection device 10 then presents the determination result to the user.

[0023] Furthermore, when the centroid error exceeds the first threshold and the ratio of the change in the imaging conditions to the centroid error is equal to or less than the second threshold, the image inspection device 10 determines that additional learning of the learning model is required rather than readjustment of the imaging conditions.The image inspection device 10 then presents the determination result to the user.

[0024] This allows the user to understand whether a change over time in the learning model is due to a change in the imaging conditions or a trend change in the learning model, based on the centroid error of each cluster of the first cluster group relative to the second cluster group. That is, the readjustment necessity presenting method according to the embodiment allows the user to understand whether readjustment of the imaging conditions is necessary in an appearance inspection using captured images of a product and a learning model.

[0025] An example of the configuration of the image inspection device 10 to which the readjustment necessity presenting method according to this embodiment is applied will be described in more detail below.

[0026] <<2. Configuration example of image inspection device>> Fig. 2 is a block diagram showing an example of the configuration of an image inspection device 10 according to an embodiment. Note that Fig. 2 shows, in functional blocks, only components necessary for explaining this embodiment, and omits descriptions of general components.

[0027] In addition, in the description using FIG. 2, the description of components that have already been described will be appropriately simplified or omitted.

[0028] 2, the image inspection device 10 includes a storage unit 11 and a control unit 12. The image inspection device 10 is also connected to an imaging device 3 and a UI (User Interface) device 5.

[0029] The imaging device 3 is an image sensor that captures an image of a workpiece to be inspected. The imaging device 3 includes, for example, a CMOS (Complementary Metal Oxide Semiconductor). The imaging device 3 captures an image of the workpiece for each inspection target and outputs the captured image to the image inspection device 10.

[0030] The imaging device 3 is placed, for example, on a product production line. The imaging device 3 may be a wired camera that communicates with the image inspection device 10 via a wire, or may be a wireless camera that can communicate wirelessly with the image inspection device 10. The image captured by the imaging device 3 may be a color image or a monochrome image.

[0031] The UI device 5 is a device that includes an output interface that presents various information related to the appearance inspection of products to the user, such as the inspection results of the workpiece, whether or not readjustment of the imaging conditions is necessary, suggestions for additional learning, etc. The UI device 5 includes, for example, a display as an output interface. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0032] The UI device 5 also includes an input interface that accepts various operations from the user. The UI device 5 includes, for example, a keyboard, a mouse, etc. as the input interface. Note that the UI device 5 may be configured such that the output interface and the input interface are integrated, for example, by a touch panel display. In this case, the input interface may include, for example, a software component such as a GUI (Graphical User Interface).

[0033] The storage unit 11 is realized by a storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, or an HDD (Hard Disk Drive).

[0034] In the example of FIG. 2, the storage unit 11 stores a learning model 11a, a teacher data set 11b, a measurement data set 11c, inspection result information 11d, imaging condition information 11e, threshold information 11f, and clustering information 11g.

[0035] The learning model 11a is an AI (Artificial Intelligence) model for determining whether a product is good or bad. The learning model 11a is, for example, a DNN (Deep Neural Network) model trained using a deep learning algorithm.

[0036] The learning model 11a is read as a DNN model into the inspection unit 12b (described later), and is then trained in advance so that when an image captured by the imaging device 3 is input to the inspection unit 12b, the inspection unit 12b can determine whether the workpiece is good or bad based on its appearance in the captured image. That is, the inspection unit 12b that has read the learning model 11a operates as an inspection AI that inspects the appearance of a product.

[0037] Note that, although various modes are conceivable for the method of determining the quality of a product as an inspection AI, the mode is not particularly limited. For example, the inspection AI may determine the quality of a product based on image features, scores, etc. based on image-unit classification results by the learning model 11a. Also, for example, the inspection AI may determine the quality of a product based on image features, scores, classes, centroid coordinates, width, height, etc. based on rectangular object detection results by the learning model 11a. Also, for example, the inspection AI may determine the quality of a product based on image features, scores, classes, centroid coordinates, width, height, area, etc. based on pixel-unit area extraction results by the learning model 11a.

[0038] The teacher data set 11b stores a group of teacher data used when the learning model 11a was trained. The measurement data set 11c stores a group of captured images captured by the imaging device 3 during actual operation of the image inspection device 10.

[0039] The inspection result information 11d stores and accumulates the inspection results obtained by the inspection unit 12b. The imaging condition information 11e stores information including each item of the imaging conditions and a reference value for each item. An example of the items of the imaging conditions will now be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the items of the imaging conditions.

[0040] 3, the imaging conditions include the position and angle of the workpiece, brightness, white balance, focus, etc. A reference value (not shown) is set for each item. The amount of change is acquired as an error from this reference value by the change amount acquisition unit 12c (described later).

[0041] For the position of the workpiece, the amount of parallel movement (Δx, Δy, Δz) along each axis in the XYZ Cartesian coordinate system relative to a reference value is acquired as the amount of change. For the angle of the workpiece, the amount of rotation (θx, θy, θz) around each axis in the XYZ Cartesian coordinate system relative to a reference value is acquired as the amount of change. Note that specific examples of acquiring the amount of change in the position and angle of the workpiece will be described later using Figure 4.

[0042] For brightness, the error from the reference value in the entire captured image or in any area is acquired as the amount of change. For white balance, the error in the RGB ratio from the reference value is calculated as the amount of change. For focus, (brightness of the pixel of interest - brightness of the adjacent pixel) 2 The error from the reference value due to the change is acquired as the amount of change.

[0043] Returning to the explanation of Fig. 2, the threshold information 11f stores information including various thresholds used in information processing by the control unit 12. The clustering information 11g stores information related to the clustering process executed by the clustering unit 12d, which will be described later. The clustering information 11g includes the clustering results of the clustering process, etc.

[0044] The control unit 12 corresponds to a so-called processor or controller. The control unit 12 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphical Processing Unit). The control unit 12 executes a readjustment necessity presentation program according to an embodiment (not shown) stored in the storage unit 11, using RAM as a work area. The control unit 12 can also be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0045] The control unit 12 has an image acquisition unit 12a, an inspection unit 12b, a change amount acquisition unit 12c, a clustering unit 12d, a calculation unit 12e, a judgment unit 12f, and a presentation unit 12g, and realizes or executes the functions and actions of information processing described below.

[0046] The image acquisition unit 12a acquires a captured image of each workpiece captured by the imaging device 3. Furthermore, the image acquisition unit 12a stores the acquired captured image in the measurement data set 11c.

[0047] The inspection unit 12b operates as an inspection AI by reading the learning model 11a as described above. The inspection unit 12b inputs the captured image acquired by the image acquisition unit 12a to the learning model 11a and acquires the inspection result by the learning model 11a.

[0048] The change amount acquiring unit 12c acquires the amount of change in the imaging conditions based on the captured image of the workpiece. A specific example of acquiring the amount of change in the position and angle of the workpiece will now be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of acquiring the amount of change in the position and angle of the workpiece.

[0049] As shown in Figure 4, the change amount acquisition unit 12c compares the reference points P11, P12, P21, and P22 of the workpiece, which are set in the imaging condition information 11e as reference values ​​of the imaging conditions, with the corresponding measurement points Pa11, Pa12, Pa21, and Pa22 in the captured image.

[0050] Regarding the position of the workpiece, in the X-axis direction, the change amount acquiring unit 12c calculates the amount of change by comparing the x-coordinates of four points. In the Y-axis direction, the change amount acquiring unit 12c calculates the amount of change by comparing the y-coordinates of four points. In the Z-axis direction, the change amount acquiring unit 12c calculates the amount of change by comparing the lengths of the four sides.

[0051] Regarding the angle of the workpiece, for rotation around the X-axis, the change amount acquiring unit 12c calculates the amount of change by comparing the lengths of sides P11P12 and Pa11Pa12, and sides P21P22 and Pa21Pa22. For rotation around the Y-axis, the change amount acquiring unit 12c calculates the amount of change by comparing the lengths of sides P11P21 and Pa11Pa21, and sides P12P22 and Pa12Pa22. For rotation around the Z-axis, the change amount acquiring unit 12c calculates the amount of change by comparing the angles of sides P11P12 and Pa11Pa12.

[0052] The change amount acquiring unit 12c associates each calculated change amount with a corresponding item in the imaging condition information 11e, for example, and stores the amount in the imaging condition information 11e.

[0053] Returning to the description of Fig. 2, the clustering unit 12d executes clustering processing using as input the inspection results from the inspection unit 12b and the amount of change in the imaging conditions acquired by the amount-of-change acquisition unit 12c.

[0054] Here, the algorithm used by the clustering unit 12d for the clustering process is not particularly limited. The clustering unit 12d can use various clustering algorithms, such as "K-Means," "DBSCAN," "Mean Shift," "Gaussian Mixture Model," and "Agglomerative Clustering (Hierarchical clustering)."

[0055] The clustering unit 12d generates a first cluster group by clustering processing using the test results included in the test result information 11d and the changes in the imaging conditions included in the imaging condition information 11e as inputs. The clustering unit 12d also generates the second cluster group by clustering using the training data of the learning model 11a.

[0056] Specifically, for the second cluster group, the clustering unit 12d inputs the teacher data group of the teacher data set 11b into the learning model 11a and obtains each test result corresponding to the teacher data group from the learning model 11a. Then, the clustering unit 12d generates the second cluster group by performing a clustering process using each test result of this teacher data group as input.

[0057] It should be noted that the number of sampling data used to generate the first group of clusters and the number of sampling data used to generate the second group of clusters do not need to be the same.

[0058] The calculation unit 12e compares each cluster in the second cluster group with each cluster in the first cluster group, and calculates the centroid error between corresponding clusters. The calculation unit 12e also calculates imaging condition components in the centroid error. The imaging condition components in the centroid error correspond to the contribution rate of the amount of change in the imaging conditions in the centroid error. The calculation unit 12e calculates the contribution rate for each imaging condition item in the amount of change in the imaging conditions included in the centroid error, and the total value of the contribution rate.

[0059] The determination unit 12f determines whether or not readjustment of the imaging conditions is necessary based on the centroid error calculated by the calculation unit 12e. Specifically, the determination unit 12f determines that readjustment of the imaging conditions may be necessary if the centroid error exceeds a predetermined first threshold. Furthermore, the determination unit 12f determines that readjustment of the imaging conditions is unnecessary if the centroid error is equal to or less than the first threshold.

[0060] Furthermore, when the centroid error exceeds the first threshold and the total value of the aforementioned contribution rates exceeds a predetermined second threshold, the determination unit 12f determines that the imaging conditions need to be readjusted. Furthermore, when the centroid error exceeds the first threshold and the total value of the aforementioned contribution rates is equal to or less than the second threshold, the determination unit 12f determines that additional learning of the learning model 11a is needed.

[0061] The presentation unit 12g presents the determination result by the determination unit 12f to the user. Here, examples of presentations presented by the presentation unit 12g to the UI device 5 will be described with reference to Figs. 5 to 7. Fig. 5 is a diagram showing an example of presentation of the amount of change in imaging conditions. Fig. 6 is a diagram showing an example of a presentation layout presented by the image inspection device 10. Fig. 7 is a diagram showing an example of the layout of the imaging condition change amount display unit 53.

[0062] As shown in Fig. 5, the presentation unit 12g visualizes the amount of change in the imaging conditions, for example, by a graph, so that the user can understand the contribution rate of each item of the imaging conditions calculated by the calculation unit 12e. Fig. 5 shows an example in which the presentation unit 12g presents the contribution rate of each item to the amount of change in the imaging conditions in a pie chart. This allows the user to easily determine which item of the imaging conditions should be readjusted when the imaging conditions need to be readjusted.

[0063] The presentation unit 12g presents to the UI device 5 in a presentation layout such as that shown in Fig. 6 and Fig. 7, including the presentation example shown in Fig. 5. Such a presentation layout includes, for example, an image display unit 51, an examination result display unit 52, an imaging condition change amount display unit 53, and a setting unit 54, as shown in Fig. 6.

[0064] The image display unit 51 displays a captured image of the workpiece being inspected. The inspection result display unit 52 displays the inspection result of the workpiece displayed on the image display unit 51, for example, text such as "good" or "defective". The imaging condition change amount display unit 53 displays information regarding the amount of change in the imaging conditions. The setting unit 54 displays, for example, a GUI or the like that accepts various settings and other operations from the user.

[0065] As shown in FIG. 7, the imaging condition change amount display section 53 includes, for example, an adjustment necessity display section 53a, a factor analysis result display section 53b, an adjustment-requiring item display section 53c, and an additional learning suggestion section 53d.

[0066] The adjustment necessity display section 53a displays text such as "Adjustment required" when the imaging conditions need to be readjusted, and "Adjustment not required" when the imaging conditions do not need to be readjusted. The factor analysis result display section 53b displays the result of the factor analysis when the imaging conditions need to be readjusted. For example, Fig. 7 shows an example in which the presentation example shown in Fig. 5 is displayed on the factor analysis result display section 53b.

[0067] When the imaging conditions need to be readjusted, the adjustment-required item display section 53c displays, for example, items among the imaging conditions with a relatively high contribution rate. For example, the adjustment-required item display section 53c displays the top several items with contribution rates equal to or greater than a predetermined value in descending order of contribution rate. This allows the user to easily determine which imaging condition item should be prioritized for adjustment when the imaging conditions need to be readjusted.

[0068] If the position or angle of the workpiece needs to be readjusted, the user adjusts, for example, the positional relationship of the imaging device 3 or the workpiece jig. If the brightness needs to be readjusted, the user adjusts the exposure time and gain of the imaging device 3, the illumination intensity, etc.

[0069] Furthermore, if the white balance needs to be readjusted, the user may, for example, capture an image of a gray card and adjust the sensitivity of the imaging device 3 so that the RGB components in the image are uniform. Furthermore, if the focus needs to be readjusted, the user may, for example, adjust the focus ring of the lens of the imaging device 3 and the position of the imaging device 3. The adjustment-required item display unit 53c may display guidance on how to adjust each of these items.

[0070] Returning to the explanation of Fig. 6, when the determination unit 12f determines that additional learning is required, the additional learning suggestion unit 53d notifies the user that additional learning of the learning model 11a is required. That is, when the center of gravity error is larger than the first threshold but the contribution rate of the change in the imaging conditions is smaller than the second threshold, the presentation unit 12g suggests to the user that additional learning of the learning model 11a be performed.

[0071] <<3. Processing Procedure>> Next, the processing procedure executed by the image inspection device 10 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the processing procedure executed by the image inspection device 10 according to the embodiment.

[0072] First, the image acquisition unit 12a acquires captured images (step S101). Here, the image acquisition unit 12a acquires a group of captured images, i.e., a measurement data set 11c. Then, the inspection unit 12b inputs the acquired captured images to the learning model 11a and acquires inspection results by the learning model 11a (step S102).

[0073] Furthermore, the change amount acquiring unit 12c acquires the amount of change in the imaging conditions of the workpiece based on the captured image (step S103). Then, it is determined whether or not the amount of change acquired by the change amount acquiring unit 12c exceeds a predetermined value (step S104).

[0074] The amount of change compared with the predetermined value may be each amount of change obtained from each captured image, or may be, for example, the sum of each amount of change. In other words, it is sufficient if it can be roughly determined that the amount of change in the imaging conditions is large when it exceeds a predetermined value.

[0075] If the amount of change is equal to or less than the predetermined value (No in step S104), the presentation unit 12g presents the test results (step S105) and ends the process. The test results may be presented so that each captured image in the measurement data set 11c can be displayed one by one, or so that multiple images can be displayed in a list.

[0076] On the other hand, if the amount of change exceeds the predetermined value (Yes at step S104), the clustering unit 12d generates a first cluster group by clustering including the amount of change in the examination results and imaging conditions (step S106).

[0077] Furthermore, the clustering unit 12d generates a second cluster group by clustering using the teacher data, and the calculation unit 12e calculates the centroid error for each cluster in the second cluster group and the first cluster group (step S107).

[0078] Then, the determination unit 12f determines whether or not there is a centroid error exceeding the first threshold (step S108). If there is no centroid error exceeding the first threshold (step S108, No), the determination unit 12f determines that readjustment of the imaging conditions is unnecessary, and the presentation unit 12g presents the inspection result (step S109), and the process ends.

[0079] On the other hand, if there is a centroid error that exceeds the first threshold (step S108, Yes), the determining unit 12f then determines whether or not the imaging condition component of this centroid error exceeds a second threshold (step S110).

[0080] If the imaging condition component exceeds the second threshold (step S110, Yes), the determination unit 12f determines that the imaging conditions need to be readjusted (step S111). Then, the presentation unit 12g proposes readjustment of the imaging conditions (step S112), and the process ends.

[0081] On the other hand, if the imaging condition component is equal to or less than the second threshold (step S110, No), the presenting unit 12g proposes additional learning of the learning model 11a (step S113), and ends the process.

[0082] <<4. Modifications>> Although the embodiment of the present disclosure has been described so far, the readjustment necessity presenting method according to the present embodiment can be modified in several other ways.

[0083] For example, in the above-described embodiment, the center of gravity error for each cluster of the first cluster group and the second cluster group is calculated, but it is also possible to project the measurement data points for each captured image onto the cluster space of the second cluster group without generating the first cluster group.

[0084] An overview of this modified example will be described with reference to Fig. 9. Fig. 9 is an explanatory diagram of an overview of a readjustment necessity presentation method according to the modified example. As shown in Fig. 9, the readjustment necessity presentation method according to the modified example projects measurement data points of the captured image, i.e., the inspection results and the amount of change in the imaging conditions by the learning model 11a, onto the cluster space of the second cluster group, which is a cluster group based on the teacher dataset.

[0085] Then, the error amount between the center of gravity of the cluster closest to this projected measurement data point and the measurement data point is calculated as the center of gravity error, and if it exceeds the first threshold value mentioned above, it is determined that there is a possibility that the imaging conditions need to be readjusted, as in the above-mentioned embodiment.

[0086] In addition, in the method for presenting the need for readjustment in the modified example, the center of gravity error is calculated for each sample of the captured image, so if the imaging condition component of the center of gravity error is less than the second threshold, additional learning is not suggested, but the captured image is simply added to the additional learning candidates of the learning model 11a.

[0087] In addition, even if the imaging condition component of the center of gravity error exceeds the second threshold, if the cumulative value of the change in the imaging conditions over the past N samples (N is any natural number) exceeds a predetermined third threshold, it is determined that the imaging conditions need to be readjusted.

[0088] The processing procedure executed by the image inspection device 10 according to this modified example will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the processing procedure executed by the image inspection device 10 according to the modified example. Note that Fig. 10 corresponds to Fig. 8, and therefore the explanation using Fig. 10 will mainly focus on the points that are different from Fig. 8.

[0089] First, steps S101 to S105 are almost the same as those in Fig. 8, but in Fig. 10, a captured image is acquired for each sample in step S101. Then, the inspection unit 12b inputs the acquired captured image to the learning model 11a and acquires an inspection result by the learning model 11a (step S102).

[0090] Furthermore, the change amount acquiring unit 12c acquires the amount of change in the imaging conditions of the workpiece based on the captured image (step S103). Then, it is determined whether the amount of change acquired by the change amount acquiring unit 12c exceeds a predetermined value (step S104). Note that the amount of change compared with the predetermined value here is the amount of change acquired from one sample of the captured image.

[0091] If the amount of change is equal to or less than the predetermined value (step S104, No), the presenting unit 12g presents the test result (step S105) and ends the process. On the other hand, if it is determined in step S104 that the amount of change in the imaging conditions exceeds the predetermined value (step S104, Yes), the calculating unit 12e projects the test result and the measurement data points that are the amount of change in the imaging conditions onto the cluster space of the second cluster group (step S201).

[0092] Then, the calculation unit 12e calculates the amount of error between the measurement data point projected onto the cluster space and the center of gravity of the cluster closest to the measurement data point as a center of gravity error (step S202).

[0093] Then, the determination unit 12f determines whether the centroid error exceeds the first threshold value (step S108). If the centroid error is equal to or smaller than the first threshold value (step S108, No), the determination unit 12f determines that readjustment of the imaging conditions is unnecessary, and the presentation unit 12g presents the inspection result (step S109), and the process ends.

[0094] On the other hand, if the centroid error exceeds the first threshold (step S108, Yes), the determining unit 12f then determines whether or not the imaging condition component of this centroid error exceeds a second threshold (step S110).

[0095] If the imaging condition component is equal to or less than the second threshold (step S110, No), the determination unit 12f adds the captured image to the additional learning candidates of the learning model 11a (step S203), and ends the process. If the imaging condition component exceeds the second threshold (step S110, Yes), the determination unit 12f then determines whether the cumulative value of the change amount of the imaging condition (for the past N samples) exceeds a predetermined third threshold (step S204).

[0096] If the cumulative value exceeds the third threshold (Yes at step S204), the determination unit 12f determines that the imaging conditions need to be readjusted (step S111). Then, the presentation unit 12g proposes that the imaging conditions be readjusted (step S112), and the process ends.

[0097] On the other hand, if the cumulative value is equal to or less than the third threshold (No at step S204), the determining unit 12f determines that readjustment of the imaging conditions is not necessary, and the presenting unit 12g presents the examination result (step S205), and the process ends.

[0098] Furthermore, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0099] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0100] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any contradiction in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0101] <<5. Hardware Configuration>> The image inspection device 10 according to the embodiment of the present disclosure described above is realized by a computer 1000 having a configuration as shown in Fig. 11, for example. Fig. 11 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the image inspection device 10. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0102] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0103] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0104] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records at least the readjustment necessity presentation program according to this embodiment.

[0105] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0106] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disk), magneto-optical recording media such as an MO (Magneto-Optical disk), tape media, magnetic recording media, and semiconductor memories.

[0107] For example, when the computer 1000 functions as the image inspection device 10, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit 12. The secondary storage device 1400 stores the readjustment necessity presentation program according to this embodiment and data in the storage unit 11. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may acquire these programs from another device via an external network 1550.

[0108] <<6. Conclusion>> As described above, according to one embodiment of the present disclosure, the image inspection device 10 includes an inspection unit 12b, a change amount acquisition unit 12c, a clustering unit 12d, a calculation unit 12e, a determination unit 12f, and a presentation unit 12g. The inspection unit 12b corresponds to an example of a "first acquisition unit." The change amount acquisition unit 12c corresponds to an example of a "second acquisition unit." The inspection unit 12b inputs captured images of a product to be visually inspected into the learning model 11a to acquire inspection results from the learning model 11a. The change amount acquisition unit 12c acquires the amount of change in the imaging conditions of the product based on the captured images. The clustering unit 12d generates a first cluster group by clustering including the inspection results and the amount of change. The calculation unit 12e calculates a second cluster group by clustering using the training data of the learning model 11a, and a centroid error, which is the error in the centroid of each cluster in the first cluster group. The determination unit 12f determines whether the imaging conditions need to be readjusted based on the center of gravity error. The presentation unit 12g presents the determination by the determination unit 12f as to whether the imaging conditions need to be readjusted to the user. This allows the user to understand whether the imaging conditions need to be readjusted in the appearance inspection using the captured image of the product and the learning model 11a.

[0109] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0110] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0111] The present technology can be configured as follows. (1) a first acquisition unit that inputs a captured image of a product to be inspected into a learning model and acquires an inspection result based on the learning model; a second acquisition unit that acquires a change amount of the imaging condition of the product based on the captured image; a clustering unit that generates a first group of clusters by clustering the test results and the amount of change; a calculation unit that calculates a second cluster group by clustering using teacher data of the learning model and a centroid error that is an error in the centroid of each cluster of the first cluster group; a determination unit that determines whether or not readjustment of the imaging conditions is necessary based on the centroid error; a notification unit that notifies a user of the necessity of the readjustment determined by the determination unit; and An image inspection device comprising: (2) The determination unit determining that the readjustment may be necessary when the center of gravity error exceeds a first threshold value; If the center of gravity error is equal to or less than the first threshold value, it is determined that the readjustment is unnecessary. The image inspection device according to (1) above. (3) The calculation unit calculating a contribution rate of each of the inspection result and the amount of change to the center of gravity error when the center of gravity error exceeds the first threshold value; The presentation unit If the contribution rate of the amount of change exceeds a second threshold, notifying the user that the readjustment is required; If the contribution rate of the amount of change is equal to or less than the second threshold, a suggestion to perform additional learning of the learning model is presented to the user. The image inspection device according to (2) above. (4) The calculation unit calculating the contribution rate of each item included in the imaging conditions to the amount of change; The presentation unit When the readjustment is necessary, the contribution rate for each of the items is presented to the user. The image inspection device according to (3) above. (5) The presentation unit When the readjustment is necessary, the item having the relatively high contribution rate among the items is indicated to the user. The image inspection device according to (4) above. (6) The items include at least the position, angle, and brightness of the product in the captured image. The image inspection device according to (4) or (5) above. (7) The items further include white balance and focus in the captured image. The image inspection device according to (6) above. (8) A computer-implemented method for indicating whether or not readjustment is necessary, comprising: a first acquisition step of inputting a captured image of a product to be inspected into a learning model and acquiring an inspection result based on the learning model; a second acquisition step of acquiring a change amount of the imaging conditions of the product based on the captured image; a clustering step of generating a first group of clusters by clustering the test results and the amount of change; a calculation step of calculating a second cluster group obtained by clustering using the training data of the learning model and a centroid error which is an error in the centroid of each cluster of the first cluster group; a determination step of determining whether or not readjustment of the imaging conditions is necessary based on the centroid error; a presenting step of presenting to a user whether or not the readjustment is necessary, the presenting step being determined in the determining step; A method for indicating whether readjustment is necessary, including: (9) a first acquisition step of inputting a captured image of a product to be inspected into a learning model and acquiring an inspection result based on the learning model; a second acquisition step of acquiring a change amount of the imaging condition of the product based on the captured image; a clustering step for generating a first group of clusters by clustering the test results and the amount of change; a calculation step of calculating a second cluster group by clustering using the training data of the learning model and a centroid error which is an error in the centroid of each cluster of the first cluster group; a determination step of determining whether or not readjustment of the imaging conditions is necessary based on the centroid error; a presentation step of presenting to a user whether or not the readjustment is necessary, the readjustment being determined by the determination step; A program that causes a computer to execute the above to indicate whether readjustment is necessary. [Explanation of symbols]

[0112] 3. Imaging device 5 UI device 10. Image inspection equipment 11 Storage section 11a Learning Model 11b Training dataset 11c Measurement Dataset 11d Test result information 11e Imaging condition information 11f Threshold Information 11g Clustering Information 12 Control Unit 12a Image acquisition unit 12b Inspection Department (First Acquisition Department) 12c Change amount acquisition unit (second acquisition unit) 12d Clustering Department 12e Calculation part 12f Judgment section 12g presentation part 51 Image display unit 52 Test result display section 53 Imaging condition change amount display section 53a Adjustment necessity display section 53b Factor analysis result display section 53c Adjustment required item display area 53d Additional Learning Suggestions 54 Settings

Claims

1. a first acquisition unit that inputs a captured image of a product to be inspected into a learning model and acquires an inspection result based on the learning model; a second acquisition unit that acquires a change amount of an imaging condition of the product based on the captured image; a clustering unit that generates a first group of clusters by clustering the test results and the amount of change; a calculation unit that calculates a second cluster group by clustering using teacher data of the learning model and a centroid error that is an error in the centroid of each cluster of the first cluster group; a determination unit that determines whether or not readjustment of the imaging conditions is necessary based on the centroid error; a notification unit that notifies a user of the necessity of the readjustment determined by the determination unit; and An image inspection device comprising:

2. The determination unit If the center of gravity error exceeds a first threshold, it is determined that the readjustment may be required; If the center of gravity error is equal to or less than the first threshold value, it is determined that the readjustment is unnecessary. The image inspection device according to claim 1 .

3. The calculation unit calculating contribution rates of the inspection result and the amount of change to the center of gravity error when the center of gravity error exceeds the first threshold value; The presentation unit If the contribution rate of the amount of change exceeds a second threshold, notifying the user that the readjustment is required; If the contribution rate of the amount of change is equal to or less than the second threshold, a suggestion to perform additional learning of the learning model is presented to the user. The image inspection device according to claim 2 .

4. The calculation unit calculating the contribution rate of each item included in the imaging conditions to the amount of change; The presentation unit When the readjustment is necessary, the contribution rate for each of the items is presented to the user. The image inspection device according to claim 3.

5. The presentation unit When the readjustment is necessary, the item having the relatively high contribution rate among the items is indicated to the user. The image inspection device according to claim 4.

6. The items include at least the position, angle, and brightness of the product in the captured image.

6. An image inspection device according to claim 4 or 5.

7. The items further include white balance and focus in the captured image. The image inspection device according to claim 6.

8. A computer-implemented method for indicating whether or not readjustment is necessary, comprising: a first acquisition step of inputting a captured image of a product to be inspected into a learning model and acquiring an inspection result based on the learning model; a second acquisition step of acquiring a change amount of the imaging conditions of the product based on the captured image; a clustering step of generating a first group of clusters by clustering the test results and the amount of change; a calculation step of calculating a second cluster group by clustering using teacher data of the learning model and a centroid error which is an error in the centroid of each cluster of the first cluster group; a determination step of determining whether or not readjustment of the imaging conditions is necessary based on the centroid error; a presenting step of presenting to a user whether or not the readjustment is necessary, the presenting step being determined in the determining step; A method for indicating whether readjustment is necessary, including:

9. a first acquisition step of inputting a captured image of a product to be inspected into a learning model and acquiring an inspection result based on the learning model; a second acquisition step of acquiring a change amount of the image capturing condition of the product based on the captured image; a clustering step for generating a first group of clusters by clustering the test results and the amount of change; a calculation step of calculating a second cluster group by clustering using teacher data of the learning model and a centroid error which is an error in the centroid of each cluster of the first cluster group; a determination step of determining whether or not readjustment of the imaging conditions is necessary based on the centroid error; a presentation step of presenting to a user whether or not the readjustment is necessary, the readjustment being determined by the determination step; A program that causes a computer to execute the above to indicate whether readjustment is necessary.

Images