Appearance inspection device, and appearance inspection method

Abstract

To achieve a flexibly operationable appearance inspection that allows for setting an inspection corresponding to a kind of an inspection object and a desired inspection content.SOLUTION: An appearance inspection device comprises a processor that uses a machine learning model to detect defects included in an inspection object image having an inspection object imaged. A first machine learning model is more likely to detect the defect having position dependency than a second machine learning model, and the second machine learning model is more likely toto detect the defect having no position dependency than the first machine learning model. The processor is configured to determine a use ratio of the first machine learning model and second machine learning model on the basis of defect detection accuracy of the first machine learning model and second machine learning model with respect to a verification image; and uses at least one of the first machine learning model and second machine learning model to detect the defect included in the inspection object image on the basis of the use ratio.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an appearance inspection technique for detecting defects included in an inspection target image.

Background Art

[0002] Conventionally, it has been known to detect defects included in an inspection target image obtained by imaging an inspection target object. In recent years, for example, it has been considered to detect defects included in an inspection target image using a learned model obtained by training a model expressed by a deep neural network by machine learning.

[0003] Patent Document 1 discloses a prior art in which image data obtained by imaging an object is input into five learned convolutional neural networks (CNNs), and each of the CNNs performs a pass / fail determination to determine whether the object shown in the image data is a defective product. In Patent Document 1, when each of the CNNs is trained by machine learning, machine learning using mutually different image data sets is performed for each CNN, so that the image data set and the processing load increase. Further, in Patent Document 1, if there is even one CNN that determines a defective product, the final determination is made as a defective product. Therefore, while detection omission can be suppressed by preparing many CNNs, over-detection increases.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, there are various types of defects to be inspected visually, such as scratches, stains, foreign matters, misalignments, etc. Here, in this specification, for example, scratches, stains, foreign matters, cracks, and corrosion are expressed as "defects independent of position" as defects related to the surface state of the inspection object. On the other hand, for example, misalignment, inclination, and warp are expressed as "defects dependent on position" as defects where the inspection object (a part of it) is not in its original position. Depending on the characteristics of the machine learning model and the parameter design of the algorithm, which of the "defects dependent on position" and "defects independent of position" can be detected more accurately will change. Also, the inspection needs of users are diverse, and whether to detect "defects dependent on position", "defects independent of position", or both depends on what kind of inspection object the user wants to perform what kind of inspection on.

[0006] The present disclosure has been made in view of such points, and an object thereof is to provide detection performance according to the type of defect that the user wants to detect.

Means for Solving the Problem

[0007] In order to achieve the above object, the features that the present disclosure can have are, for example, as follows. One aspect of the present disclosure is an appearance inspection apparatus including a processor that detects defects included in an inspection target image in which an inspection target object is imaged, using a machine learning model. The machine learning model includes a first machine learning model and a second machine learning model. The first machine learning model is more likely to detect position-dependent defects than the second machine learning model, and the second machine learning model is more likely to detect non-position-dependent defects than the first machine learning model. The processor trains the first machine learning model and the second machine learning model using a learning image in which the inspection target object is imaged, detects defects included in a verification image in which the inspection target object is imaged, using the trained first machine learning model and the second machine learning model, determines a usage ratio of the first machine learning model and the second machine learning model based on the accuracy of defect detection of the first machine learning model and the second machine learning model with respect to the verification image, and detects defects included in the inspection target image using at least one of the first machine learning model and the second machine learning model based on the usage ratio.

Advantages of the Invention

[0008] As described above, the present disclosure determines the usage ratio of the first machine learning model that is likely to detect position-dependent defects and the second machine learning model that is likely to detect non-position-dependent defects, and based on the usage ratio, uses at least one of the first machine learning model and the second machine learning model to detect defects included in the inspection target image. Therefore, it is possible to realize flexible operation of appearance inspection according to the type of the inspection target object and the desired inspection content by setting according to the type of the inspection target object and the desired inspection content.

[0009] An appearance inspection method and an appearance inspection program that achieve the same thing as the processing realized by the above appearance inspection apparatus can also obtain the same effects as the above appearance inspection apparatus. Furthermore, in the case of a program, costs are often reduced. In a program, it is also easy to make design changes related to processing. Features that the present disclosure may include other than those described above, and the effects corresponding to those features are disclosed in this specification, the claims, or the drawings.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. It should be noted that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0012] FIG. 1 shows the functional configuration of an embodiment of the present disclosure. FIG. 2 shows defects that are each easily detectable by examples of models used in the embodiment of the present disclosure. FIG. 3 shows model output information output by the model in the embodiment of the present disclosure and defect estimation information based on the usage ratio of each model output information.

[0013] As shown in FIG. 1, the appearance inspection apparatus 100 includes, as functional units, a model realization unit 102, a usage ratio determination unit 103, and a defect estimation information generation unit 104. Here, the model realization unit 102 realizes a first machine learning model 121 and a second machine learning model 122. Hereinafter, the first machine learning model 121 and the second machine learning model 122 may be simply referred to as the first model and the second model, respectively. The appearance inspection apparatus 100 further includes, as functional units, a learned neural network 101 and a display output control unit 105. The model realization unit 102 may realize other machine learning models. The usage ratio determination unit 103 includes, as functional units, a usage ratio automatic determination unit 134, a usage ratio temporary designation reception unit 136, and a usage ratio main designation reception unit 137. The functional units of the appearance inspection apparatus 100 are realized by a processor (including, for example, a CPU or a GPU) as an information processing apparatus 501 described later.

[0014] The inspection target image is an image obtained by imaging an inspection target object. The inspection target images of the present embodiment include an inspection target image related to the defect-free inspection target object 201 shown in FIG. 4, an inspection target image related to the inspection target object 202 including a position-dependent defect, and an inspection target image related to the inspection target object 203 including a position-independent defect. The inspection target object 201 includes four parts indicated by small circles inside the outer edge of the object indicated by a large circle. In the defect-free inspection target object 201, (1) each of the parts has a shape of a predetermined small circle, (2) the distance from the center of the object to each of the parts is equal and a predetermined length, and (3) a quadrilateral having the four parts as vertices is a square. The inspection object 202 including position-dependent defects satisfies (1) among the above conditions but does not satisfy (2) and (3). Specifically, in the inspection object 202 including position-dependent defects, one part is too close to the outer edge when viewed from the normal position. The inspection object 203 including non-position-dependent defects satisfies (2) and (3) among the above conditions but does not satisfy (1). Specifically, in the inspection object 203 including non-position-dependent defects, one part has a square shape. The verification image may be an image obtained by the same method as the inspection target image. Each of the verification images may be associated with each of the verification image correct labels (also simply referred to as correct labels). The correct label is correct value information indicating whether or not there are defects in the verification image. The correct label may also include information for specifying the part where the defect exists when there are defects in the verification image. The verification image is used when automatically determining the usage ratio using the usage ratio automatic determination unit 134 as described later. The verification image is also used when manually specifying the usage ratio.

[0015] The learned neural network 101 inputs the inspection target image or the verification image and outputs a feature map. For example, the feature map is an array of feature amounts formed by arranging the feature amounts output from the output layer or the like (which may be a hidden layer) of the learned neural network 101 in the spatial dimension (and the channel dimension if it exists). Hereinafter, although an example in which the image itself (for example, the inspection target image or the verification image) is input to the first machine learning model 121 and the second machine learning model 122 is described, a feature map obtained from the image may also be input. In the present specification, the feature map includes not only the learned neural network 101 but also an image output after the inspection target image or the verification image is filtered by an edge detection filter.

[0016] The model implementation unit 102 realizes each machine learning model using the model program and model parameters corresponding to each of the first machine learning model 121 and the second machine learning model 122 stored in the storage unit. Alternatively, each model may be implemented on hardware specialized for executing the machine learning model (e.g., ASIC or FPGA). Also, the number of models realized by the model implementation unit 102 is arbitrary. Hereinafter, for ease of understanding, it is assumed that the number of models to be realized is 2. Each of the first model 121 and the second model 122 outputs model output information regarding defects included in the inspection target image and the verification image.

[0017] In the present embodiment, the first model 121 is more likely to detect defects with positional dependence than the second model 122, and the second model 122 is more likely to detect defects without positional dependence than the first model 121. FIG. 2 shows the features of the first model 121 and the second model 122. The first model 121 detects defects for each block of an image or a feature map divided into a plurality of blocks 204, and the second model 122 detects defects for each block of an image or a feature map divided into a smaller number of blocks than the first model 121. Also, the first model 121 and the second model 122 each have parameters for each block of a plurality of blocks, and detect defects for each block based on the parameters. The first model 121 and the second model 122 each divide the input to the model (the inspection target image, the verification image, or the feature map obtained from these images) into a plurality of blocks 204, and detect defects for each block 204. Here, for the same input (image or feature map) to the model, the number of blocks 204 in the first model 121 is larger than the number of blocks 204 in the second model 122. In other words, the size of the area of one block 204 in the first model 121 is smaller than the size of the area of one block 204 in the second model 122. With the above configuration, the first model 121 is more likely to detect position-dependent defects than the second model 122. For example, when an image of the inspection object 202 including a position-dependent defect is input to the model, the first model 121 is more likely to detect the deviation of a part (a small circle in FIG. 2) included in the inspection object 202 including the position-dependent defect from its normal position than the second model 122. Note that since the first model 121 is more likely to detect position-dependent defects, incorrect positioning of the inspection objects 202, 203, and 204 during imaging may lead to over-detection of defects. Therefore, when performing defect detection using the first model 121, it is preferable that measures are taken to correctly position the inspection objects 202, 203, and 204 during imaging. On the other hand, the second model 122 is more likely to detect defects that are less position-dependent than the first model 121. For example, when an image of the inspection object 203 including a non-position-dependent defect is input to the model, the second model 122 is more likely to detect the deviation of a part (a small square instead of the small circle in FIG. 2) included in the inspection object 203 including the non-position-dependent defect from its normal shape (in the example of FIG. 2, the deviation that should be a small circle but is a small square).

[0018] The model output information (also simply referred to as output information) output by each of the first model 121 and the second model 122 is numerical information (hereinafter referred to as "defect degree") indicating the degree of presumption of the presence of a defect, which is calculated based on, for example, feature amounts, as information regarding the defects included in the inspection image and the verification image. For example, the value of the defect degree for a pixel where the presence of a defect is not presumed at all is set to zero, while the value of the defect degree for a pixel where the presence of a defect is strongly presumed is set to a value with a large absolute value. The upper half of FIG. 3 schematically represents the defect degree information in the image indicated by the output information from the first model 121 and the output information from the second model 122, respectively. For example, when an image related to the inspection object 202 including position-dependent defects is used as the input to the model, the output information output by the first model 121 reflects the deviation from the normal position of one part (a small circle in FIG. 3), and the absolute value of the defect degree value becomes large at both the normal position and the position where the part has deviated. The defect degree values at other positions are close to zero. In FIG. 3, pixels with large absolute values of the defect degree values are shown by being filled in black. Also, for example, when an image related to the inspection object 203 including non-position-dependent defects is used as the input to the model, the output information output by the second model 122 reflects the deviation from the normal shape of one part (instead of a small circle in FIG. 3, a small square), and the absolute value of the defect degree becomes large at the position of the part that has deviated from the normal shape. The defect degree values at other positions are close to zero.

[0019] The defect estimation information generation unit 104 generates defect estimation information regarding the defects included in the inspection target image and the verification image based on the model output information and the usage ratio of each model output information. The defect estimation information is information regarding the defects included in the inspection target image and the verification image. For example, the defect estimation information is numerical information (defect degree) indicating the degree to which the presence of a defect in the inspection target image and the verification image is estimated. The defect degree value for a pixel where the presence of a defect is not estimated at all is set to zero, while the defect degree value for a pixel where the presence of a defect is strongly estimated is set to a value with a large absolute value.

[0020] The lower left part of FIG. 3 shows an example of the functional configuration of the defect estimation information generation unit 104. The defect estimation information generation unit 104 includes, as internal functional units, a first model usage ratio reflection unit 341, a second model usage ratio reflection unit 342, and a composition function unit 343. The first model usage ratio reflection unit 341 may obtain the value of a function (for example, the responsibility of each argument) with the defect degree value indicated by the output information and the value of the first model usage ratio as arguments for each pixel constituting the image. The second model usage ratio reflection unit 342 may obtain the value of a function (for example, the product of each argument) using, as arguments, the defect degree value indicated by the output information from the second model 122 and the value of the second model usage ratio for each pixel constituting the image. The composite function unit 343 may obtain the value of a function (for example, the sum of each argument) using, as arguments, the output value of the first model usage ratio reflection unit 341 and the output value of the second model usage ratio reflection unit 342. The output information of the composite function unit 343 is information on the defect degree of each corresponding pixel, element, or region in the defect estimation information. The defect estimation information generation unit 104 outputs the first defect degree and the second defect degree of the verification image using the first model 121 and the second model 122, respectively, obtains a plurality of usage ratio candidates for the first model 121 and the second model 122, and outputs a composite defect degree obtained by combining the first defect degree and the second defect degree based on each ratio of the plurality of usage ratio candidates. The usage ratio determination unit 103 evaluates the accuracy of the defect detection corresponding to each usage ratio candidate based on the composite defect degree, and determines the usage ratio from the plurality of usage ratio candidates based on the evaluation result of the accuracy of the defect detection. As a result, since the usage ratio that can most accurately determine the verification image can be adopted from the plurality of usage ratio candidates, it is possible to achieve both suppression of detection omission and suppression of over-detection.

[0021] The lower right part of FIG. 3 schematically shows information on the defect degree of each image included in the defect estimation information when the value of the first model usage ratio (α) is 0.0, 0.5, and 1.0, respectively. Here, since an example with two models is shown, the sum of the value of the first model usage ratio (α) and the value of the second model usage ratio is always 1, and the second model usage ratio is 1 - α. Also, the value of the first model usage ratio (α) may be, for example, 0.25 or 0.75. In the lower right part of FIG. 3, pixels with a large absolute value of the defect degree value are filled in black, and pixels with a medium absolute value of the defect degree value are represented by a black-and-white checkered pattern. For example, when the value of the first model usage ratio (α) is set to 0.0, the defect estimation information becomes substantially the same as the output information of the second model 122. When the value of the first model usage ratio (α) is set to 1.0, the defect estimation information may become substantially the same as the output information of the first model 121. When the value of the first model usage ratio (α) is set to a value greater than 0.0 and less than 1.0 (for example, 0.25, 0.5, 0.75), the defect estimation information has an intermediate value between the output information of the first model 121 and the output information of the second model 122.

[0022] The display output control unit 105 controls to display a defect degree map that visibly shows the defect estimation information. Note that the object controlled to display the defect degree map 151 may be the display or output device 507 in FIG. 5 described later. Alternatively, the object controlled to display the defect degree map 151 may be any display device (display unit) capable of communicating with the appearance inspection device 100. For example, the display output control unit 105 outputs a defect degree map indicating the defect degree of the inspection target image based on the usage ratio, and causes the display unit to display the defect degree map. The defect estimation information generation unit 104 receives, for example, via the input device 506, a change in the usage ratio from the user, and outputs an updated defect degree map indicating the defect degree of the inspection target image based on the changed usage ratio. Then, the display output control unit 105 causes the display unit to display the updated defect degree map. Thereby, it becomes possible for the user to adjust to a more appropriate usage ratio while observing the change in the defect degree map. The display output control unit 105 causes the display unit to collectively display a plurality of defect degree maps output based on the first usage ratio for each of the plurality of verification images or inspection target images. The defect estimation information generation unit 104 receives, via the input device 506, an input from the user to change the first usage ratio to the second usage ratio, outputs a plurality of updated defect degree maps based on the second usage ratio for each of the plurality of verification images or inspection target images, and the display output control unit 105 causes the display unit to collectively display them. Thereby, since it is possible to collectively confirm how the plurality of defect degree maps change according to the changed usage ratio, the convenience for the user is improved.

[0023] The display output control unit 105 causes the display unit to display the defect degree map as a heat map. The heat map reflects the output defect degree values. For example, the heat map may be such that one or more of hue, lightness, and saturation are determined according to the defect degree value. Also, after setting one or more threshold values for the defect degree, the display in the heat map may be determined according to which numerical range delimited by the threshold values the defect degree is included in. By the heat map display, the user of the appearance inspection apparatus 100 can visually grasp the degree of the numerical level of the defect degree and the distribution status of the portions where the absolute value of the defect degree is large.

[0024] The display output control unit 105 may perform control to include the numerical display of the representative value 1112 of the defect estimation information in the defect degree map. For example, it displays the ratio that the number of pixels having an absolute value of the defect degree greater than or equal to the representative value 1112 of the defect estimation information occupies in the entire defect degree map as a predetermined value (for example, several percent). Thereby, the user of the appearance inspection apparatus 100 can simply grasp the numerical value that serves as a measure of whether a defect is estimated or not. Note that the numerical display of the maximum value among the defect degrees may be included in the defect degree map instead of, or together with, the representative value 1112 of the defect estimation information.

[0025] When displaying the defect degree map, the display output control unit 105 can perform control to perform the same ratio different image list display 153 and the same image different ratio list display 154. The same ratio different image list display 153 indicates a mode in which defect degree maps corresponding to a plurality of images or feature maps having a common usage ratio are collectively displayed or output. Examples of the same ratio different image list display 153 are the display screens 1101, 1102, and 1103 of FIG. 11 described later. The same image different ratio list display 154 indicates a mode in which defect degree maps with different usage ratios corresponding to the same image or feature map are collectively displayed or output. Those who use the appearance inspection device 100 can grasp the judgment materials for determining the usage ratio during defect detection operation by utilizing the same ratio different image list display 153 and the same image different ratio list display 154.

[0026] The usage ratio determination unit 103 determines the usage ratios of the first model 121 and the second model 122. Specifically, the usage ratio determination unit 103 determines the first model usage ratio, which is the usage ratio of the output information from the first model 121, and the second model usage ratio, which is the usage ratio of the output information from the second model 122. For example, the usage ratio determination unit 103 acquires a plurality of usage ratio candidates pre-stored in the storage unit, or acquires a plurality of usage ratio candidates by receiving a specification from the user. When the defect estimation information generation unit 104 generates defect estimation information, the usage ratio is used. Depending on the setting of the usage ratio, the defect estimation information generation unit 104 can generate defect estimation information by synthesizing the output information from each model, can use the output information from the first model 121 as the defect estimation information, or can use the output information from the second model 122 as the defect estimation information. That is, the usage ratio can also set the usage ratio of the output information from a specific model to zero. Therefore, compared with an appearance inspection device that can only realize either the first model 121 or the second model 122, the performance of detecting defects included in the image will not deteriorate. Note that when the usage ratio of the output information from a specific model is set to zero, the model realization unit 102 may not realize the specific model and may not output the model output information corresponding to the specific model. Since the usage ratio determination unit 103 can determine the usage ratio, it is possible to realize a flexible operation appearance inspection according to the type of the inspection object and the desired inspection content. For example, when it occurs only in the case of the inspection object 202 including defects with position dependence, or when the inspection object 202 including defects with position dependence is treated as a defect while the inspection object 203 including defects without position dependence is not treated as a defect (in the example of FIG. 2, slight deformation of the shape of the parts shown by small circles is not a problem), an appropriate usage ratio can be obtained by increasing the first model usage ratio and decreasing the second model usage ratio. (2) When it occurs only in the case of the inspection object 203 including defects without position dependence, or when the inspection object 203 including defects without position dependence is treated as a defect while the inspection object 202 including defects with position dependence is not treated as a defect (in the example of FIG. 2, slight displacement of the parts shown by small circles is not a problem, or it is allowed that the arrangement of the parts inside the inspection object is irregular), an appropriate usage ratio can be obtained by decreasing the first model usage ratio and increasing the second model usage ratio. (3) When the defects of the inspection object can occur in both the case of the inspection object 202 including defects with position dependence and the case of the inspection object 203 including defects without position dependence, and both the inspection object 202 including defects with position dependence and the inspection object 203 including defects without position dependence are treated as defects, an appropriate usage ratio can be obtained by setting both the first model usage ratio and the second model usage ratio to a certain size. In addition, since the usage ratio determination unit 103 determines the usage ratio, flexible operation can be performed according to the type of the inspection object and the desired inspection content. Therefore, even if the number of models realized by the model realization unit 102 is small, the appearance inspection device 100 can realize the appearance inspection desired by the user of the appearance inspection device 100. If the number of models is reduced, the processing load in the appearance inspection device 100 can also be reduced. Furthermore, by the usage ratio determination unit 103 determining the usage ratio, an appearance inspection that can suppress both over-detection and detection omission in defect inspection can be realized.

[0027] The usage ratio determination unit 103 may include a usage ratio automatic determination unit 134 for automatically determining the usage ratio. The usage ratio automatic determination unit 134 determines the usage ratio based on the defect detection accuracy in the defect estimation information corresponding to a plurality of usage ratios when the verification image or the feature map obtained from the verification image is input to each model. More specifically, for each of the usage ratio candidates, based on the defect estimation information generated by the defect estimation information generation unit 104, the usage ratio automatic determination unit 134 obtains the defect detection accuracy corresponding to each of the usage ratio candidates. Here, the usage ratio candidates may be stored in the appearance inspection apparatus 100, may be accessible from the appearance inspection apparatus 100, or may be input to the appearance inspection apparatus 100. Note that the input of the usage ratio candidates may be manually performed by a person using the appearance inspection apparatus 100. Also, the defect detection accuracy may be the accuracy rate, F-value, separation degree, or a combination of two or more of the accuracy rate, F-value, and separation degree as shown in FIG. 8 described later. The usage ratio automatic determination unit 134 determines the usage ratio based on the defect detection accuracy obtained for each of the usage ratio candidates.

[0028] The usage ratio determination unit 103 may include a usage ratio temporary designation reception unit 136. The usage ratio temporary designation reception unit 136 is mainly used to display the defect degree map that serves as the basis for the determination when the usage ratio during defect detection operation is manually determined. The usage ratio temporary designation reception unit 136 receives an input for designating the usage ratio in order for the defect estimation information corresponding to the defect degree map to be generated. The usage ratio temporary designation reception unit 136 notifies the defect estimation information generation unit 104 of the received usage ratio information. Upon receiving the notification of the usage ratio, the defect estimation information generation unit 104 generates defect estimation information based on the usage ratio and the output information from each model when the inspection target image or verification image (or feature map) is input to the model. The defect estimation information generation unit 104 transmits the generated defect estimation information to the display output control unit 105. The display output control unit 105 that has received the defect estimation information causes the display unit to display a defect degree map based on the received defect estimation information. As described above, since the usage ratio temporary designation reception unit 136 receives an input for designating the usage ratio and a defect degree map corresponding to the usage ratio is displayed, a person using the appearance inspection apparatus 100 can obtain a reference for determining the usage ratio during defect detection operation.

[0029] The usage ratio determination unit 103 may include a usage ratio final designation reception unit 137. The usage ratio final designation reception unit 137 receives information on the usage ratio determined and input by a person using the appearance inspection apparatus 100 when the usage ratio during defect detection operation is manually determined. A person using the appearance inspection apparatus 100 can adjust the usage ratio during defect detection operation as intended by using the usage ratio final designation reception unit 137. In addition, if a person using the appearance inspection apparatus 100 visually confirms the defect degree map that will be displayed by the usage ratio temporary designation reception unit 136, the defect estimation information generation unit 104, and the display output control unit 105, and then manually determines the usage ratio during defect detection operation using the usage ratio final designation reception unit 137, the person using the appearance inspection apparatus 100 can adjust the usage ratio during defect detection operation based on appropriate reference materials.

[0030] Since the appearance inspection apparatus 100 in the present disclosure has the functional configuration as described above, it can achieve the effects shown in the above [Effects of the Invention].

[0031] FIG. 4 shows the system configuration of the entire system including the appearance inspection apparatus 100 which is an embodiment of the present disclosure. An imaging device 401 that images inspection objects 201, 202, or 203 that are the objects to be inspected for defect inspection and an appearance inspection device 100 according to an embodiment of the present disclosure can communicate with each other. In FIG. 4, it appears as if the appearance inspection device 100 and the imaging device 401 can communicate with each other by a wired connection, but they may communicate with each other wirelessly. The imaging device 401 transmits the captured image data (inspection target image) to the appearance inspection device 100. During defect detection operation, the appearance inspection device 100 determines whether there are defects included in the inspection target image received from the imaging device 401 (or the inspection objects 201, 202, or 203 imaged in the inspection target image). Note that the system configuration shown in FIG. 4 is merely an example. The mode in which the image data (inspection target image) obtained by the imaging device 401 is transmitted to the appearance inspection device 100 is not limited to real time or a delay degree according to real time. That is, the image data (inspection target image) captured by the imaging device 401 may be given to the appearance inspection device 100 in a batch. When performing such batch processing, the imaging device 401 and the appearance inspection device 100 do not necessarily need to be able to communicate with each other. The imaging device 401 may be integrated with the appearance inspection device 100. What is integrated with the imaging device 401 and the appearance inspection device 100 may be, for example, a smartphone with a camera or what is called a so-called smart camera. Separate from the appearance inspection device 100, there may be some display device or output device for displaying or outputting the defect degree map 151. Separate from the appearance inspection device 100, there may be a display device for displaying the setting screen 1000 in FIG. 10 and some input device for receiving an input using the setting screen 1000.

[0032] FIG. 5 shows a computer architecture for realizing an embodiment of the present disclosure. To implement the appearance inspection apparatus 100, an information processing apparatus (e.g., a CPU, which may be one or more processors), a storage device (e.g., a memory) 502, a non-volatile recording medium (e.g., a non-volatile memory) 503, an external recording medium drive 504, an input device (e.g., a mouse, a keyboard, an imaging device, a sensor, a touch panel, a pointing device) 506, a display or output device (e.g., a display) 507, a communication device (e.g., a communication device for wired communication) 508, an external input / output port 509, and part or all of an imaging device connection port 510 may be interconnected by an interconnecting section (e.g., a bus) 511. The non-volatile recording medium 503 may record one or more of a functional unit program group 521 (e.g., a program for implementing the functional configuration according to the present disclosure), a model program group 522, a model parameter group 523, image data, etc. 524 (image data, feature maps, model output information, defect estimation information, defect degree maps, etc.), various databases 525 (in the drawings, the database is abbreviated as "DB"), and various information 526. Instead of recording the information shown in 521 to 526 above on the non-volatile recording medium 503, part or all of the information shown in 521 to 526 above may be acquired (accessed) by the appearance inspection apparatus 100 from the outside as shown in FIG. 5.

[0033] Hereinafter, the processing of the embodiments of the present disclosure (in addition, the detailed content of some functional configurations) will be described along the flow of some processing that can be performed in the embodiments of the present disclosure. Note that it is not essential to implement all of the functional configurations described below and perform all of the processing. An appearance inspection apparatus, an appearance inspection method, and an appearance inspection program for implementing a part of the functional configurations described below and performing a part of the processing may also be used.

[0034] FIG. 6 shows a flowchart of the processing when the appearance inspection apparatus 100 is operated to detect a defect included in an inspection target image by using the inspection target image obtained by imaging the inspection target object or a feature map obtained from the inspection target image as inputs to a first model 121 and a second model 122. In step 602 of FIG. 6, the appearance inspection apparatus 100 determines, for each of the first model 121 and the second model 122, whether the input to the model is a feature map obtained from the inspection target image. If the determination result in step 602 is affirmative, the control transitions to step 603. If the determination result in step 602 is negative, the control transitions to step 606. Note that the determination in step 602 may be performed separately for each of the first model 121 and the second model 122. Also, although it was described above that the conditional branch determination is made in step 602, if it is clear in the appearance inspection apparatus 100 whether the input to the model is the inspection target image itself or a feature map obtained from the inspection target image due to the specifications of the first model 121 or the second model 122, step 602 may be omitted and the flowchart in FIG. 6 may start from step 603 or step 606. In step 603, the learned neural network 101 takes in the inspection target image as the input to the learned neural network 101. In step 604, the learned neural network 101 outputs a feature map obtained from the inspection target image. In step 605, the first model 121 and the second model 122 each receive the input of the feature map. Alternatively, in step 606, the first model 121 and the second model 122 each receive the input of the inspection target image.

[0035] In step 607 of FIG. 6, each of the first model 121 and the second model 122 realized by the model realization unit 102 outputs the first model output information and the second model output information respectively. As already pointed out, the first model output information and the second model output information may be numerical information (defect degree) indicating the degree of presence of defects in each of the pixels, elements or regions constituting the input to the model (inspection target image or feature map). While the first model output information is likely to reflect defects with position dependence, the second model output information may be likely to reflect defects without position dependence. Steps 605, 606, and 607 above may be collectively referred to as the "model realization step". In step 610 of FIG. 6, the defect estimation information generation unit 104 generates defect estimation information based on the first model output information, the second model output information, and the usage ratio. This step 610 may be called the "defect estimation information generation step".

[0036] In step 611 of FIG. 6, the defect estimation information generation unit 104 (or any functional unit included in the appearance inspection apparatus 100) may determine whether or not the inspection target image includes a defect based on the defect estimation information. For example, if the defect estimation information is the degree of defect in each pixel, element, or region constituting the input to the model (inspection target image or feature map), the defect estimation information generation unit 104 (or any functional unit included in the appearance inspection apparatus 100) may determine that the inspection target image (or the inspection target object imaged in the inspection target image) includes a defect when the maximum value of the degree of defect for each pixel, element, or region included in the defect estimation information exceeds a predetermined threshold (or is equal to or greater than the predetermined threshold). Alternatively, the defect estimation information generation unit 104 (or any functional unit included in the appearance inspection apparatus 100) may determine that the inspection target image (or the inspection target object imaged in the inspection target image) includes a defect when a representative defect estimation value 1112 derived from the defect estimation information (which has already been described, but the representative defect estimation value 1112 may be defined, for example, such that the ratio of the number of pixels, elements, or regions having an absolute value of the degree of defect equal to or greater than the representative defect estimation value 1112 to the whole is a predetermined value (e.g., several percent)) exceeds a predetermined threshold (or is equal to or greater than the predetermined threshold). Note that if the role required of the appearance inspection apparatus 100 is only to generate defect estimation information as numerical information (degree of defect) indicating the degree of presence of a defect estimated in each pixel, element, or region constituting the input to the model (inspection target image or feature map), and the determination process of whether or not the inspection target image (or the inspection target object imaged in the inspection target image) includes a defect based on the defect estimation information is not required as a role of the appearance inspection apparatus 100, then this step 611 becomes unnecessary.

[0037] As already pointed out, before the process shown in the flowchart of FIG. 6 in which the appearance inspection apparatus 100 is operated to detect defects included in the inspection target image (or the inspection target object imaged in the inspection target image) is started, if the usage ratios (the first model usage ratio, the second model usage ratio) are appropriately determined, it is possible to realize a flexible operation appearance inspection according to the type of the inspection target object and the desired inspection content by setting. Hereinafter, in order to appropriately determine the usage ratios (the first model usage ratio, the second model usage ratio), the process when automatically determining the usage ratios and the process when manually determining the usage ratios will be described.

[0038] FIG. 7 shows a flowchart of the process for automatically determining the usage ratios in the case where the process for automatically determining the first model usage ratio and the second model usage ratio is performed prior to the process shown in the flowchart of FIG. 6. In the process shown in the flowchart of FIG. 7, after a set of verification images associated with correct labels is prepared in advance, the appearance inspection apparatus 100 generates defect estimation information for each combination of the verification images and the usage ratio candidates in steps 701 to 710. Then, the appearance inspection apparatus 100 calculates the defect detection accuracy for each usage ratio candidate in step 712. The appearance inspection apparatus 100 determines one usage ratio candidate based on the defect detection accuracy in step 714. Hereinafter, the steps shown in FIG. 7 will be described step by step.

[0039] In step 701 of FIG. 7, the usage ratio automatic determination unit 134 (or any functional unit included in the appearance inspection apparatus 100) selects one of the verification images prepared in advance. In steps 702 to 706 of FIG. 7, the functional units (for example, the learned neural network 101, the model realization unit 102) included in the appearance inspection apparatus 100 perform the same processes as steps 602 to 606 of FIG. 6. In step 707 of FIG. 7, each of the first model 121 and the second model 122 outputs output information. In step 708 of FIG. 7, the usage ratio automatic determination unit 134 determines whether all the verification images to be selected in the process shown in the flowchart of FIG. 7 have been selected in step 701. If the determination result in step 708 is affirmative, the control proceeds to step 709. If the determination result in step 708 is negative, the control returns to step 701, and the process continues for the next selected verification image. In step 709 of FIG. 7, the usage ratio automatic determination unit 134 selects one of the usage ratio candidates. In step 710 of FIG. 7, the defect estimation information generation unit 104 generates defect estimation information for each verification image based on each output information for each verification image and the usage ratio candidate selected in step 709.

[0040] In step 711 of FIG. 7, the defect estimation information generation unit 104 (or any functional unit of the appearance inspection apparatus 100) determines whether each verification image contains a defect based on the defect estimation information for each verification image. For example, when the maximum value among the defect degrees indicated by the defect estimation information is equal to or greater than a predetermined threshold, it is determined that the verification image contains a defect.

[0041] In step 712 of FIG. 7, the usage ratio automatic determination unit 134 calculates the defect detection accuracy corresponding to the usage ratio candidate selected in step 709. For calculating the defect detection accuracy, the defect estimation information for each verification image generated in step 710 and the determination information on the presence or absence of defects for each verification image determined in step 711 are used. FIG. 8 shows the process performed in step 712 (and step 714) when the usage ratio automatic determination unit 134 uses any one of the accuracy rate, F-value, and separation degree, or a combination of two or more as the defect detection accuracy.

[0042] The F-value is the harmonic mean of the precision rate and the recall rate, calculated for the combination of the correct value regarding the presence or absence of a defect indicated by the verification image correct label and the predicted value regarding the presence or absence of a defect based on the defect estimation information for each verification image. The method for calculating the F value will be specifically described using the upper left part of FIG. 8. Let the number of verification images in which the correct label indicates "correct value: there is a defect" and the result (predicted value) determined in step 711 based on the defect estimation information indicates "predicted value: there is a defect" be N_TP. Similarly hereinafter, let the number of verification images in which the correct label indicates "correct value: no defect" and the predicted value indicates "predicted value: there is a defect" be N_FP. Let the number of verification images in which the correct label indicates "correct value: there is a defect" and the predicted value indicates "predicted value: no defect" be N_FN. Let the number of verification images in which the correct label indicates "correct value: no defect" and the predicted value indicates "predicted value: no defect" be N_TN. Here, the precision X is calculated as N_TP / (N_TP + N_FP). The precision X indicates the ratio of the verification images in which the correct label indicates "there is a defect" as the correct value among the verification images determined to be "there is a defect" in step 711 based on the defect estimation information. Also, the recall Y is calculated as N_TP / (N_TP + N_FN). The recall Y indicates the ratio of the verification images determined to be "there is a defect" in step 711 based on the defect estimation information among the verification images in which the correct label indicates "there is a defect" as the correct value. The F value is calculated as 2XY / (X + Y), which is the harmonic mean of the precision X and the recall Y.

[0043] The separation degree may be the separation degree in the cumulative histogram 821 based on the defect estimation information between the set of verification images in which the correct label indicates "there is a defect" and the set of verification images in which the correct label indicates "no defect". The separation degree will be described using the upper right part of FIG. 8. In step 710, defect estimation information is generated for each verification image. For example, by using the maximum value of the defect estimation information as the value on the horizontal axis in the cumulative histogram 821, each of the verification images is conceptually arranged on the graph and divided into a set of verification images in which the correct label indicates "there is a defect" and a set of verification images in which the correct label indicates "no defect". The separation degree is the value obtained by subtracting the maximum value of the set of verification images in which the correct label indicates "no defect" from the minimum value of the set of verification images in which the correct label indicates "there is a defect". The more appropriate the estimation of the defect by the defect estimation information is, the larger this separation degree value should be. When the estimation of defects based on defect estimation information is not appropriate or sufficient, the maximum value of the set of verification images with the correct label indicating "no defect" is larger than the minimum value of the set of verification images with the correct label indicating "defect", and the value of the separation degree may become negative. In this case, it cannot be said that the two sets are properly separated.

[0044] The defect detection accuracy calculated in step 712 is associated with the usage ratio candidate selected in step 709. By using the above-mentioned accuracy rate, F value, and separation degree, the accuracy of the estimation of the presence or absence of defects based on the defect estimation information generated corresponding to the usage ratio candidate can be appropriately quantified.

[0045] In step 713 of FIG. 7, the usage ratio automatic determination unit 134 determines whether all the usage ratio candidates to be selected have been selected in step 709. If the determination result in step 713 is affirmative, the control transfers to step 714. If the determination result in step 713 is negative, the control transfers to step 709, and the process continues for the next usage ratio candidate to be selected. In step 714 of FIG. 7, the usage ratio automatic determination unit 134 determines the usage ratio when the process shown in the flowchart of FIG. 6 is executed. The usage ratio automatic determination unit 134 determines the usage ratio candidate with relatively high defect detection accuracy as the usage ratio (the first model usage ratio, the second model usage ratio) when the process shown in the flowchart of FIG. 6 is executed. Step 714 is explained using the lower right part of FIG. 8. When reaching step 714, as indicated by the "×" marks, the defect detection accuracy for each usage ratio candidate is obtained. The usage ratio automatic determination unit 134 selects the usage ratio candidate indicated by the "×" mark with the highest value or a value equivalent to the highest among these "×" marks.

[0046] By executing the process shown in the flowchart of FIG. 7 above, the usage ratios (the usage ratio of the first model, the usage ratio of the second model) when the process shown in the flowchart of FIG. 6 is executed can be automatically and reasonably determined.

[0047] Prior to the execution of the process shown in the flowchart of FIG. 6, a process of manually determining the usage ratios (the usage ratio of the first model, the usage ratio of the second model) can also be executed. First, a person using the appearance inspection device 100 tentatively specifies several usage ratios (the usage ratio of the first model, the usage ratio of the second model). The appearance inspection device 100 generates defect estimation information for each verification image corresponding to the tentatively specified usage ratios. Next, the appearance inspection device 100 displays a defect degree map 151 corresponding to the generated defect estimation information. Then, a person using the appearance inspection device 100 who has viewed the defect degree map 151 specifies (this specification) the usage ratio when the appearance inspection device 100 executes the process shown in the flowchart of FIG. 6. Hereinafter, the above-described procedure will be explained step by step. FIG. 9 shows a flowchart of the process until the appearance inspection device 100 is controlled to display or output the defect degree map 151 when a process of manually determining the usage ratios (the usage ratio of the first model, the usage ratio of the second model) is executed prior to the execution of the process shown in the flowchart of FIG. 6.

[0048] In step 901 of FIG. 9, the usage ratio determination unit 103 selects one of the inspection target images or verification images for which the corresponding defect degree map 151 is to be displayed. In steps 902 to 906 of FIG. 9, the same processes as those in steps 602 to 606 of FIG. 6 are performed on the inspection target image or verification image selected in step 901. In step 907 of FIG. 9, each of the first model 121 and the second model 122 realized by the model realization unit 102 outputs output information. In step 908 of FIG. 9, the usage ratio determination unit 103 determines whether all of the images have been selected in step 901. If the determination result in step 908 is affirmative, the control proceeds to step 915. If the determination result in step 908 is negative, the control returns to step 901, and the processing for the next selected image continues.

[0049] In step 915 of FIG. 9, the provisional usage ratio specification reception unit 136 receives input of settings for displaying the defect degree map 151 using a setting screen 1000 as illustrated in FIG. 10.

[0050] On the setting screen 1000, either "defect degree map display" or "usage ratio determination during operation" is selectively selected. When "defect degree map display" is selected, the provisional usage ratio specification reception unit 136 causes the defect estimation information generation unit 104 and the display output control unit 105 to generate defect estimation information and the defect degree map 151 and to perform display output control of the defect degree map 151 based on the information of the input made using the setting screen 1000. When "usage ratio determination during operation" is selected, the actual usage ratio specification reception unit 137 determines the usage ratio when the processing shown in the flowchart of FIG. 6 is executed based on the information of the input made using the setting screen 1000.

[0051] On the setting screen 1000, either "display list of defect degree maps with the same usage ratio" or "display list of defect degree maps of the same image" is selectively selected.

[0052] When "display list of defect degree maps with the same usage ratio" is selected, the different images with the same ratio list display 153 is used. The different images with the same ratio list display 153 collectively displays the defect degree maps 151 for some different images that have the same provisionally specified usage ratio. When "display list of defect degree maps of the same image" is selected, the same image different ratios list display 154 is used, and the defect degree maps 151 corresponding to different usage ratios for the same image are collectively displayed.

[0053] The usage ratio specifying column is for inputting the value of the usage ratio of the first model temporarily specified for displaying the defect degree map 151. It may be in such a manner that one can be selected from several candidate values that can be taken as the value of the usage ratio of the first model. The image specifying column is for inputting the path name indicating the image to be the target for displaying the defect degree map 151 by the input device 506 or the like.

[0054] According to the setting screen 1000 as shown in FIG. 10, regarding the display output method (list mode) of the defect degree map 151, the usage ratio temporarily specified for displaying or outputting the defect degree map 151, and the method of specifying and selecting the image used for displaying the defect degree map 151, it becomes easier for the person using the appearance inspection device 100 to understand.

[0055] In step 916 of FIG. 9, it is determined what the list display mode is. When the same ratio different image list display 153 is selected, the control transitions to step 917. When the same image different ratio list display 154 is selected, the control transitions to step 920.

[0056] In step 917 of FIG. 9, the defect estimation information generation unit 104 generates defect estimation information based on the input received by the usage ratio temporary specification reception unit 136. In step 918 of FIG. 9, the display output control unit 105 displays the defect degree map 151 corresponding to the specified image in the same ratio different image list display 153 based on the defect estimation information generated in step 917.

[0057] FIG. 11 shows the same ratio different image list display 153. The display screen shows a plurality of defect degree maps 151. Here, the plurality of defect degree maps 151 included in the display screen 1101 are defect degree maps 151 corresponding to a plurality of images using the same usage ratio. The defect degree map 151 may include a heat map 1111.

[0058] The display screen 1101 has a user interface for re - executing the same - image different - ratio list display 153 after changing the image that displays the defect degree map 151 or the usage ratio used when displaying the defect degree map 151. The first - model usage ratio specification field 1123 is for inputting the first - model usage ratio that is to be applied when re - executing the same - image different - ratio list display 153. FIG. 11 shows an example in which, on each display screen, only the usage ratio is changed without changing the target image, and the same - image different - ratio list display 153 is re - executed. Specifically, it shows each display screen when the first - model usage ratio is switched to 0.0, 0.5, and 1.0. As described above, the person using the appearance inspection device 100 can smoothly consider the determination of the usage ratio used when the process shown in the flowchart of FIG. 6 is performed while visually checking the same - image different - ratio list display 153 with various specifications of images and usage ratios.

[0059] In step 920 of FIG. 9, the defect estimation information generation unit 104 generates defect estimation information based on the input received by the usage ratio temporary specification reception unit 136. In step 921 of FIG. 9, the display output control unit 105, based on the defect estimation information generated in step 920, displays the defect degree map 151 corresponding to each of the specified usage ratios (first - model usage ratio, second - model usage ratio) for the specified image in the same - image different - ratio list display 154.

[0060] FIG. 12 shows the same - image different - ratio list display 154. The display screen shows a plurality of defect degree maps 151. The plurality of defect degree maps 151 are defect degree maps 151 corresponding to different usage ratios for the same image.

[0061] The display screen has a user interface for re - executing the same - image different - ratio list display 154 after changing the image that is the target for displaying the defect degree map 151 or the usage ratio used when displaying the defect degree map 151. The first model usage ratio specification column 1223 is for inputting the first model usage ratio to be applied when re-executing the same image different ratio list display 154. FIG. 12 shows an example of re-executing the same image different ratio list display 154 by changing only the image without changing the usage ratio. As described above, the user of the appearance inspection apparatus 100 can smoothly consider the determination of the usage ratio used when the process shown in the flowchart of FIG. 6 is executed while visually confirming the same image different ratio list display 154 with various specifications of the image and the usage ratio.

[0062] As described above, since the user of the appearance inspection apparatus 100 can manually determine the usage ratio used when executing the process shown in the flowchart of FIG. 6, the usage ratio can be adjusted after actually visually confirming the defect degree map 151.

[0063] An example of training (machine learning) of the model for constructing each of the models used in the embodiment of the present disclosure will be described. FIG. 13 shows the functional configuration during machine learning. The process of training the model by machine learning and the process of defect detection using the learned model may be performed in the same system or in separate systems. What kind of image is suitable as the learning image depends on the machine learning method. For example, if the model is for detecting defects included in the image and the model is trained so that the feature amount of the defect-free image can be reproduced in the output of the model, the learning image is an image obtained by imaging an object without defects. Defect-free images are easier to collect than defective images. It is also possible to use images including defects for the learning image. When training a model by machine learning, the input to the model can be commonly applied to the first model 121 and the second model 122. That is, each of the first model 121 and the second model 122 is trained by inputting the same learning image or a feature map obtained from the learning image to each other during the execution of machine learning. In this way, since the same learning image can be commonly used for the training of a plurality of models, the total number of learning images can be suppressed. Based on the output information of each model, the learning control unit 1307 performs training so that each model exhibits a desirable behavior. The training is, for example, an update calculation of parameters (model parameters) of a neural network included in each model.

[0064] FIG. 14 shows a flowchart of processing when training a model by machine learning. In step 1401 of FIG. 14, the learning control unit 1307 selects one of the learning images to be used for machine learning. In steps 1402 to 1406 of FIG. 14, the same processing as in steps 602 to 606 of FIG. 6 is performed. In step 1407 of FIG. 14, the first model 121 and the second model 122 realized by the model realization unit 102 each output output information. The output information from the first model 121 is likely to reflect a defect with positional dependency, while the output information from the second model 122 is likely to reflect a defect without positional dependency. In step 1423 of FIG. 14, the learning control unit 1307 obtains the value of the loss function in the first model 121 and the value of the loss function in the second model 122 based on the output information from each model. For example, the loss function is based on the square of the difference between the actual value of the output of the neural network included in the model and the desirable value as the value of the output. In step 1424 of FIG. 14, the learning control unit 1307 determines whether all of the learning images to be used for machine learning have been selected in step 1401. If the determination result in step 1424 is affirmative, the control proceeds to step 1425. If the determination result in step 1424 is negative, the control returns to step 1401, and the process continues for the next selected learning image 3. In step 1425 of FIG. 14, the learning control unit 1307 controls to train the first model 121 and the second model 122. For example, the learning control unit 1307 controls to perform an update calculation of the parameters (model parameters) of the neural network included in the model so that the value (absolute value) of the loss function approaches zero.

[0065] The present disclosure is not limited to the above-described embodiments and includes various modifications. A part of the configuration and processing of the embodiments may be replaced with the configuration and processing of other conceivable embodiments. The configuration and processing of the embodiments may be added with the configuration and processing of other conceivable embodiments. For example, in the present disclosure, there may be the following modification examples of the embodiments.

[0066] (α) Combined use of automatic designation and manual designation In the description of the above embodiments, regarding the method for determining the usage ratio (the first model usage ratio, the second model usage ratio), the automatic method described with reference to FIGS. 7 and 8 and the manual method described with reference to FIGS. 9, 10, 11, and 12 were shown as separate ones. However, the above-described automatic method and the above-described manual method may be used in combination. For example, the appearance inspection apparatus 100 may first obtain approximate values of the usage ratios (the usage ratio of the first model, the usage ratio of the second model) by an automatic method using the verification image group. Next, the appearance inspection apparatus 100 may be controlled to display the defect degree map 151 corresponding to the usage ratios (the usage ratio of the first model, the usage ratio of the second model) which are values in the vicinity of the above-described approximate values in a manual method using the inspection target image group and the verification image group. Then, fine adjustment of the usage ratios (the usage ratio of the first model, the usage ratio of the second model) may be performed using the display or output of the defect degree map 151. In the case of the above-described modification, it can be expected that the time until obtaining the optimum value as the usage ratio (the usage ratio of the first model, the usage ratio of the second model) is shortened.

[0067] (β) Mode of display or output of the defect degree map 151 In the description of the above embodiment, the mode of display or output of the defect degree map 151 was the same ratio different image list display 153 shown in FIG. 11, or the same image different ratio list display 154 shown in FIG. 12. However, the mode of display or output of the defect degree map 151 is not limited to the above-described same ratio different image list display 153 and the same image different ratio list display 154. For example, a mode of displaying or outputting one defect degree map 151 may be used. Also, a mode of designating a plurality of usage ratios and a plurality of images (inspection target images or verification images) and collectively displaying or outputting the defect degree map 151 for each combination of the usage ratio and the image (inspection target image or verification image) may be used. In the case of the above-described modification, it is possible to perform the display or output of the defect degree map 151 according to the circumstances of the person using the appearance inspection apparatus 100.

[0068] The technical matters shown in each of the above-described embodiments of the present disclosure and the modifications of the embodiments can be appropriately combined as long as no technical contradiction occurs.

Claims

1. An appearance inspection apparatus including a processor that detects a defect included in an inspection target image in which an inspection target object is imaged, using a machine learning model, wherein the machine learning model includes a first machine learning model and a second machine learning model, the first machine learning model is more likely to detect a defect having position dependence than the second machine learning model, the second machine learning model is more likely to detect a defect having no position dependence than the first machine learning model, the processor, trains the first machine learning model and the second machine learning model using a learning image in which the inspection target object is imaged, detects a defect included in a verification image in which the inspection target object is imaged, using the trained first machine learning model and the second machine learning model, determines a usage ratio of the first machine learning model and the second machine learning model based on the accuracy of defect detection of the first machine learning model and the second machine learning model with respect to the verification image, and detects a defect included in the inspection target image using at least one of the first machine learning model and the second machine learning model based on the usage ratio. Appearance inspection apparatus.

2. The first machine learning model detects a defect for each block of an image or a feature map divided into a plurality of blocks, and the second machine learning model detects a defect for each block of the image or the feature map divided into a smaller number of blocks than the first machine learning model. The appearance inspection apparatus according to claim 1.

3. The first machine learning model and the second machine learning model each have parameters for each block of the plurality of blocks, and detect a defect for each block based on the parameters. The appearance inspection apparatus according to claim 2.

4. The processor, outputs a first defect degree of the verification image using the first machine learning model, outputs a second defect degree of the verification image using the second machine learning model, acquires a plurality of usage ratio candidates of the first machine learning model and the second machine learning model, outputs a combined defect degree obtained by combining the first defect degree and the second defect degree based on each ratio of the plurality of usage ratio candidates, evaluates the accuracy of defect detection corresponding to each usage ratio candidate based on the combined defect degree, and determines the usage ratio from the plurality of usage ratio candidates based on the evaluation result of the accuracy of defect detection. The appearance inspection device according to claim 1.

5. The processor acquires the plurality of usage ratio candidates pre-stored in the storage unit, or acquires the plurality of usage ratio candidates by receiving a designation from a user. The appearance inspection device according to claim 4.

6. Each of the verification images is associated with a correct label regarding the presence or absence of a defect. The accuracy of the defect detection is The accuracy rate of the predicted value regarding the presence or absence of a defect for the verification image. The F value, which is the harmonic mean of the precision rate and the recall rate, calculated for the combination of the correct value regarding the presence or absence of a defect indicated by the correct label for the verification image and the predicted value regarding the presence or absence of a defect based on the defect estimation information. Calculated based on at least any one of the separation degrees in the histogram between the set of verification images having no defect and the set of verification images having a defect. The appearance inspection device according to claim 1.

7. The processor Based on the usage ratio, causes the display unit to display a defect degree map indicating the defect degree of the verification image. When receiving a change in the usage ratio from a user, outputs an updated defect degree map indicating the defect degree of the verification image based on the changed usage ratio. Causes the display unit to display the updated defect degree map. The appearance inspection device according to claim 1.

8. The processor For each of the plurality of verification images, causes the display unit to collectively display a plurality of defect degree maps output based on a first usage ratio. Is configured to be able to receive an input from a user to change the first usage ratio to a second usage ratio. For each of the plurality of verification images, causes the display unit to collectively display a plurality of updated defect degree maps output based on the second usage ratio. The appearance inspection device according to claim 7.

9. Each of the first machine learning model and the second machine learning model is trained by inputting the same learning image or a feature map obtained from the learning image to each other when executing machine learning. The appearance inspection device according to claim 1.

10. An appearance inspection method for detecting a defect included in an inspection target image in which an inspection target object is imaged, using a machine learning model, the method including: A step of training a first machine learning model and a second machine learning model using a learning image in which the inspection target object is imaged. Using the trained first machine learning model and the second machine learning model, detecting defects included in a verification image in which the object to be inspected is imaged; Based on the defect detection accuracy of the first machine learning model and the second machine learning model with respect to the verification image, determining the usage ratio of the first machine learning model and the second machine learning model; Based on the usage ratio, detecting defects included in the inspection target image using at least one of the first machine learning model and the second machine learning model; comprising: The machine learning model includes a first machine learning model and a second machine learning model; The first machine learning model is more likely to detect defects with position dependence than the second machine learning model; The second machine learning model is more likely to detect defects without position dependence than the first machine learning model. Appearance inspection method.

Images