Learning method, learning device, learning program, and recording medium

Abstract

An aspect of the present invention provides a learning method, a learning device, a learning program, and an image processing device. A learning method according to an aspect of the present invention is performed by a learning device provided with a processor, wherein the processor performs a data acquisition step for acquiring training data that comprises pairs of a patch image and correct data for a class label with respect to a unit region of the patch image; a determination step for using a learning model and the training data to perform segmentation of the patch image and determine, for each patch image, whether or not the learning model has correctly detected a second unit region; a weighting step for setting a first weight in learning on the basis of the result of the determination; and an updating step for updating the learning model on the basis of the result of the weighting.

Description

[Technical Field]

[0001] The present invention relates to a learning method, a learning apparatus, and a learning program for learning images, as well as an image processing apparatus that uses a trained model constructed by such learning. [Background technology]

[0002] Regarding image learning techniques, for example, Patent Documents 1 and 2 describe learning based on cross-entropy loss. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-061066 [Patent Document 2] Japanese Patent Publication No. 2020-091543 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] One aspect of the present invention provides a learning method, a learning apparatus, and a learning program that can obtain a learning model with a good detection rate, and an image processing apparatus that can detect scratches and / or defects in an object with good accuracy using the learned model. [Means for solving the problem]

[0005] To achieve the above-mentioned objectives, a learning method according to a first aspect of the present invention is a learning method performed by a learning device equipped with a processor, wherein the processor performs a data acquisition step of acquiring learning data consisting of pairs of patch images and correct class label data for unit regions of the patch images; a determination step of performing segmentation of the patch images using the learning model and the learning data and determining for each patch image whether the learning model correctly detected a second unit region; a weighting step of setting first weights based on the determination results; and an update step of updating the learning model based on the weighting results.

[0006] In the second embodiment of the learning method, in the first embodiment, the processor sets a first weight on a patch image basis for a first loss, which is the loss for each individual patch image, in the weighting step.

[0007] In the third embodiment of the learning method, in the first or second embodiment, if the processor determines in the weighting step that the second unit region has not been correctly detected, it sets a larger weight as the first weight than when it determines that the second unit region has been correctly detected.

[0008] In the learning method relating to the fourth embodiment, in any one of the first to third embodiments, the learning model detects a second unit region belonging to a specific class in the determination step.

[0009] In the learning method according to the fifth embodiment, in the fourth embodiment, the processor determines in the determination step that the second unit region was not correctly detected in the first case in which the learning model incorrectly detected a second unit region belonging to a specific class, and in the second case in which the learning model failed to detect a second unit region belonging to a specific class.

[0010] In the learning method according to the sixth embodiment, in the fifth embodiment, the processor sets a larger weight in the second case than in the first case during the weighting step.

[0011] In the learning method relating to the seventh aspect, in the fifth or sixth aspect, the processor determines in the determination step that the detection result is correct if the detection result is neither the first case nor the second case, but a third case.

[0012] In the learning method relating to the eighth aspect, in any one of the fourth to seventh aspects, the processor makes a determination regarding scratches and defects on the subject in the determination step.

[0013] In the learning method relating to the ninth aspect, in any one of the fourth to eighth aspects, the learning model outputs the probability of detection, and the processor, in the determination step, determines whether the probability is above a threshold or not, that the second unit region belongs to a specific class.

[0014] In the learning method according to the tenth embodiment, the processor changes the threshold during the learning process.

[0015] In the learning method relating to the 11th embodiment, in any one of the first to tenth embodiments, the processor performs weighting on the cross-entropy loss of the patch image in the weighting step.

[0016] In the learning method according to the twelfth embodiment, in any one of the first to eleventh embodiments, the processor further performs a loss function derivation step to derive a loss function for a batch consisting of patch images, and in the update step, updates the learning model using the loss function.

[0017] In the learning method according to the 13th embodiment, in the 12th embodiment, the processor derives a first loss function as the loss function in the loss function derivation step, which is obtained by averaging the weighted results across the entire batch composed of patch images.

[0018] In the learning method according to the 14th aspect, in the 13th aspect, in the loss function derivation step, the processor uses, as the loss function, a function obtained by combining a first loss function and a second loss function which is a loss function for a batch and different from the first loss function.

[0019] In the learning method according to the 15th aspect, in any one of the 1st to 14th aspects, in the update step, the processor updates the parameters of the learning model to minimize the loss function.

[0020] In the learning method according to the 16th aspect, in any one of the 1st to 15th aspects, in the data acquisition step, the processor inputs an image and acquires, as a patch image, an image obtained by dividing the input image.

[0021] In the learning method according to the 17th aspect, in any one of the 1st to 16th aspects, in the data acquisition step, the processor acquires a patch image having a size corresponding to the size of a defect and / or a defect of the subject to be detected.

[0022] In the learning method according to the 18th aspect, in any one of the 1st to 17th aspects, the learning model includes a neural network that performs segmentation.

[0023] To achieve the above object, a learning device according to the 19th aspect is a learning device including a processor, the processor performs a data acquisition process of acquiring learning data composed of a pair of a patch image and correct data of a class label for a unit area of the patch image, performs segmentation of the patch image using the learning model and the learning data, and determines for each patch image whether the learning model correctly detects a second unit area, a weighting process of setting a first weight based on the determination result, process and an update process of updating the learning model based on the weighted result. The learning device according to the 19th aspect may have a configuration that performs the same processes as those in the 2nd to 18th aspects.

[0024] To achieve the above-mentioned objectives, a learning program according to a 20th aspect of the present invention is a learning program that causes a learning device equipped with a processor to execute a learning method, wherein the learning method causes the processor to execute: a data acquisition step of acquiring learning data consisting of pairs of patch images and correct class label data for unit regions of the patch images; a determination step of performing segmentation of the patch images using the learning model and the learning data and determining for each patch image whether the learning model correctly detected a second unit region; a weighting step of setting a first weight based on the determination result; and an update step of updating the learning model based on the weighting result. The learning program according to the 20th aspect may have a configuration that performs the same processing as the second to 18th aspects. Furthermore, non-temporary recording media that record computer-readable code of learning programs according to these aspects are also included in the scope of the present invention.

[0025] To achieve the above-mentioned objectives, an image processing device according to the 21st aspect of the present invention includes a trained model trained using a learning method according to any one of the 1st to 18th aspects, and detects scratches and / or defects in an input image using the trained model. [Brief explanation of the drawing]

[0026] [Figure 1] Figure 1 shows a schematic configuration of the learning device. [Figure 2] Figure 2 shows a schematic configuration of the processing unit. [Figure 3] Figure 3 shows the processing performed by a convolutional neural network. [Figure 4] Figure 4 is another diagram illustrating the processing performed by a convolutional neural network. [Figure 5] Figure 5 is a flowchart showing an overview of the learning method's processing. [Figure 6] Figure 6 is a flowchart showing the details of the learning method's processing. [Figure 7]Figure 7 shows the effect of weighting on learning. [Figure 8] Figure 8 is a flowchart showing the process for acquiring patch images. [Figure 9] Figure 9 is a flowchart showing variations in weighting processing. [Figure 10] Figure 10 shows how a portion of a large defect goes undetected. [Figure 11] Figure 11 shows how a minute defect can go undetected entirely. [Modes for carrying out the invention]

[0027] [Detection rate in segmentation learning] As an example of conventional segmentation (classification), we will explain the case of detecting defects in a subject.

[0028] In machine learning, cross-entropy loss is commonly used for semantic segmentation. For each pixel of an image being classified, a penalty (=loss) is applied based on the difference between the correct label (p=0 or 1) and the predicted value of the learning model (0≦q≦1). The learning model is then updated using this loss. For example, in a defect detection task, p=1 can be associated with "defective" and p=0 with "non-defective". Note that in the following explanation, cross-entropy loss may sometimes be referred to as "CE loss".

[0029] In segmentation tasks, within the same class to be classified, there may be both very small and very large defects present simultaneously. That is, in defect detection, small and large defects may coexist. However, with typical CE loss algorithms, predictions are made for each pixel and treated equally. As a result, large defects are learned more frequently, while small defects are less likely to be learned.

[0030] The CE loss is expressed by the following equations (1) and (2), where p ∈ {0,1} is the correct label when performing binary classification of {defective, non-defective} for each pixel, and q ∈ [0,1] is the output value representing the defect probability.

[0031]

number

[0032]

number

[0033] Equation (1) is defined for each pixel. The CE loss depends on the coordinates (x,y) of the pixels that make up each input patch image Ii (1≦i≦B) when the batch size, which is the unit for updating the model weights, is B, but the subscripts are omitted. Equation (2) means taking the average over these (total number of pixels N := B × H × W, where H and W are the sizes of the input patch images), so that pixels in the image are learned equally.

[0034] For example, as shown in Figure 10, there is a region where (i) a few pixels of a large defect 900 (see part (a) in the same figure) are undetected (region 902 in part (b) of the same figure), and a region where the entire small defect 910 consisting of a few pixels (see part (a) in the same figure) is undetected (region 912 in part (b) of the same figure), as shown in Figure 11. If the number of undetected pixels is the same in region 902 of (i) and region 912 of (ii), then their contributions to the loss function will also be similar. However, from the perspective of evaluating the detection rate, it is preferable to prioritize detecting defects that have not been detected at all (small defects 910), as in (ii), rather than reproducing a more detailed shape of a defect that has been successfully detected (large defect 900), as in (i).

[0035] Thus, in tasks where detection targets of varying sizes are present, and it is important to prevent even minute oversights and improve the detection rate based on the number of targets, the conventional loss functions described above cannot take these into account. Therefore, it is preferable to set a loss that more directly improves the detection rate.

[0036] Under these circumstances, the inventors of the present invention have diligently conducted research and conceived of a learning method, learning apparatus, and learning program that can obtain a learning model with a good detection rate, as well as an image processing apparatus with a good detection rate for scratches and / or defects in a subject. Embodiments of the present invention based on these concepts will be described below. In the description, the accompanying drawings will be referred to as necessary. In addition, for the sake of clarity, some components may be omitted from the accompanying drawings.

[0037] [Configuration of the learning device] Figure 1 is a diagram showing the schematic configuration of a learning device (image processing device) according to one embodiment of the present invention. As shown in Figure 1, the learning device 10 (learning device, image processing device) according to the first embodiment comprises a processing unit 100 (processor, computer), a storage unit 200, a display unit 300, and an operation unit 400, and these components are connected to each other to transmit and receive necessary information. Various installation configurations can be adopted for these components, and each component may be installed in one location (in one enclosure, in one room, etc.), or it may be installed in separate locations and connected via a network. In addition, the learning device 10 can connect to an external server 500 and / or an external database 510 via a network NW such as the Internet, and can acquire samples and learning data to be used for processing as needed, and can also store processing results etc. in the external server 500 and / or external database 510.

[0038] [Configuration of the processing area] As shown in Figure 2, the processing unit 100 includes a processor 102 (processor, computer), a ROM 130 (ROM: Read Only Memory), and a RAM 150 (RAM: Random Access Memory). The processor 102 provides overall control of the processing performed by each part of the processing unit 100 and has the functions of a data acquisition unit 104, a detector 106, a determination unit 108, a weighting unit 110, an update unit 112, a display control unit 114, a recording control unit 116, and a communication control unit 118. The data acquisition unit 104 can receive input via a network NW or from a recording medium such as a storage unit 200, and the display control unit 114 and the recording control unit 116 can display on the monitor 310 and output to the storage unit 200, respectively. These processes are performed via the communication control unit 118 as needed.

[0039] [Detector Configuration] The detector 106 (processor) is equipped with a learning model, which includes a neural network that performs segmentation. Examples of neural networks that can be used include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and SVM (Support Vector Machine). As described later, the parameters of the learning model are updated by a learning method according to one aspect of the present invention. The detector 106 (trained model) after training is completed can be used to detect flaws and / or defects in an image of a subject. That is, after training is completed, the learning device 10 operates as an image processing device according to one aspect of the present invention (described later).

[0040] [Example of CNN layer structure] This section describes an example of a layer configuration when using a CNN (an example of a learning model) in the detector 106. Figure 3 shows an example of the layer configuration of the learning model 224. In the example shown in part (a) of Figure 3, the learning model 224 includes an input layer 250, an intermediate layer 252, and an output layer 254. The input layer 250 takes the image (patch image) acquired by the data acquisition unit 104 as input and outputs features. The intermediate layer 252 includes a convolutional layer 256 and a pooling layer 258, and calculates other features by taking the features output by the input layer 250 as input. These layers have a structure in which multiple "nodes" are connected by "edges" and hold multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The learning model 224 may also include a fully connected layer 260, as shown in the example in part (b) of Figure 3. The layer configuration of the learning model 224 is not limited to a sequence of one convolutional layer 256 and one pooling layer 258; any of the layers (for example, multiple consecutive convolutional layers 256) may be included. Furthermore, multiple consecutive fully connected layers 260 may also be included.

[0041] [Processing in the intermediate layer] The intermediate layer 252 calculates features through convolution and pooling operations. The convolution operation performed in the convolution layer 256 is a process that obtains a feature map using a filter, and is responsible for feature extraction such as edge extraction from images. This convolution operation using a filter generates one channel (one image) of "feature map" for each filter. The size of the "feature map" is downscaled by the convolution, and becomes smaller as convolution is performed in each layer. The pooling operation performed in the pooling layer 258 is a process that shrinks (or expands) the feature map output by the convolution operation to create a new feature map, and is responsible for providing robustness so that the extracted features are not affected by translation, etc. The intermediate layer 252 can be composed of one or more layers that perform these operations.

[0042] Figure 4 shows the convolution operation in the learning model 224 shown in Figure 3. In the first (1st) convolutional layer of the intermediate layer 252, a convolution operation is performed between an image set (a training image set during learning, and a recognition image set during recognition such as detection) and a filter F1. The image set can consist of N images (N channels) with an image size of H vertically and W horizontally. When a visible light image is input, the images that make up the image set are 3-channel images of R (red), G (green), and B (blue). Since the image set has N channels (N images), for example, if the filter is of size 5 (5x5), the filter size F1 that is convolved with this image set will be a filter with a size of 5x5xN. Through the convolution operation using this filter F1, a "feature map" of 1 channel (1 image) is generated for one filter F1. The filter F2 used in the second convolutional layer will have a filter size of 3x3xM if it is a filter of size 3 (3x3).

[0043] Similar to the first convolutional layer, the second through nth convolutional layers use filters F2~F n A convolution operation is performed using [this method]. The reason why the size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer is because it has been downscaled by the preceding convolutional or pooling layers.

[0044] Of the 252 intermediate layers, the convolutional layers closer to the input side perform low-order feature extraction (edge ​​extraction, etc.), while higher-order feature extraction (extraction of features related to the shape, structure, etc. of the object) is performed as the layers approach the output side. When segmentation is performed for the purpose of measuring a region of interest (in this embodiment, scratches and / or defects), upscaling is performed in the latter half of the convolutional layers, and the final convolutional layer yields a "feature map" of the same size as the input image set. On the other hand, when detecting a region of interest, upscaling is not necessary as it is sufficient to output positional information.

[0045] The hidden layer 252 may also include a layer that performs batch normalization in addition to the convolutional layer 256 and the pooling layer 258. Batch normalization is a process that normalizes the distribution of data in units of minibatches during training, and plays a role in speeding up training, reducing dependence on initial values, and suppressing overfitting.

[0046] [Processing in the output layer] The output layer 254 is a layer that detects the location of the region of interest in the input image (patch image, etc.) based on the features output from the intermediate layer 252 and outputs the result. When performing segmentation (classification), the output layer 254 uses the "feature map" obtained from the intermediate layer 252 to determine the location of the region of interest in the image at the pixel level. That is, it can detect whether or not each pixel in the image belongs to the region of interest (whether or not it belongs to a specific class) and output the detection result. On the other hand, when performing object detection, pixel-level judgment is not necessary, and the output layer 254 outputs the location information of the target object. Note that the "specific class" is, for example, "defect" or "scratch". The output layer 254 may also classify and output defects and scratches in detail (bubbles, foreign matter contamination, holes, cracks, delamination, etc.).

[0047] The output layer 254 (learned model 224) can output detection results (segmentation results, classification results) for a unit region. The output layer 254 can also output the likelihood (probability, confidence level) of the detection results. A "unit region" may consist of one pixel or multiple pixels.

[0048] The output layer 254 may perform discrimination (classification) of the region of interest and output the discrimination results. For example, the output layer 254 may classify the image into three categories: "normal," "blemished," and "defective," and output three scores corresponding to "normal," "blemished," and "defective" as the discrimination results (the sum of the three scores is 100%, or 1). Alternatively, if a clear classification can be made from the three scores, the output layer 254 may output the classification results. The classification may be two-class or multiple-class.

[0049] When outputting the identification results, the intermediate layer 252 or output layer 254 may or may not include a fully connected layer as the last one or more layers (see part (b) of Figure 3).

[0050] The output layer 254 may also output the measurement results of the region of interest. When performing measurements, the measured values ​​of the target region of interest can be directly output from the learning model 224. Alternatively, the target region of interest can be segmented as described above, and then measured by the processor 102 or similar based on the segmentation results. When the measured values ​​are output directly, the measurement values ​​themselves are learned from the image, so it becomes a regression problem of the measured values.

[0051] When using the learning model 224 with the configuration described above, it is preferable that during the learning process, the results output by the output layer 254 are compared with the correct recognition of the image set to calculate the loss function (error function), and that the weight parameters in the hidden layer 252 are updated from the output layer to the input layer (backpropagation) so that the loss function becomes small (minimized). The derivation of the loss function and the updating of the learning model based on the loss function in this embodiment will be described in detail later.

[0052] [Recognition using methods other than neural networks] The learning model 224 may perform detection using methods other than neural networks. For example, it can detect areas of interest based on the features of the acquired image. In this case, the learning model 224 divides the image to be detected into, for example, multiple rectangular regions, sets each divided rectangular region as a local region, calculates the feature of the pixels within each local region of the image to be detected (for example, hue), and determines the local region with a specific hue from among the local regions as the area of ​​interest. Similarly, the learning model 224 may perform classification or measurement based on features.

[0053] [Realization of the functions of the processing unit and processor] The functions of each part of the processing unit 100 and processor 102 (including the detector 106) described above can be realized using various processors and recording media. These various processors include, for example, a CPU (Central Processing Unit), a general-purpose processor that executes software (programs) to realize various functions. These various processors also include a GPU (Graphics Processing Unit), a processor specialized for image processing, and a Programmable Logic Device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. A configuration using a GPU is effective for image learning and recognition. Furthermore, dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform particular processing, are also included in these various processors.

[0054] The functions of each part may be realized by a single processor, or by multiple processors of the same or different types (for example, multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Furthermore, multiple functions may be realized by a single processor. Examples of configuring multiple functions with a single processor include, firstly, as exemplified by computers, a single processor composed of one or more CPUs and software, where this processor realizes multiple functions; and secondly, as exemplified by System-on-a-Chip (SoC), a processor that realizes the functions of the entire system on a single IC (Integrated Circuit) chip. Thus, various functions are configured as hardware structures using one or more of the aforementioned processors. More specifically, the hardware structures of these various processors are electrical circuits combining circuit elements such as semiconductor elements. These electrical circuits may be circuits that realize the aforementioned functions using logical operations such as logical OR, logical AND, logical negation, exclusive OR, and combinations thereof.

[0055] When the aforementioned processor or electrical circuit executes software (program), it stores code readable by the computer (for example, the various processors and electrical circuits constituting the processing unit 100, and / or combinations thereof) of the software to be executed in a non-temporary recording medium such as ROM 130, and the computer refers to that software. The software stored in the non-temporary recording medium is according to the present invention. study A program to execute the method ( studyThis includes the program and the data used during its execution (first and second DNA profile data, training data such as cancer type labels described later, weight parameters used in machine learning, etc.). The code may be recorded on a non-temporary recording medium such as various magneto-optical recording devices or semiconductor memory instead of ROM 130. When processing using software, for example, RAM 150 is used as a temporary storage area, and data stored on a non-temporary recording medium such as an EEPROM (Electronically Erasable and Programmable Read Only Memory) or flash memory (not shown) can also be referenced. The storage unit 200 may be used as a "non-temporary recording medium". Note that the above-mentioned "non-temporary recording medium" does not include intangible recording media such as carrier signals or propagation signals themselves.

[0056] Details of the processing using the processing unit 100 with the above configuration will be described later.

[0057] [Memory Unit Configuration] The memory unit 200 is composed of various memory devices such as hard disks and semiconductor memory, and their control units, and can store images of the subject (input images, patch images), training data, weight parameters used by the learning model, execution conditions and results of the learning method, execution conditions and results of the image processing method, and so on.

[0058] [Display Unit Configuration] The display unit 300 includes a monitor 310 (display device) which is composed of a display such as a liquid crystal display, and can display images of the subject, learning methods, learning programs, the results of image processing methods, etc. The monitor 310 may be composed of a touch panel type display and can accept user input.

[0059] [Configuration of the control panel] The operation unit 400 includes a keyboard 410 and a mouse 420 as examples of operating devices, and the user can perform operations related to the execution of the learning method, learning program, and image processing method according to the present invention, display of results, etc., via the operation unit 400. The operation unit 400 may also include other operating devices.

[0060] [Processing of learning methods] The processing of the learning method (learning program) according to this embodiment will now be described. Figure 5 is a flowchart showing an overview of the learning method's processing. Note that the following description is just one example of the processing in the present invention, and the order of processing may be changed as needed.

[0061] [Process Overview] (Acquisition of training data) When processing begins, the data acquisition unit 104 (processor) acquires training data consisting of pairs of patch images and correct class label data for unit regions of the patch images (step S100: data acquisition process). The data acquisition unit 104 may acquire the training data from the storage unit 200 or from an external database 510. The data acquisition unit 104 may also acquire the training data all at once or in multiple batches.

[0062] (Segmentation of patch images) The detector 106 (processor) uses the learning model 224 and training data described above to segment the acquired patch image and outputs the likelihood that a unit region belongs to each class (step S110: judgment step, judgment processing). The "classes" can be defined as, for example, "flaw," "defect," or "normal" (other than "flaw" or "defect"). Here, "flaw" can be defined as a discontinuous part of the object, and "defect" can be defined as a flaw that exceeds the standards specified in the standards or specifications and results in a failed inspection (for example, a flaw that exceeds the standard in size or degree). However, in this invention, flaws and defects may not be distinguished and both may be treated as "defects." Furthermore, flaws and defects may be divided into multiple types (for example, foreign matter, bubbles, holes, cracks, peeling, etc.) and degrees (number, size, density, etc.), and flaws and defects of different types and / or degrees may be treated as different classes. Furthermore, it is preferable to pre-configure the layer configuration of the detector 106 (learning model 224) (such as the presence or absence of fully connected layers and the number of outputs) according to the type of class to be classified, and to prepare ground truth data that matches the type, number, and degree of the classes to be classified.

[0063] In this invention, a "unit region" is a region composed of one pixel or multiple pixels.

[0064] The following describes the case where each pixel is classified into two classes, "defective" and "normal," that is, where no distinction is made between flaws and defects, and the unit region consists of one pixel. Furthermore, the detector 106 outputs a probability q (q∈[0,1]) that each pixel belongs to the defect class.

[0065] (Determining whether a specific class was detected or not) The determination unit 108 (processor) determines, for each patch image (for each second unit region), whether the learning model 224 correctly detected a second unit region belonging to a specific class (defect class) based on the probability q for each pixel and the ground truth data p. , judgment Determine (Step S120: Determination process, determination process).

[0066] (Weighting and updating) The weighting unit 110 (processor) sets weights for the loss (loss function) on a patch image basis based on the results of the judgment described above (step S130: weighting process, weighting operation), and the update unit 112 (processor) updates the learning model of the detector 106 based on the weighting results (step S140: update process, update operation). The update unit 112 (processor) causes each part of the processor 102 to repeat the processing in steps S100 to S140 until the termination condition is met (until YES is reached in step S150). The termination condition is, for example, that the learning of all learning data has been completed, that a predetermined number of repetitions have been completed, or that the loss has converged.

[0067] [Processing details] The details of the learning method described above will be explained with reference to the flowchart in Figure 6.

[0068] [Class determination] The determination unit 108 (processor) compares the probability q (the probability that each pixel belongs to a defect class) output by the detector 106 (learning model) with a threshold θ (step S122: determination process). If the probability is greater than or equal to the threshold (q≧θ), the determination unit 108 can determine that "the pixel to be judged belongs to a defect class (a specific class)."

[0069] [Weighting of losses and derivation of the loss function] The weighting unit 110 (processor) uses the CE loss, which is the loss for each individual patch image, as the first loss, and performs weighting on a patch image basis as follows.

[0070] The weighting unit 110 calculates the CE loss using the above-described equation (1) based on the probability q of each pixel and the ground truth data p (step S131: weighting process). The weighting unit 110 also compares the probability q with the ground truth data p (step S132: weighting process). If it is a false detection or not detected (YES in step S133; see equation (4) below), i.e., if the comparison result is a specific result, it sets α=1 as the first weight for the CE loss on a patch image basis (step S134). If it is not a false detection or not detected (NO in step S133; see equation (4) below), i.e., if the comparison result is not a specific result, it sets α=0 as the first weight on a patch image basis. In this way, the weighting unit 110 sets a larger weight as the first weight when the comparison result is a specific result than when the comparison result is not a specific result.

[0071] The weighting unit 110 (processor) averages the weighted results across the entire batch composed of patch images to derive a first loss function (see equation (3) below; loss function) (step S138: loss function derivation step, loss function derivation process).

[0072] The weighting and loss function described above can be derived from the following equations (3) and (4). In equation (3), B is the number of patch images that make up the batch, and N is the total number of pixels in the batch (= B × H × W, where H and W are the sizes of the input patch images).

[0073]

number

[0074]

number

[0075] In equation (4), θ∈(0,1) represents a predetermined threshold for defect probability, and processor 102 can determine that a pixel is defective if its probability q is greater than or equal to the threshold θ. In this case, if a region that is not actually defective is misdetected (first case), or if a defect goes undetected despite being defective (second case), then α(Ii)=1 in equation (3), and a loss is added to that patch. Conversely, as learning progresses and even a portion of the defects are successfully detected in some pixels, α(Ii)=0 (third case), and the priority of learning decreases relatively.

[0076] To explain the weighting in equation (4) in detail, "(as a patch) false positive" is defined as "when all pixels in the patch have the correct label p=0, and the output value (likelihood) q of the learning model for any one pixel in the patch is greater than or equal to the threshold θ (q≧θ)", and "(as a patch) undetected" is defined as "when there is at least one pixel in the patch with the correct label p=1, and the output value (likelihood) q of the learning model for all pixels in the patch is less than the threshold (q<θ)". If it is a "(as a patch) false positive" or an "(as a patch) undetected", then α=1, and in all other cases, α=0. Note that equation (4) does not determine "undetected" or "false positive" for each individual pixel in the patch, but rather a single judgment result is determined for the entire patch (second unit region) based on the above criteria.

[0077] [The impact of weighting on learning] The impact of the weighting described above on learning will now be explained. For example, consider the case where there is a large defect 922 in patch image 920, as shown in part (a) of Figure 7 (a diagram showing the correct range of defects in the patch image) (p=1 for pixels inside defect 922, and p=0 for pixels outside defect 922). In this case, the detection result by detector 106 is as shown in part (b) of the same figure, where a region narrower than the correct range in patch image 920A is detected as defect 922A (p=1 for pixels inside defect 922A, and p=0 for pixels outside defect 922A).

[0078] In this case, according to the weighting described above, α=0 (a judgment that it was detected correctly). This can be interpreted as "if even a small part of a huge defect is detected, the penalty for not detecting it (weighting it as α=1) is eliminated," and conversely, "it is not necessary to detect the entire area of ​​a huge defect."

[0079] In the example shown in Figure 7, conventional methods attempt to further learn (weight) the large defect 922 because a large area remains undetected at the pixel level. In contrast, the method of the present invention completes the learning of such a large defect 922 at an early stage (the weight becomes zero). On the other hand, in the case of minute defects, which consist of only a few pixels, the criterion of "it is sufficient to detect even a part of the defect" does not relax the conditions.

[0080] As a result, the method of the present invention loosens the criteria for "detection" only for large defects, making it relatively easier to learn about minute defects.

[0081] Based on the above, this embodiment is advantageous in that even minute defects can be more easily learned from areas that have not yet been detected, thereby improving the detection rate.

[0082] [Combinations of loss functions] In updating the learning model, the weighting unit 110 (processor) may use the first loss function (Lweighted) described above as the loss function alone, or it may use a function that combines Lweighted and a second loss function which is a batch loss function different from Lweighted as the loss function. For example, as shown in equation (5) below, the normal CE loss may be used as the second loss function (L CE Alternatively, the loss function may be a function obtained by adding the first loss function and the second loss function with an arbitrary coefficient (λ>0) (Step S139: Loss function derivation process, loss function derivation process).

[0083]

number

[0084] Further variations of the loss function will be discussed later in the section "Variations of the Loss Function".

[0085] The processor 102 may determine what loss function or weights to use in response to user input via the operation unit 400, or it may determine them automatically without user input.

[0086] [Updating the learning model] The update unit 112 (processor), if it is time to update the learning model 224 (YES in step S142), updates the parameters of the learning model 224 using the derived loss function (step S144: update process). For example, the update unit 112 updates the parameters of the learning model to minimize the loss function. The timing of the update in step S142 may be per patch image, per step (per batch), or per multiple steps.

[0087] [Examples] For a dataset containing both minute defects of 4px or less and large defects of 10,000px or more, segmentation training was performed under the following conditions. In the condition using CE loss as the loss function (conventional method), the detection probabilities for FMLD (Foreign Material Less Dense) and FMMD (Foreign Material More Dense) were 74.5% and 74.1%, respectively. However, in the condition using weighted CE loss as described above (the method of the present invention), after a total of 200 epochs, the detection probabilities for FMLD and FMMD improved to 89.7% and 88.1%, respectively. Here, "detection probability" refers to the number of successfully detected defects relative to the total number of defects, when a continuous region with a correct label is counted as "1".

[0088] <Detailed conditions> (a) The weighting described above was applied to the derivation of the loss function for each epoch. (b) The weighting conditions were set by applying equation (4) and setting the weight parameter (coefficient) λ = 0.1 in equation (5). (c) The threshold for judgment was fixed at θ = 0.5.

[0089] [Effects of the first embodiment] As described above, the learning method, learning device, and learning program according to the first embodiment can be used to obtain a learning model with a good detection rate.

[0090] [Variations in processing methods for learning] [Image variations] In this invention, radiation images, visible light images, near-infrared light images, etc., can be used for learning (or image processing after learning is complete), and the frequency band used to acquire the images is not particularly limited.

[0091] [Obtaining patch images by splitting an image] In some cases, the patch image acquired in step S100 described above may not be the same size as the image input to the detector 106 (learning model 224). Specifically, the image of the subject (for example, several thousand pixels x several thousand pixels) may be larger than the patch image (for example, several tens of pixels x several tens of pixels to several hundred pixels x several hundred pixels). In such cases, the data acquisition unit 104 (processor) can acquire the input image as a patch image by dividing it to match the size of the image input to the detector 106 (data acquisition process, data acquisition processing). This reduces the load on processors such as CPU and GPU, and memory such as RAM.

[0092] Furthermore, when acquiring patch images, the data acquisition unit 104 may resize the input image instead of splitting it (dividing it into patch images), or in addition to splitting it (data acquisition step, data acquisition process). For example, the data acquisition unit 104 can input an entire rectangular image as a square.

[0093] Furthermore, the data acquisition unit 104 (processor) may acquire a patch image of a size corresponding to the size of the object to be detected (detection size). For example, as shown in the flowchart of Figure 8, the data acquisition unit 104 inputs an image and ground truth data (step S102) and sets the size of the object to be detected (e.g., scratches and / or defects on the subject) (step S104). The data acquisition unit 104 can set the size of the object automatically (without user operation) in response to user operation via the operation unit 400 or according to predetermined conditions.

[0094] The data acquisition unit 104 divides the input image into patch images according to the set size (step S106). The resizing described above may also be performed. Subsequent processing is the same as described above for the flowcharts in Figures 5 and 6.

[0095] [Threshold for judgment] Processor 102 (the processor) may change the threshold θ during the learning process. For example, based on the idea that "in the early stages of learning, flaws and defects cannot be detected accurately, so the threshold (the hurdle or standard for detection) is set low, and as learning progresses and accurate detection becomes possible, the threshold can be raised," the threshold can be set low in the early stages of learning and then raised later.

[0096] [Weighting variations] In the first embodiment described above, the weight for the patch image is set to α=1 in the case of false detection or non-detection (first case and second case), and α=0 in all other cases (third case). However, in the present invention, the weighting of the patch image is not limited to this form. In addition to the above-described form, for example, the detection rate of minute defects may be improved by making the weight for the case of non-detection (second case) larger than the weight for the case of false detection (first case). Figure 9 is a flowchart of the weighting process in such a case. As shown in the figure, the weighting unit 110 (processor) can set α=2 as the weight in the case of non-detection (YES in step S133A) (step S134A), set α=1 in the case of false detection (YES in step S136) (step S137A), and set α=0 in all other cases (NO in step S136).

[0097] [Variations in loss functions] In the first embodiment described above, L is based on CE loss. CE , L weighted While we have described cases where these or combinations are used as loss functions, the present invention also allows the use of different loss functions. For example, the weighting unit 110 (processor) may use the mean sum of squared errors, Dice loss, IoU loss (Jaccard loss), Tversky loss, Focal loss, combinations thereof, and further, L CE and / or L weighted The combination of these can be used as the loss function. When combining multiple loss functions, the total loss function L can be created by combining individual loss functions in any number and with any coefficient, as shown in equation (6) below. total (Loss function) can be defined as (Loss function derivation process, Loss function derivation process). Note that L weighted The coefficient of is preferably greater than 0.

[0098]

number

[0099] In Equation (6), K is the number of loss functions to be combined, which is an integer of 1 or more, λi is a coefficient (any real number) for each loss function, and Li is each loss function, which is the L described above CE 、L weighted 、such as mean squared error, Dice loss, IoU loss (Jaccard loss), Tversky loss, Focal loss, etc.

[0100] [Operation as an image processing device] After the completion of the learning described above, the learning model 224 of the detector 106 becomes a learned model. That is, after the completion of learning, the learning device 10 can operate as an image processing device. According to this image processing device, using the learned model learned by the learning method according to the present invention, it is possible to detect a defect and / or a defect of a subject with a good detection rate.

[0101] Although the embodiments of the present invention and their variations have been described above, the present invention is not limited to the above-described aspects, and various modifications are possible without departing from the spirit of the present invention.

Explanation of reference numerals

[0102] 10 Learning device 100 Processing unit 102 Processor 104 Data acquisition unit 106 Detector 108 Judgment unit 110 Weighting unit 112 Update unit 114 Display control unit 116 Recording control unit 118 Communication control unit 130 ROM 150 RAM 200 Storage unit 224 Learning model 250 Input layer 252 Intermediate layer 254 Output layer 256 Convolutional layer 258 Pooling layer 260 Fully connected layer 300 Display 310 Monitor 400 Control unit 410 Keyboard 420 mice 500 External Servers 510 External Databases 900 Huge Defect 902 areas 910 Microdefects 912 areas F1 filter F2 filter NW Network S100~S150 Steps in the Learning Method

Claims

1. A learning method performed by a learning device equipped with a processor, wherein the processor is A data acquisition step involves acquiring training data consisting of pairs of patch images and ground truth data of class labels for unit regions of the patch images. A determination step is performed to segment the patch image using the learning model and the learning data, and to determine for each patch image whether the learning model correctly detected the second unit region. Based on the result of the determination, a weighting step is performed in which a first weight is set for each patch image, with respect to the first loss which is the loss for each patch image. A loss function derivation step for deriving a loss function for a batch composed of the patch images based on the weighted results, comprising: a loss function derivation step for deriving a first loss function as the loss function by adding or averaging the weighted results over the entire batch; An update step in which the parameters of the learning model are updated to minimize the loss function, A learning method that executes this.

2. In the weighting step, the processor The learning method according to claim 1, wherein if it is determined that the second unit region is not correctly detected, a larger weight is set as the first weight than when it is determined that the second unit region is correctly detected.

3. The learning method according to claim 1, wherein the learning model detects the second unit region belonging to a specific class in the determination step.

4. In the determination step, the processor The learning method according to claim 3, wherein in the first case the learning model incorrectly detects the second unit region belonging to the specific class, and in the second case the learning model fails to detect the second unit region belonging to the specific class, it is determined that the second unit region has not been correctly detected.

5. In the weighting step, the processor The learning method according to claim 4, wherein in the second case, a larger weight is set than in the first case.

6. In the determination step, the processor The learning method according to claim 4 or 5, wherein in a third case where the detection result is neither the first case nor the second case, the detection result is determined to be correct.

7. In the determination step, the processor The learning method according to claim 3, which performs the determination of scratches and defects on a subject.

8. The learning model outputs the probability of the detection, In the determination step, the processor The learning method according to claim 3, wherein it is determined that the second unit domain belongs to the specific class based on whether the probability is above a threshold.

9. The learning method according to claim 8, wherein the processor changes the threshold during the learning process.

10. The learning method according to claim 1, wherein the processor performs the weighting step by applying the weights to the cross-entropy loss of the patch image.

11. In the loss function derivation step, the processor The learning method according to claim 1, wherein the loss function is a function that combines the first loss function and a second loss function which is a loss function for the batch and is different from the first loss function.

12. The processor, in the data acquisition step, The learning method according to claim 1, wherein an image is input, and the images obtained by dividing the input image are obtained as the patch images.

13. The learning method according to claim 1, wherein the processor acquires the patch image in the data acquisition step, the patch image being sized according to the size of the scratches and / or defects of the subject to be detected.

14. The learning method according to claim 1, wherein the learning model comprises a neural network that performs the segmentation.

15. A learning device comprising a processor, wherein the processor is A data acquisition process that acquires training data consisting of pairs of patch images and ground truth data of class labels for unit regions of the patch images, A determination process is performed to segment the patch image using the learning model and the learning data, and to determine for each patch image whether the learning model correctly detected the second unit region. Based on the result of the determination, a weighting process is performed to set a first weight for each patch image, which is the loss for each patch image. A loss function derivation process for deriving a loss function for a batch composed of the patch images based on the weighted results, comprising: a loss function derivation process for deriving a first loss function as the loss function by adding or averaging the weighted results over the entire batch; An update process that updates the parameters of the learning model to minimize the loss function, A learning device that performs this task.

16. A learning program that causes a learning device equipped with a processor to execute a learning method, wherein in the learning method, the processor A data acquisition step involves acquiring training data consisting of pairs of patch images and ground truth data of class labels for unit regions of the patch images. A determination step is performed to segment the patch image using the learning model and the learning data, and to determine for each patch image whether the learning model correctly detected the second unit region. Based on the result of the determination, a weighting step is performed in which a first weight is set for each patch image, with respect to the first loss which is the loss for each patch image. A loss function derivation process for deriving a loss function for a batch composed of the patch images based on the weighted results, comprising: a loss function derivation process for deriving a first loss function as the loss function by adding or averaging the weighted results over the entire batch; An update step in which the parameters of the learning model are updated to minimize the loss function, A learning program that executes the commands.

17. A non-temporary and computer-readable recording medium on which the program described in claim 16 is recorded.

Images