Deep learning model generation apparatus and deep learning model generation method

Abstract

To reduce workload and generate a highly accurate deep learning model.SOLUTION: A deep learning model generation apparatus includes: a segmentation evaluation unit which evaluates, for an image with per-pixel label, that a difference between a per-pixel label put to the image with per-pixel label and a per-pixel label of an estimation result has been reduced; a classification evaluation unit which evaluates, for the image with per-image label, that a difference between a per-image label put to the image with per-image label and a per-image label of an estimation result has been reduced; and an estimation result correction unit which corrects the results of estimation of the segmentation evaluation unit and the classification evaluation unit. A plurality of labels is hierarchized. The estimation result correction unit corrects the results of estimation of the segmentation evaluation unit and the classification evaluation unit based on hierarchical relationships between the hierarchical labels, and performs, based on the corrected estimation results, learning using the segmentation evaluation unit and learning using the classification evaluation unit at the same time.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a deep learning model generation device and a deep learning model generation method. [Background technology]

[0002] For example, a surface texture inspection device that inspects the surface texture of an object such as a steel plate is an inspection device that uses a deep learning (DL) model to estimate the type and shape of defects in an image of the surface of the object, and outputs these values. The trained deep learning model used in the above-mentioned inspection device is a model that is trained by preparing a large number of defect images (marking images) that have been labeled on a pixel-by-pixel basis in advance, and then training the model to minimize the error between the estimated results when each defect image is input and the correct marking image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-154602 [Patent Document 2] Patent Publication No. 2021-149160 Summary of the Invention [Problem to be solved by the invention]

[0004] As mentioned above, in order to build a deep learning model, it was necessary to prepare a large number of marking images, but preparing such a large number of marking images poses a significant workload. A concrete example of the work involved in creating marking images is to project the defect image onto a pen tablet and label (mark) each pixel by defect type, but it takes several minutes to complete marking a single defect image, and marking the hundreds or thousands of defect images required to build a deep learning model is an enormous task, so the challenge is how to create marking images and build a DL model in a labor-efficient manner.

[0005] The following are known techniques / prior art for collecting such marking images as efficiently as possible and building a deep learning model. (1) Crowdsourcing marking This method involves paying an external company or organization to create the marking image. However, while this is not a problem for general images (such as animals or buildings) that are not particularly confidential, it cannot be used for highly confidential images. (2) Automatic marker (Patent Document 1) This method uses a trained model to infer unmarked data, marks the model's output results as they are for confident data and adds them to the training data, and manually marks data for unconfident data and adds it to the training data. This method is expected to reduce the workload because only data for which there is uncertainty is marked. However, since the starting point is a trained model, it is necessary to create a certain amount of marked data in advance, and the workload remains high.

[0006] (3) Artificial generation of marking images using an image generation model (Patent Document 2) This method uses a well-known image generation model (generative adversarial network, GAN) to artificially pad a marking image. The image generation model is a model that outputs an artificial marking image labeled on a pixel-by-pixel basis through the calculation of a trained model based on an arbitrary input random number. (2) As with automatic markers, the starting trained model must be prepared in advance, so the workload is still high. As can be seen from (1), (2), and (3) above, a large number of defect marking images are required to build a pixel-by-pixel defect estimation model, but no method has been proposed at present that can significantly reduce this workload.

[0007] Therefore, an object of the present invention is to provide a deep learning model generation device and a deep learning model generation method that enable the generation of highly accurate deep learning models while reducing the workload. [Means for solving the problem]

[0008] [1] A deep learning model generation device that, when an image is input, generates a deep learning model that estimates at least one of a pixel-by-pixel label, which is a label for each pixel in the image, and a per-image label, which is a label for each image. The deep learning model generation device includes: a segmentation evaluation unit that evaluates, for at least a pixel-by-pixel-labeled image, which is an image to which a pixel-by-pixel label has been assigned, whether a difference between the pixel-by-pixel label assigned to the pixel-by-labeled image and the per-image label of an estimated result has become small; and a classification evaluation unit that evaluates, for at least a per-image-labeled image, which is an image to which a per-image label has been assigned, whether a difference between the image-by-image label assigned to the image-by-image-labeled image and the per-image label of an estimated result has become small; and the segmentation evaluation unit and an estimation result correction unit that corrects the estimation results of the class classification evaluation unit, wherein the multiple labels are hierarchically organized into a hierarchy of labels used in at least one of the segmentation evaluation units and a hierarchy of labels used in at least one of the class classification evaluation units, and the estimation result correction unit corrects the estimation results of the segmentation evaluation unit and the class classification evaluation unit based on the estimation results of the segmentation evaluation unit and the estimation results of the class classification evaluation unit, and the relationship between upper and lower hierarchies of the multiple hierarchical labels, and performs learning by the segmentation evaluation unit and learning by the class classification evaluation unit simultaneously based on the estimation results corrected by the estimation result correction unit. [2] The deep learning model generation device described in [1], wherein the hierarchical labels have a hierarchy of labels used in the class classification evaluation unit in at least the lowest layer, and have at least one hierarchy of labels used in the segmentation evaluation unit that aggregates labels from one or more lower layers other than the lowest layer. [3] The deep learning model generation device according to [1] or [2], wherein the estimation result correction unit ignores the estimation results of labels that do not belong to the next higher layer among the estimation results of multiple labels in the next lower layer when the confidence of the estimation results of the labels in the next higher layer exceeds a first threshold. [4] The deep learning model generation device described in [3], wherein the estimation result correction unit ignores the estimation result of the label in the next higher layer when the confidence of the estimation result of the label in the next lower layer exceeds a second threshold, even if the confidence of the estimation result of the label in the next higher layer exceeds the first threshold. [5] The deep learning model generation device according to [4], wherein the first threshold and / or the second threshold is changed for each label. [6] The deep learning model generation device according to [1] or [2], wherein the estimation result correction unit corrects an output corresponding to the confidence level of the label estimation results by the segmentation evaluation unit and the class classification evaluation unit in the deep learning model. [7] The deep learning model generation device according to [1] or [2], wherein the class classification evaluation unit further performs learning on the pixel-labeled image by adding a term that evaluates whether the difference between the image-by-image label extracted from the pixel-by-pixel labels assigned to the pixel-labeled image and the image-by-image label of the estimation result has become smaller. [8] The deep learning model generation device according to [1] or [2], wherein the segmentation evaluation unit further performs learning on the image-labeled image by adding a term that evaluates whether the difference between the pixel-by-pixel labels of the estimation result and the pseudo-pixel labels obtained by replacing the image-by-image labels for the classification evaluation unit assigned to the image-labeled image with image-by-image labels converted to the image-by-image labels for the segmentation evaluation unit has become smaller. [9] The deep learning model generation device according to [1] or [2], wherein an evaluation index is defined for at least the pixel-labeled image among the pixel-labeled image, the image-labeled image, and an image to which no label is assigned, the evaluation index being used to evaluate whether the difference that can be calculated from the estimation result has become smaller, and the evaluation is performed by adding the evaluation index.

[10] The deep learning model generation device according to [1] or [2], wherein the images are obtained of steel products during and after manufacturing.

[0009]

[11] A deep learning model generation method for generating a deep learning model that, when an image is input, estimates at least one of a pixel-by-pixel label, which is a label for each pixel in the image, and a per-image label, which is a label for each image, the method comprising: a segmentation evaluation step of evaluating, for at least a pixel-labeled image, which is an image to which a pixel-by-pixel label has been assigned, whether a difference between the pixel-by-pixel label assigned to the pixel-labeled image and the per-image label of an estimated result has become small; a classification evaluation step of evaluating, for at least a per-image-labeled image, which is an image to which a per-image label has been assigned, whether a difference between the image-by-image label assigned to the image-by-image labeled image and the per-image label of an estimated result has become small; and a combination of the segmentation evaluation step and the classification evaluation step. a step of correcting estimation results in a step of evaluating the segmentation and classification evaluation, wherein the plurality of labels are hierarchized into a hierarchical layer of labels used in at least one of the segmentation evaluation steps and at least one hierarchical layer of labels used in the classification evaluation step, and the estimation result correction step corrects the estimation results in the segmentation evaluation step and the classification evaluation step based on the estimation results in the segmentation evaluation step and the classification evaluation step, and the relationship between upper and lower hierarchies of the plurality of hierarchized labels, and [Effects of the Invention]

[0010] According to the present invention, by hierarchizing multiple labels and utilizing labeled images, which are images in which a label is assigned to each image, as training data, and performing learning that simultaneously performs segmentation and class classification, it is possible to reduce the workload and generate a highly accurate deep learning model. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 10 is a schematic diagram illustrating hierarchical labeling. [Figure 2]FIG. 2 is a conceptual diagram illustrating processing by the surface texture inspection device. [Figure 3] FIG. 10 is a diagram showing a pixel-by-pixel labeled image to which a pixel-by-pixel label has been assigned. [Figure 4] FIG. 1 is a schematic diagram illustrating the configuration of a deep learning model generation device. [Figure 5] FIG. 10 is a diagram illustrating a first method for obtaining a pseudo pixel-by-pixel label. [Figure 6] FIG. 10 is a diagram illustrating a second method for obtaining a pseudo pixel-by-pixel label. [Figure 7] FIG. 10 is a diagram illustrating a method for extracting an image label from an assigned pixel label. [Figure 8] FIG. 10 is a schematic diagram illustrating the correction by the estimation result corrector when the certainty of the estimation result of the immediately higher label is high. [Figure 9] FIG. 10 is a schematic diagram illustrating correction by the estimation result corrector when the certainty of the estimation result of the next higher label is high. [Figure 10] FIG. 10 is a schematic diagram illustrating the correction by the estimation result corrector when the certainty of the estimation result of the immediately upper label is low. [Figure 11] FIG. 1 is an explanatory diagram of the CutMix technique. [Figure 12] 10 is a graph illustrating the effect of using a deep learning model in a surface texture inspection device. DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The present invention relates to so-called deep learning technology, and to a device (deep learning model generation device) that generates a deep learning model that has been trained using deep learning technology. Deep learning is based on neural networks with deep layers and is an advanced machine learning technique. A neural network consists of an input layer, an intermediate layer (hidden layer), and an output layer, and a deep neural network is a model with multiple intermediate layers. Deep learning uses this deep neural network for learning, and can automatically extract and learn features using large amounts of data.

[0013] This invention uses deep learning technology, particularly for image recognition, and is characterized by using marking data (referred to herein as pixel-labeled images) in which a label (information indicating the correct answer) is assigned to each pixel as training data for the image, as well as labeling data (referred to herein as image-labeled images) in which a label is assigned to each image and which can be created with little effort, as weak training data. Furthermore, this invention is characterized by performing regularization using data including unlabeled data in order to prevent overlearning.

[0014] Labels are assigned to each pixel or each image and consist of a label indicating normality (no defects) (normal label N), a label indicating the presence of defects (defect label D), and a label indicating the type of defect.

[0015] In deep learning, labels assigned to pixels or images in training data are generally treated as independent information. In other words, deep learning models that estimate labels contained in an image from multiple labels typically do not consider the relationships between the multiple labels. However, multiple labels may be related to each other. For example, when estimating defects on the surface of a steel plate, it is possible to classify multiple defects according to the cause and shape of the defect.

[0016] The inventors further considered that by hierarchizing multiple labels in consideration of the relationships between the labels, it may be possible to significantly reduce the workload involved in generating a deep learning model. The layering of labels will be explained in detail below. For example, as shown in Figure 1, multiple labels can be layered as follows: a first layer L1 containing two types of labels, a second layer L2 one layer below the first layer L1 containing three types of labels, and a third layer L3 one layer below the second layer L2 containing seven types of labels. The number of labels and the number of layers are not limited to these.

[0017] The first layer L1 can have two types of labels: a normal label N indicating that there is no defect, and a defect label D simply indicating that there is a defect. The second layer L2 can be made up of three types of labels: a normal A label belonging to normal label N, a flaw A label indicating flaw A belonging to flaw label D, and a flaw B label indicating flaw B belonging to flaw label D. The third layer L3 can be made up of seven types of labels: normal A1 label belonging to normal A label, normal A2 label, flaw A1 label indicating flaw A1 belonging to flaw A label, flaw A2 label indicating flaw A2 belonging to flaw A label, flaw labels B1 and B2 belonging to flaw label B, and flaw C label belonging to flaw label D. The flaw C label does not belong to the flaw A label or flaw B label, but belongs directly to flaw label D. For example, the flaw A1 and flaw A2 labels on the third layer L3 share the same properties, shape, and cause as the flaw A label on the second layer L2. Similarly, the flaw B1 and flaw B2 labels share the same properties, shape, and cause as the flaw B label.

[0018] In other words, layering is the classification of multiple labels into upper layers (larger classifications) and lower layers (more detailed classifications). Labels in higher layers belong to labels in lower layers, but labels in lower layers do not necessarily belong to labels in the layer immediately above. In the example shown in Figure 1, the flaw C label in the third layer L3 does not belong to labels in the second layer L2, which is one layer above, but belongs to flaw label D in the first layer L1, which is two layers above.

[0019] Next, a surface texture inspection device will be described as an example of a device equipped with a deep learning model generated by a deep learning model generation device. <Surface texture inspection device> A surface texture inspection device is a device that inspects the surface texture of an object being inspected, such as a steel product. When a surface image of the object being inspected (hereinafter referred to as the input image) is input, the surface texture inspection device uses a deep learning model to estimate the presence or absence of defects (normal or present), and if defects are present, the type and shape (area) of the defect, and outputs the estimation result. The input image is an image taken by a digital camera or the like, and is composed of, for example, 800 x 600 pixels. Specifically, this includes not only images of steel products taken with optical cameras during and after manufacturing, but also images obtained with equipment that measures or estimates various physical quantities such as the temperature distribution on the surface. It also includes images obtained with internal flaw detection equipment that uses ultrasound or magnetism to confirm the absence of internal flaws. It also includes images obtained with visual inspection equipment when steel products, such as coiled products, are shipped. The defects include, for example, dirt, foreign matter, local temperature drop, and poor coil winding shape.

[0020] The surface texture inspection device estimates and outputs a label for each pixel of the input image from multiple preset types of labels. The output image corresponds to the type of label that can be output, and if three types of labels are supported, three images will indicate the presence or absence of each label and its area.

[0021] FIG. 2 is a conceptual diagram illustrating processing by the surface texture inspection device 1. When an input image I having a flaw B is input to the surface texture inspection device 1, an output image OD corresponding to flaws A to C is output by the deep learning model M1. When the input image I includes flaw B, the deep learning model M1 of the surface texture inspection device 1 shows the shape of the flaw B by indicating the pixels where the flaw B exists as white pixels in the output image DB corresponding to the flaw B in the output image OD, and indicates the pixels of the surface (texture) where no flaw exists as black pixels. Of the output image OD in FIG. 2, output images DA and DC indicate that the corresponding type of flaw does not exist in the input image. The output method is not limited to this, and the output image may be one in which the location estimated to be a flaw is indicated by a rectangular frame and the type of flaw is indicated by a symbol.

[0022] Next, a deep learning model generation device that generates the above-described deep learning model will be described. <Deep learning model generator> A deep learning model generation device is a device that generates a deep learning model by learning from images (hereinafter also referred to as training data) to which a label selected from multiple labels is assigned for each pixel or image. The deep learning model generation device of the present invention uses multiple hierarchical labels as labels, and performs learning taking into account the relationship between upper and lower hierarchies.

[0023] The terms and symbols used in this specification will now be explained. [Images and Pixels] Let I be the total number of images I used in deep learning, and each image I i It is represented as (i∈1,2,…,i,…,I). Let J and K be the maximum number of pixels in the vertical and horizontal directions of an image, respectively. i Each pixel of I i (j,k) It is expressed as (j,k∈J,K).

[0024] [Label per pixel] A pixel-by-pixel label is a label for each pixel in an image. If the deep learning model is used in the surface texture inspection device described above, the pixel-by-pixel label indicates the presence or absence of a defect and the type of defect for each pixel. Each of the multiple pixels in an image may be assigned a pixel-by-pixel label, which is estimated by the deep learning model.

[0025] [Image with pixel-by-pixel labeling] Labeling is performed by a worker, who identifies the presence or absence of a defect and the type of defect for each image, and also assigns a label record to each pixel. Figure 3 shows an image to which pixel-by-pixel labels have been assigned, with the black areas representing pixels identified as having no defect and the white areas representing pixels identified as having defect A. In this specification, an image to which pixel-by-pixel labels have been assigned, including information on the type of defect, is referred to as a pixel-by-labeled image. Typically, when generating a deep learning model, a large number of pixel-by-pixel labeled images are prepared as training data, but the assignment process is a heavy burden. The index set of pixel-by-pixel labeled images is I pix It is expressed as: Pixel-by-pixel labeled image I i The pixel-by-pixel labels assigned to T i (j,k) (0≦T i (j,k) ≦T1). Note that index:i is the number of images I i It is the same as the index of

[0026] [Label for each image] An image label is a label for each image. Basically, one image label is assigned to one image, but multiple image labels may be assigned. [Images with labels added per image] Labeling is performed by a worker who identifies the type of defect for each image and assigns a label record to each image. In this specification, an image to which a label has been assigned is referred to as a labeled image. Basically, it is only necessary to assign one label to each image, so the burden of the labeling work is small. The index set of labeled images for each image is I im It is expressed as: Label each image I i The label for each image is given by t i (0≦t i ≦T2). Note that index:i is the number of images I i It is the same as the index of T1. <T2である。

[0027] 〔segmentation〕 Segmentation is a deep learning method that divides an image into pixels and estimates a label for each pixel. In other words, the training data required for segmentation is an image with labels attached to each pixel (marking data). More specifically, segmentation learning is a learning method that targets an image with labels attached to each pixel and evaluates whether the difference between the estimated label and the labeled area and the image with labels attached to each pixel has become smaller.

[0028] [Classification] Classification is a deep learning method that estimates labels for each image, rather than for each pixel. The training data required for classification may be images with labels for each pixel or images with labels for each image (labeling data). More specifically, learning by classification is a learning method that targets at least one of images with labels for each pixel and images with labels for each image, and evaluates whether the label estimation results match the labels of images with labels for each pixel or images with sub-labels.

[0029] <Configuration of deep learning model generation device> Next, the configuration of the deep learning model generation device will be described. The deep learning model generation device is a device that generates a deep learning model by learning from images that serve as training data. The training data used by the deep learning model generation device includes images with labels assigned to each pixel, images with labels assigned to each image, and unlabeled images. The unlabeled images may not be present.

[0030] As shown in Figure 4, the deep learning model generation device 100 performs learning using a plurality of images I to generate a deep learning model M1 (see Figure 2). The deep learning model generation device 100 uses hierarchical labels such as labels in the first layer L1, labels in the second layer L2, and labels in the third layer L3 as shown in Figure 1. Note that Figure 4 assumes a deep learning model that is a simplified version of U-Net.

[0031] The deep learning model generation device 100 includes a first evaluation unit 10 that uses pixel-labeled images to which labels from the first layer L1 are assigned as pixel-by-pixel labels as training data and outputs estimation results by segmentation; a second evaluation unit 20 that uses pixel-labeled images to which labels from the second layer L2 are assigned as pixel-by-pixel labels as training data and outputs estimation results by segmentation; and a third evaluation unit 30 that uses image-labeled images to which labels from the third layer L3 are assigned as image-by-image labels as training data and outputs estimation results by class classification. The deep learning model generation device 100 also includes an estimation result correction unit 40 that corrects the output (estimation result) of each of the evaluation units 10, 20, and 30.

[0032] The deep learning model generation device 100 simultaneously performs learning using a first evaluation unit 10, learning using a second evaluation unit 20, and learning using a third evaluation unit 30, and further corrects the estimation results by these evaluation units 10, 20, and 30 using an estimation result correction unit 40, and generates a deep learning model M1 by performing learning while evaluating that the difference between the label assigned to the input image I and the label of the estimation result output as the final estimation result (the label of the corrected estimation result) has become smaller.

[0033] Next, the details of each of the evaluation units 10, 20, and 30 will be described. <First evaluation unit, second evaluation unit (segmentation evaluation unit)> The first evaluation unit 10 uses a known method for creating a deep learning model such as a CNN (convolutional neural network) to evaluate whether the difference between the training data and the estimation result has become smaller, and performs learning by segmentation. The first evaluation unit 10 includes an encoder 51 that encodes an image I, which is training data, to extract features, a decoder 52 that decodes and reconstructs the extracted feature representation, and a processing layer 61 for the first hierarchical layer L1.

[0034] The first evaluation unit 10 of this embodiment performs learning by segmentation using a pixel-by-pixel labeled image to which a label of the first layer L1 is assigned from among a plurality of hierarchical labels. That is, the first evaluation unit 10 functions as a segmentation evaluation unit. When a pixel-by-pixel labeled image MD to which labels of the first layer L1 have been assigned is input, the first evaluation unit 10 outputs a plurality of labels to which confidence levels have been assigned as estimation results, and outputs the area of ​​the label with the highest confidence level.

[0035] The second evaluation unit 20 of this embodiment, like the first evaluation unit 10, is equipped with an encoder 51, a decoder 52, and a processing layer 62 for the second layer L2, and performs learning by segmentation by evaluating that the difference between the training data and the estimation result has become smaller. The second evaluation unit 20 of this embodiment performs learning by segmentation using a pixel-by-pixel labeled image to which a label of the second layer L2 is assigned from among a plurality of hierarchical labels. That is, the second evaluation unit 20 also functions as a segmentation evaluation unit. When the second evaluation unit 20 receives a pixel-by-pixel labeled image MD with labels of the second layer L2, it outputs multiple labels with confidence levels as estimation results, and outputs the area with the label with the highest confidence level.

[0036] <Third evaluation section (classification evaluation section)> The third evaluation unit 30 of this embodiment includes an encoder 51 and a processing layer 63 for the third layer L3, and performs learning by class classification using a plurality of labeled images LD (weak teacher data) for each image. Specifically, using a known deep learning model creation method, learning is performed by evaluating whether the labels of the estimated results match the labels of the labeled images LD for each image, using the labeled images LD as the target. In other words, the third evaluation unit 30 functions as a class classification evaluation unit. When the labeled image LD for each image is input, the third evaluation unit 30 estimates a plurality of labels each having a certainty factor assigned thereto, and outputs the label with the highest certainty factor as the estimation result.

[0037] The third evaluation unit 30 of this embodiment performs learning by class classification using labeled images for each image to which labels in the third layer L3 have been assigned.

[0038] The first evaluation unit 10, the second evaluation unit 20, and the third evaluation unit 30 share a common encoder 51. In other words, these evaluation units 10, 20, and 30 share the encoder 51. Similarly, the first evaluation unit 10 and the second evaluation unit 20 share a common decoder 52. This allows the deep learning model generation device 100 of the present invention to simultaneously perform learning using these evaluation units 10, 20, and 30. Note that by simultaneously performing learning, information from each evaluation unit can be commonly used, rather than learning the evaluation units 10, 20, and 30 independently. This allows for more accurate predictions and fewer parameters, which is expected to reduce the time and memory required for learning.

[0039] <Estimation result correction unit> The estimation result correction unit 40 corrects the outputs (estimation results) of the first evaluation unit 10, the second evaluation unit 20, and the third evaluation unit 30 based on the hierarchical relationship of the hierarchical labels and the confidence of the estimation results of the labels. Specifically, the estimation result correction unit 40 sets a threshold value in advance for the confidence of the estimation results of the labels, and performs correction based on the threshold value. The deep learning model generation device 100 performs learning using the estimation result corrected by the estimation result correction unit 40.

[0040] The estimation result correction unit 40 corrects the output of the bottleneck layer for classification and the output of the final layer for segmentation when a deep learning model such as U-Net is used, since the location of the output corresponding to the feature, which is the confidence for each label, differs depending on the selected deep learning model. When an FCN is used, the estimation result correction unit 40 corrects the output of the final layer for classification and segmentation.

[0041] <Evaluation indicators> Next, details of the evaluation indexes used when creating a deep learning model using the above-described deep learning model generation device 100 will be described. Table 1 is a table summarizing each term when the evaluation index is expressed as a mathematical formula. The mathematical formula expressing the evaluation index consists of a first learning term, a second learning term, and a regularization term. The deep learning model generation device 100 performs learning so that the sum of each term (first learning term + second learning term + regularization term) becomes small. The mathematical formula for each term will be described later.

[0042] R1 in the regularization term is a formula related to the pixel ratio (first regularization structure) and image ratio (second regularization structure) described later, and is a formula obtained by adding formulas (19) and (20) described later, and R2 represents a formula related to CutMix (third regularization structure) among the regularization terms. Furthermore, α1, α2, β1, β2, and β3 are the weighting coefficients of the respective terms (α1>0, α2≧0, β1≧0, β2≧0, β3≧0). As shown in Table 1, the first half of the first learning term and the first half of the second learning term are required, but it is sufficient that at least one of the other terms be added. In other words, it is possible to have α2=β1=β2=β3=0, or α2, β2, and β3 can all be set to 0.

[0043] [Table 1]

[0044] <First learning item> The first learning term shown in Table 1 is a mathematical expression of the evaluation index used mainly in segmentation learning, and is the sum of a formula (hereinafter referred to as Formula 1) that calculates the difference between the training data and the estimated result, and a formula (hereinafter referred to as Formula 2) that calculates the difference between the pseudo-pixel-by-pixel label (described later) and the estimated result.

[0045] <Formula 1> Among these, Equation 1 (Equation (1) below) is an equation for calculating the difference between the pixel-by-pixel label assigned to the pixel-labeled image (teacher data) and the estimation result of the pixel-by-pixel label.

[0046]

number

[0047] L(x,y) is a function that calculates the difference between the actual label x and the estimated result y. The formula for this term should be a formula in which the value of L(x,y) monotonically increases as the difference increases. For example, you can use CategoricalCrossentropy, which is mainly used in multi-class classification, BinaryCrossentropy, which is mainly used in two-class classification, or the commonly used MeanSquaredError. L seg (x,y) is the function set for segmentation.

[0048] Here, F(I i ) is the image I i is the estimation result obtained by inputting F(I i ) (j,k) is the estimated label for each pixel. The function represented by g(x) is a function that corrects x (the estimated result of the label for each pixel) by taking into account the relationship between the upper and lower layers of the label. i ) (j,k) ) is a value obtained by correcting the estimation result of the pixel label by taking into account the relationship between the upper and lower layers of the label. The specific correction method will be described later. That is, this term is the actual result T i (j,k) and the corrected pixel-by-pixel label estimation result F(I i ) (j,k) The difference between the calculated values ​​and the calculated values ​​is accumulated for all pixels, and then all the pixel-by-pixel labeled images (I pix ) is calculated.

[0049] <Formula 2> Equation 2 (Equation (2) below) is a formula that uses "pseudo pixel-by-pixel labels," which are the results of pseudo-created pixel-by-pixel labels, to calculate the difference between the pseudo pixel-by-pixel labels and the estimated results of the pixel-by-pixel labels. g(F(Ii ) (j,k) ) is the same as in Equation 1.

[0050]

number

[0051] The pseudo pixel-by-pixel label can be selected appropriately from those described below. For example, the first pseudo pixel-by-pixel label may be selected when applied to labels in the same hierarchy, and the second pseudo pixel-by-pixel label may be selected when applied to labels in a different hierarchy. <First pseudo pixel-by-pixel label> The first pseudo pixel-by-pixel label is obtained by replacing the label estimation result by the deep learning model with the image-by-image label assigned to the image-by-image labeled image. FIG. 5 is a diagram illustrating a method for obtaining pseudo pixel-by-pixel labels. FIG. 5(a) shows the estimation results using a deep learning model, and FIG. 5(b) shows the image-by-image labels t that have been assigned information about defects in the estimation results shown in FIG. 5(a). i The pseudo pixel-by-pixel labels are replaced with As shown in Fig. 5, the pseudo pixel-by-pixel labels are created by changing the labels corresponding to defects indicated by 1 to 3 to the per-image labels assigned to the per-image-labeled image, while leaving the information indicating no defect indicated by 0 as is. Note that although the pixel-by-pixel labels and per-image labels have been described using integers, they may also be symbols. In this case, they can be processed as integers by separately replacing the symbols with integers.

[0052] The pseudo pixel-by-pixel label is T' i (j,k) It is expressed as: Here, image I i This deep learning model M m The estimated results obtained by inputting i ) and image I i The deep learning model M m-1 The estimation results obtained by inputting -1 (I i ) Also, as shown in equations (3) to (5), M EMA1 This deep learning model M m (weights and coefficients) and the previous deep learning model M m-1 is a deep learning model obtained by weighted averaging (exponential averaging), and M EMA2 This deep learning model M m and M EMA2 The deep learning model is obtained by gradually averaging the previous values ​​of M EMA3 The previous deep learning model M m-1 and M EMA3 When the deep learning model is obtained by weighting the previous value of i M EMA1 , M EMA2 , M EMA3 The estimation results obtained by inputting EMA1 (I i ), F EMA2 (I i ), F EMA3 (I i ) where η is the weighted average coefficient (0≦η≦1). EMA2 and M EMA3 The initial value of M -1 can be set to zero.

[0053]

number

[0054] The function of the pseudo pixel-by-pixel labels, i.e., the estimation result of the pixel-by-pixel labels F(I i ) (j,k) The defect information is stored as a label for each image. i The function for converting into can be any one of the following equations (6) to (10).

[0055]

number

[0056] In addition, the estimated result F(I i ) and F -1 (I i) and calculate the weighted average EMA1(I i ), EMA2(I i ), EMA3(I i ) may be used to obtain one of the following equations (14) to (16) instead of equations (6) to (10).

[0057]

number

[0058] <Second pseudo pixel-by-pixel label> The second pseudo pixel-by-pixel label is an image-by-image label t' that corrects information about defects in the pixel-by-pixel label estimation results by taking into account the relationship between the label hierarchies. i The image label t' is replaced by i is the label for each image t i The image-specific labels are corrected by taking into account the relationship between the label hierarchies. The estimation result of the pixel-specific labels is the same as the first pseudo pixel-specific label, F(I i ), F -1 (I i ), F EMA1 (I i ), F EMA2 (I i ), F EMA3 (I i ), EMA1(I i ), EMA2(I i ), EMA3(I i ) may be used.

[0059] Figure 6 shows the estimation results F(I i ) (j,k)6(a) is a diagram illustrating a method for obtaining second pseudo pixel-by-pixel labels for the second layer L2 using the deep learning model. FIG. 6(a) shows the estimation results using a deep learning model, which are pixel-by-pixel label estimation results for the second layer L2. The image-by-image label t assigned for the third layer L3 is flaw B1. As shown in FIG. 1, the layer above flaw B1 belongs to flaw B, so the image-by-image label t' for the second layer L2 is flaw B. FIG. 6(b) shows pseudo pixel-by-pixel labels obtained by replacing information related to flaws in the estimation results shown in FIG. 6(a) with the image-by-image label t' for the second layer L2.

[0060] As shown in Figure 6, the second pseudo pixel-by-pixel label is created by leaving the information indicating no flaw indicated by 0 as it is, and changing the label corresponding to a flaw to flaw B, taking into account the relationship between the label hierarchies.

[0061] <Second learning item> The second learning term shown in Table 1 is a mathematical expression of the evaluation index used mainly in learning by class classification, and is the sum of a formula (hereinafter referred to as Formula 3) that calculates the difference between the training data and the estimated result, and a formula (hereinafter referred to as Formula 4) that calculates the difference between the image-specific labels extracted from the pixel-specific labeled image and the estimated result.

[0062] <Formula 3> Among these, Equation 3 (Equation (17) below) is an equation for calculating the difference between the per-image label assigned to the per-image labeled image and the per-image label of the estimation result.

[0063]

number

[0064] L class (x,y) is a function that calculates the difference between the actual label x and the estimated result y set for classification. i is the image-specific label. F(I i ) class Here is the image I i These are the estimated label results for each image obtained by inputting the above data into a deep learning model. g(F(I i ) class ) is the estimated result of the label for each image F(I i ) class is a value corrected by a function g taking into account the relationship between the upper and lower layers of the labels. A specific correction method using the function g will be described later. That is, the second learning term, Equation 3, is the result of labeling each image t i and the estimated label for each image F(I i ) class The difference between the corrected values ​​and the calculated values ​​is summed for all images.

[0065] <Formula 4> Equation 4 (Equation (18) below) is a mathematical formula for calculating the difference between the label for each image extracted from the pixel-labeled image and the estimation result.

[0066]

number

[0067] h(T i (j,k) ) is the assigned pixel-by-pixel label T i (j,k) Hereinafter, we will explain the function h that extracts the image label from the pixel-by-pixel label assigned to the pixel-labeled image. Usually, one type of pixel label is assigned to one pixel-labeled image, but multiple types are also possible. If multiple pixel labels are assigned to one pixel-labeled image as shown in Figure 7, the difference in the evaluation index for class classification can be evaluated using, for example, one of the following methods. [1] The label “3” with the largest number of pixels is considered the actual label for each image (= correct answer), and the result is evaluated based on whether it matches the estimated label for each image. [2] All labels (1, 2, 3) are considered as the actual labels (= correct answers) for each image, and the estimated image labels are evaluated based on whether they are included in 1, 2, 3.

[0068] <Correction taking into account the relationship between upper and lower hierarchies> The following describes the correction that is performed by the estimation result corrector 40 and that takes into account the relationship between the upper and lower hierarchies and is used in the calculation of the first learning term and the second learning term described above. In the formula, the function represented by g(x) is a function that corrects x (the label estimation result) taking into account the relationship between upper and lower hierarchies. Any appropriate correction method can be selected as long as it takes into account the relationship between upper and lower hierarchies. Examples of correction methods are explained below.

[0069] <Correction method when the confidence level of the label estimation result of the next higher level is high> In this method, the certainty of the estimation result of the label in the next higher layer is referenced to the layer to which the label to be corrected belongs. Here, if the certainty of the estimation result of the label in the next higher layer is equal to or greater than a first threshold T1, that is, if it is considered that the estimation result of the label in the next higher layer is likely to be correct, the estimation result correction unit 40 corrects the label to be corrected so that it belongs to the estimation result of the label in the next higher layer. In other words, if the certainty of the estimation result of the label in the next higher layer is sufficiently high, the estimation result correction unit 40 changes the label to be corrected to a label that belongs to the estimation result of the label in the next higher layer, even if it does not belong to the estimation result of the label in the next higher layer.

[0070] For example, if the first threshold T1 of the confidence level of the estimation result of the label of the second layer L2 (one layer above) by the second evaluation unit 20 is set to 0.7, and the second evaluation unit 20 outputs the flaw A label with a confidence level of 0.8 (first threshold T1 or more) and the third evaluation unit 30 outputs the flaw B1 label that does not belong to the flaw A label with a confidence level of 0.3 as shown in Fig. 8, the estimation result correction unit 40 corrects the estimation result of the third evaluation unit 30 to the flaw A2 label with the highest confidence level out of the flaw A1 label and the flaw A2 label that belong to the flaw A label. The estimation result of the area at that time should be the area of ​​the flaw A label, not the area of ​​the flaw B label. If the label estimation result by the third evaluation unit 30 belongs to the label estimation result of the next higher layer, the label estimation result by the third evaluation unit 30 is adopted and output as is. The region estimation result at that time can be the region of the adopted label.

[0071] On the other hand, even if the confidence level of the estimation result of the label one layer higher is equal to or higher than the first threshold T1, if the confidence level of the estimation result of the label to be corrected is equal to or higher than the second threshold T2, the estimation result of the label to be corrected is adopted even if it does not belong to the estimation results of the label one layer higher.

[0072] For example, if the first threshold T1 and the second threshold T2 are 0.7, and the second evaluation unit 20 outputs a flaw A label with a certainty of 0.8, and the third evaluation unit 30 outputs a flaw C label (independent label) that does not belong to the flaw A1 label with a certainty of 0.8 (equal to or greater than the second threshold T2), as shown in Fig. 9, the estimation result correction unit 40 does not correct the estimation result of the third evaluation unit 30. In other words, if the certainty of the label estimation result by the third evaluation unit 30 is sufficiently high, the estimation result of the flaw A label by the second evaluation unit 20 is ignored, and the flaw C label is adopted.

[0073] However, since the estimation result of the area (pixel-by-pixel label) at that time is not included in the flaw C label of the third layer L3 that was adopted, it is sufficient to use an area based on the estimation result of at least one area in a layer above that has an estimation result for the area. For example, when using the second layer L2, the area labeled with the flaw A, which has the highest confidence, or the area obtained by adding together all of the flaw A and flaw B labels, may be used. Also, the area labeled with the flaws in the first layer L1 may be used. As shown in Figure 1, by segmenting the top first layer L1 into two types: no flaw (normal label N) and flaw (flaw label D), it is possible to estimate the area corresponding to the flaw even for labels in lower layers that do not have markings (per-pixel labels).

[0074] <Correction method when the confidence level of the label estimation result in the next higher layer is low> If the confidence level of the estimation result of the label in the next higher layer is smaller than the first threshold T1, that is, if it is considered that the estimation result of the label in the next higher layer is unlikely to be correct, the estimation result correction unit 40 may perform correction using at least one of the following (1) to (3). (1) The estimation result of the label one level above is not taken into account (priority is not given) (2) Consider (prioritize) the estimation result of the label one level above (3) Both (1) and (2) are adopted.

[0075] For example, when the first threshold T1 and the second threshold T2 are set to 0.7, and the second evaluation unit 20 outputs the flaw A label with a certainty of 0.6 (less than the threshold T1), as shown in FIG. 10, the correction methods are shown for each of (1) to (3).

[0076] In the case of (1), the estimation result of the flaw B2 label, which has the highest degree of certainty among the estimation results of the third evaluation unit 30, is adopted, regardless of the degree of certainty of the label estimation result by the second evaluation unit 20. As there is no area estimation result in the third layer L3 belonging to the flaw B2 label, the estimation result of the area of ​​label B, to which label B2 belongs, in the second layer L2, which is a higher layer and has area estimation results, can be adopted.

[0077] In the case of (2), priority is given to flaws A1 and A2 belonging to flaw A, which have the highest certainty in the label estimation results by the second evaluation unit 20, and the estimation result of the flaw A2 label with the highest certainty by the third evaluation unit 30 is adopted. In this case, the estimation result of the area A to which the flaw A2 label belongs can be adopted as the estimation result of the area.

[0078] In the case of (3), multiple labels may be used, so multiple flaw B2 labels used in (1) and multiple flaw A2 labels used in (2) are used. Multiple area estimation results may also be used, so multiple area estimation results for flaw label B used in (1) and flaw label A used in (2) may be used.

[0079] By adopting such a correction, it becomes possible to output segmentation even for labels that have only labeling data (images with labels assigned to each image) and no marking data (images with labels assigned to each pixel).

[0080] <Regularization term> The regularization term shown in Table 1 targets at least pixel-labeled images out of pixel-labeled images, image-labeled images, and unlabeled images, and is an index to evaluate whether the label information of the pixel-labeled images and the image-labeled images is not essential and whether the difference that can be calculated from the estimation results has become smaller. The regularization term does not require a label to be attached to the image, so it can also be used for images without labels. The regularization term can be expressed as the following equation (19).

[0081]

number

[0082] The regularization term has an appropriate regularization structure to prevent deep learning from progressing in an inappropriate direction, and evaluates whether the difference that can be calculated using the first, second, and third regularization structures described below is small. Each regularization structure is explained below.

[0083] <First regularization structure> The first regularization structure utilizes information on the defect pixel ratio. The defect pixel ratio is the ratio of the number of pixels that make up the defect to the number of pixels in the image (480,000 for an 800x600 image). The first regularization structure performs regularization based on this defect pixel ratio. Specifically, the difference between the distribution of pixel ratios of flaws calculated from the estimation results of multiple images and a preset target distribution of pixel ratios of flaws (target distribution) is added. The distribution of pixel ratios of flaws is a probability distribution of pixel ratios calculated for each type of flaw.

[0084] The evaluation index for the first regularization structure can be expressed as the following equation (20).

[0085]

number

[0086] l(X, Y) is a function that calculates the difference in probability distribution between distribution X and distribution Y, and may be calculated using squared error, Kullback-Leibler distance, or the like. U i (F(I i )) is the image I i This is a function that inputs the above and calculates the distribution of pixel ratios of flaws.

[0087] Target distribution of defect pixel ratio U aim is set based on at least one of the labels assigned to the pixel-by-pixel labeled image and the labels assigned to the image-by-image labeled image. (1) When targeting images with labels assigned to each pixel and images with labels assigned to each image, the distribution calculated based on the assigned labels can be used as the target distribution of the defect pixel ratio. (2) When unlabeled images are also targeted (i.e., when pixel-by-pixel labeled images, image-by-image labeled images, and other images are targeted), unlabeled images may also be considered to have the same distribution as the target distribution calculated in (1) above and set. Alternatively, if prior information on each target distribution is already available, for example, in the case of a surface texture inspection task performed by a human eye, that prior information may be used to set the target distribution. For example, the target distribution of pixel ratios for defects A, B, and C is U aim Then, U aim =(0.1,0.2,0.3). β 21 is the weighting function (β 21 ≧0). That is, the regularization term for the first regularization structure is the target distribution U aim The difference between the distribution of pixel ratios of the estimated flaws and the image Ii is calculated. When it is particularly desired to improve the accuracy of a predetermined defect, the weight (importance) may be set so that the difference regarding the predetermined defect becomes large.

[0088] <Second regularization structure> The second regularization structure utilizes information on the defect image ratio. The defect image ratio is the total number of images used in the second regularization structure (for example, I pix , I im , I non If you are targeting pix +I im +I non ) is the ratio of the number of defect images to the total number of images. The second regularization structure performs regularization based on this defect image ratio. Specifically, it adds the difference between the distribution of defect image ratios calculated from the estimation results and a preset target distribution of defect image ratios. The evaluation index for the second regularization structure can be expressed as the following equation (21).

[0089]

number

[0090] V i (F(I i )) is the image I i is a function that inputs the above and calculates the distribution of the defect image ratio.

[0091] Target distribution of defect image ratio V aim is set based on at least one of the labels assigned to the pixel-by-pixel labeled image and the labels assigned to the image-by-image labeled image. For example, the target distribution of image ratios for defects A, B, and C is V aim Then, V aim =(0.1,0.01,0.05). β 22is the weighting function (β 22 ≧0). That is, the regularization term for the second regularization structure is the target distribution V aim The difference between the distribution of the image ratio of the estimated flaws and the image Ii is calculated.

[0092] <Third regularization structure> The third regularization structure utilizes "CutMix," a well-known technique for expanding image data in deep learning models. CutMix is ​​a technique that creates new training data by combining two images, and then uses this new training data to improve image recognition accuracy. CutMix is ​​described in the following literature: et al. S. Yun. Cutmix: Regularization strategy to train strong classifiers with localizable features. In arXiv:1905.04899.

[0093] The CutMix technique will be described with reference to FIG. CutMix utilizes image processing (hereinafter referred to as composite image processing) that pastes a second image D2 (mixed image) onto a first image D1 (original image).

[0094] First, two input images (first image D1, second image D2) are prepared. The first image D1 is a pixel-by-pixel labeled image MD, a picture-by-picture labeled image LD, and an image with no labels. The second image D2 is a randomly determined image, and like the original image, is a pixel-by-pixel labeled image MD, a picture-by-picture labeled image LD, and an image with no labels.

[0095] Next, the two input images D1 and D2 are combined using a composite image processing (flow F2). Specifically, a predetermined rectangular area S is cut out of the second image D2, and this area S is pasted in the same position on the first image D1. This generates a composite input image DM as new training data. Note that the rectangular area S for the composite image processing may have a shape other than a rectangle, and its size is not limited. Next, estimation is performed for each of the three images D1, D2, and DM using a deep learning model, obtaining an estimation result RD1 for the first image D1, an estimation result RD2 for the second image D2, and an estimation result RDM for the composite input image DM. Meanwhile, the estimation result RD1 for the first image D1 and the estimation result RD2 for the second image D2 are combined using composite image processing (flow F3) to obtain a composite estimation result RD12. The deep learning model generation device 100 corrects the deep learning model based on an evaluation index that evaluates whether the difference between the estimation result RDM of the synthetic input image DM and the synthetic estimation result RD12 has become smaller. Note that the number of synthetic images to be data-expanded depends on the number of combinations of two images to be targeted and the number of synthetic image processing methods, but is not particularly limited and may be any number.

[0096] As mentioned above, the evaluation indexes for all regularization terms do not require the performance of the labels assigned to the images, so they can also include images without labels. In other words, the images targeted by the evaluation indexes for regularization terms are the index set I of labeled images for each image. im and / or a set of indexes for pixel-by-pixel labeled images I pix However, the index set I of unlabeled images is non may also include.

[0097] <Deep learning model generation method> A description will now be given of a deep learning model generation method using the above-described deep learning model generation device 100. The deep learning model generation method includes a label layering step, a marking step, an estimation result correction step, a first learning step (segmentation evaluation step), and a second learning step (classification evaluation step) that is performed simultaneously with the first learning step.

[0098] In the label stratification process, the operator determines a plurality of labels required for image recognition and further stratifies the plurality of labels. Alternatively, a plurality of labels are determined, and these plurality of labels are subdivided to set a plurality of stratified labels. Here, N1 types of labels are set as the first layer, N2 types of labels are set as the second layer, and N3 types of labels (0 < N1 < N2 < N3) are set as the third layer. Here, N1 types of per-pixel labels are set as the first layer, N2 types of per-pixel labels are set as the second layer, and N3 types of per-image labels are set as the third layer. The above settings, for example, the setting of the number of layers, can be appropriately changed if the per-image label is estimated at the lowest layer and the per-pixel label obtained by aggregating the labels of one or more lower layers is estimated at other layers than the lowest layer. For example, the per-image label may be estimated for the first layer and the third layer, and the per-pixel label may be estimated for the second layer.

[0099] In the marking process, the operator marks the images taken using the measuring device for the images having the labels belonging to the first layer to create a per-pixel label-added image. Similarly, the operator marks the images having the labels belonging to the second layer to create a per-pixel label-added image and stores these in the storage device. Next, for the images having the labels belonging to the third layer, labeling is performed to create a per-image label-added image and store it in the storage device. That is, for the images corresponding to the labels belonging to the third layer, one label is assigned per image without marking each pixel.

[0100] The deep learning model generation device 100 generates a deep learning model by learning using the per-pixel label-added image and the per-image label-added image stored in the storage device. In the first learning process (segmentation evaluation process), the first evaluation unit 10 and the second evaluation unit 20 learn by evaluating whether the difference between the label and label area estimation results and the pixel-labeled image has become smaller, and output the label estimation results. On the other hand, in the second learning process (classification evaluation process), the third evaluation unit 30 learns by evaluating whether the label estimation results match the labels of the labeled images for each image, and outputs the label estimation results.

[0101] The deep learning model generation device 100 simultaneously performs learning by the first evaluation unit 10 and the second evaluation unit 20, and learning by the third evaluation unit 30. That is, segmentation by the first evaluation unit 10 and the second evaluation unit 20 and a class classification task by the third evaluation unit 30 are performed in a multitasking manner.

[0102] In the estimation result correction process, the estimation result correction unit 40 of the deep learning model generation device 100 corrects the estimation result of the first evaluation unit 10, the estimation result of the second evaluation unit 20, and the estimation result of the third evaluation unit 30 based on the confidence levels of the estimation results of the first evaluation unit 10, the estimation result of the second evaluation unit 20, and the estimation result of the third evaluation unit 30, respectively. The deep learning model generation device 100 generates a deep learning model by repeating learning using the corrected label estimation results.

[0103] According to the above embodiment, by hierarchizing multiple labels and utilizing labeled images, which are images in which a lower-level label is assigned to each image, as training data, and performing learning that simultaneously performs segmentation and class classification, it is possible to reduce the workload and generate a highly accurate deep learning model. In other words, by using labeled images that lack pixel-level label information but can be prepared in large quantities, the amount of data used for learning increases, and by being able to utilize more detailed label information and information on the relationships between upper and lower hierarchies, it is expected that a highly accurate pixel judgment model can be learned.

[0104] Furthermore, even when an independent label that does not belong to a higher-level label is set, the independent label can be learned by the third evaluation unit 30, which has a low workload, thereby enabling learning to proceed with less effort.

[0105] FIG. 12 is a graph illustrating the effect of using a deep learning model generated by the deep learning model generation device 100 of the present invention in a surface texture inspection device. The graph shows the degree of match rate for the five flawed labels out of a total of seven labels (two flaw-free labels and five flawed labels). In FIG. 12, "Comparative Example" shows the results of a deep learning model that uses a large amount of conventional marking data, sets the total number of labels to seven, and performs only segmentation without considering the relationship between the upper and lower labels in FIG. 1. "Present Invention" shows the results of a deep learning model generated according to the present invention, taking into account the relationship between the upper and lower labels shown in FIG. 1.

[0106] As shown in FIG. 12, the matching rate of the type of flaws could be significantly improved by using the method of the present invention. [Explanation of symbols]

[0107] 100...deep learning model generation device, 1...surface texture inspection device, 10...first evaluation unit, 20...second evaluation unit, 30...third evaluation unit, 40...estimated result correction unit, ID...input image, LD...image with labels assigned per image, M1...deep learning model, MD...image with labels assigned per pixel, OD...output image.

Claims

1. A deep learning model generation device that, when an image is input, generates a deep learning model that estimates at least one of a pixel label, which is a label for each pixel in the image, and an image label, which is a label for each image, a segmentation evaluation unit that evaluates whether a difference between a pixel-by-pixel label assigned to a pixel-by-pixel labeled image and a pixel-by-pixel label of an estimation result has become smaller, for a pixel-by-pixel labeled image that is an image to which a pixel-by-pixel label has been assigned; a classification evaluation unit that evaluates whether a difference between the per-image label assigned to the per-image-labeled image and the per-image label of the estimation result has become smaller, for at least the per-image-labeled image that is an image assigned with the per-image label; an estimation result correction unit that corrects the estimation results of the segmentation evaluation unit and the classification evaluation unit, The plurality of labels are hierarchically organized into a hierarchy of labels used in at least one of the segmentation evaluation units and a hierarchy of labels used in at least one of the classification evaluation units, the estimation result correction unit corrects the estimation results of the segmentation evaluation unit and the classification evaluation unit based on the estimation results of the segmentation evaluation unit and the estimation results of the classification evaluation unit and on the relationship between upper and lower hierarchies of the hierarchical labels; and A deep learning model generation device that simultaneously performs learning by the segmentation evaluation unit and learning by the class classification evaluation unit based on the estimation result corrected by the estimation result correction unit.

2. the plurality of hierarchical labels have, at least in the lowest layer, a layer of labels used by the classification evaluation unit, The deep learning model generation device according to claim 1 , further comprising at least one layer of labels used in a segmentation evaluation unit that aggregates labels from one or more lower layers in addition to the lowest layer.

3. 3. The deep learning model generation device according to claim 1, wherein the estimation result correction unit ignores estimation results of labels that do not belong to the next higher layer among the estimation results of the multiple labels in the next lower layer when the certainty of the estimation results of the labels in the next higher layer exceeds a first threshold.

4. 4. The deep learning model generation device according to claim 3, wherein the estimation result correction unit ignores the estimation result of the label in the next higher layer when the certainty of the estimation result of the label in the next lower layer exceeds a second threshold, even if the certainty of the estimation result of the label in the next higher layer exceeds the first threshold.

5. The deep learning model generation device according to claim 4 , wherein the first threshold and / or the second threshold is changed for each label.

6. 3. The deep learning model generation device according to claim 1, wherein the estimation result correction unit corrects an output corresponding to a degree of certainty of the label estimation results by the segmentation evaluation unit and the class classification evaluation unit in the deep learning model.

7. 3. The deep learning model generation device according to claim 1, wherein the class classification evaluation unit further performs learning by adding, to the pixel-labeled image, a term that evaluates whether a difference between a per-image label extracted from a pixel-label assigned to the pixel-labeled image and the per-image label of the estimation result has become smaller.

8. 3. The deep learning model generation device according to claim 1, wherein the segmentation evaluation unit further performs learning for the image-labeled image by adding a term that evaluates whether a difference between a pseudo-pixel label obtained by replacing an image label for a classification evaluation unit assigned to the image-labeled image with an image label converted into an image label for a segmentation evaluation unit and the pixel label of the estimation result has become smaller.

9. 3. The deep learning model generation device according to claim 1 or 2, wherein an evaluation index is defined for at least the pixel-labeled image out of the pixel-labeled image, the image-labeled image, and an image to which no label is assigned, the evaluation index being for evaluating whether a difference that can be calculated from an estimation result has become small, and the evaluation is performed by adding the evaluation index.

10. The deep learning model generation device according to claim 1 or claim 2, wherein the images are images obtained of steel products during and after manufacturing.

11. A deep learning model generation method for generating a deep learning model that, when an image is input, estimates at least one of a pixel label, which is a label for each pixel in the image, and an image label, which is a label for each image, the method comprising: a segmentation evaluation step of evaluating whether a difference between a pixel-by-pixel label assigned to a pixel-by-pixel labeled image and a pixel-by-pixel label of an estimation result has become smaller, for the pixel-by-pixel labeled image, which is an image to which a pixel-by-pixel label has been assigned; a classification evaluation step of evaluating whether a difference between the per-image label assigned to the per-image-labeled image and the per-image label of the estimation result has become smaller, for at least the per-image-labeled image, which is an image assigned with a per-image label; an estimation result correction step of correcting estimation results in the segmentation evaluation step and the classification evaluation step, the plurality of labels are hierarchically organized into a hierarchy of labels used in at least one of the segmentation evaluation steps and a hierarchy of labels used in at least one of the classification evaluation steps; the estimation result correction step corrects the estimation results in the segmentation evaluation step and the classification evaluation step based on the estimation results in the segmentation evaluation step and the classification evaluation step, and on the relationship between upper and lower hierarchies of the hierarchical labels; A deep learning model generation method in which learning in the segmentation evaluation process and learning in the class classification evaluation process are performed simultaneously based on the estimation results corrected in the estimation result correction process.

Images