Pseudo segmentation model generation device and pseudo segmentation model generation method

Abstract

To provide a technique to determine a defect type for every pixel while reducing labor.SOLUTION: A pseudo segmentation model generation device infers a defect type of a defective image in which a defect is captured by using a classification model, generates a determination basis image indicating which pixel of the defective image easily affects on the inference of the defect type, determines a representative pixel representing the presence of a defect on the basis of luminance values of the defective image and determination basis image, determines an area in the defective image corresponding to the defect by using a target area determination model for designating an arbitrary pixel in the image to determine an area corresponding to a target including the designated pixel in the image, and for every pixel of the defective image, applies information corresponding to the defect type to every pixel of an area determined to correspond to the defect, thereby generating a pseudo segmentation model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a pseudo segmentation model generating device and a pseudo segmentation model generating method. [Background technology]

[0002] In recent years, machine learning models have been used in various technical fields to realize various types of information processing such as image processing and language processing. For example, in the case of defect inspection of an object, an inspection device is used that estimates the defect type of a defect captured in an image of the object by using a deep learning (DL) model for the captured image.

[0003] Known methods of estimating the defect type using a DL model include estimation by classification using a classification model that estimates the type of a defect captured in a captured image from the entire captured image, and estimation by segmentation using a segmentation model that estimates the type of a defect captured in each pixel in the captured image. Also known is estimation by detection using a detection model that estimates an area in the captured image where an object such as a defect exists and surrounds the object with a rectangle or the like.

[0004] When estimating the defect type using a segmentation model in an inspection device, when training the DL model used in the inspection device, a large number of defect images (marking images) labeled on a pixel-by-pixel basis are prepared, and the parameters of the DL model are adjusted so that the error between the output result when each defect image is input into the DL model and the correct marking image is reduced.

[0005] Such marking images are created, for example, by manually labeling (marking) a defect image displayed on a display by adding a label for each defect type to each pixel. It takes several minutes to manually create one defect image, and building a DL model typically requires hundreds to thousands of marking images, so building a DL model requires long and tedious work. For this reason, technologies have been proposed to reduce the labor required for creating such marking images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-154602 A [Patent Document 2] Patent Publication No. 2021-149160 Summary of the Invention [Problem to be solved by the invention]

[0007] In these conventional techniques, a marking image is created based on the pixel-by-pixel output result of a DL model, and it is still necessary to create a marking image in order to train a DL model for creating the marking image.

[0008] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a pseudo segmentation model generating device and a pseudo segmentation model generating method that are capable of determining a defect type for each pixel of image data by using a machine learning model while reducing the effort required for generating a marking image. [Means for solving the problem]

[0009] In order to solve the above problem, according to one aspect of the present invention, there is provided a pseudo segmentation model generating device that pseudo-generates a deep learning model that performs inference on an image by segmentation, the pseudo segmentation model generating device including an image acquiring unit that acquires a defect image that is an image containing a defect, a defect type inference unit that infers a defect type of a defect that is shown in the defect image by using a deep learning model that performs inference on an image by classification, a judgment basis image generating unit that generates a judgment basis image that indicates which pixel of the defect image is likely to be affected in the inference of the defect type, a representative pixel determining unit that determines a representative pixel that can be regarded as a pixel that represents the presence of the defect based on a luminance value of the defect image and a luminance value of the judgment basis image, and a pixel determining unit that determines an arbitrary pixel in the image. a pixel information assigning unit that assigns information corresponding to the defect type to each pixel in the defect image in an area determined by the area determination unit to correspond to the defect by the area determination unit, and assigns information corresponding to the absence of a defect to each pixel in the defect image other than the area determined by the area determination unit to correspond to the defect by the area determination unit; and an inference result output unit that outputs the defect type and the information assigned by the pixel information assigning unit as an inference result for the defect image.

[0010] In addition, in the pseudo segmentation model generating device of the present invention, the representative pixel determination unit may generate a summed image by adding up the brightness value of the judgment basis image and the brightness value of the target image, and determine the representative pixel based on the pixel with the highest brightness in the summed image.

[0011] In addition, in the pseudo segmentation model generating device of the present invention, the representative pixel determination unit may weight one of the luminance values ​​of the judgment basis image and the luminance values ​​of the target image, and add them together to generate an added image, and determine the representative pixel based on the pixel with the highest luminance in the added image.

[0012] In order to solve the above problem, according to another aspect of the present invention, there is provided a pseudo segmentation model generating method for pseudo-generating a deep learning model for inferring an image by segmentation, the pseudo segmentation model generating method including: an image acquiring step for acquiring a defect image which is an image including a defect; a defect type inference step for inferring a defect type of a defect included in the defect image by using a deep learning model for inferring an image by classification; a judgment basis image generating step for generating a judgment basis image which indicates which pixel of the defect image is likely to be affected in the inference of the defect type; a representative pixel determining step for determining a representative pixel which can be regarded as a pixel representative of the presence of the defect based on a luminance value of the defect image and a luminance value of the judgment basis image; and a representative pixel determining step for determining a representative pixel which can be regarded as a pixel representative of the presence of the defect by specifying an arbitrary pixel in the image. A pseudo segmentation model generating method is proposed, which includes an area determination step of inputting the defect image and determining an area in the defect image corresponding to the defect by specifying the representative pixel using an object area determination model that determines an area corresponding to an object including a specified pixel in the image; a pixel information assignment step of assigning information corresponding to the defect type to each pixel in the defect image in an area determined in the area determination step to correspond to the defect in the area determination step, and assigning information corresponding to the absence of a defect to each pixel other than the area determined in the area determination step to correspond to the defect; and an inference result output step of outputting the defect type and the information assigned in the pixel information assignment step as an inference result for the defect image.

[0013] In addition, in the pseudo segmentation model generating method of the present invention, the representative pixel determination step may generate a summed image by adding up the brightness value of the judgment basis image and the brightness value of the target image, and determine the representative pixel based on the pixel with the highest brightness in the summed image.

[0014] In addition, in the pseudo segmentation model generating method of the present invention, the representative pixel determination step may include weighting one of the luminance values ​​of the judgment basis image and the luminance values ​​of the target image and adding them together to generate an added image, and determining the representative pixel based on the pixel with the highest luminance in the added image. Effect of the Invention

[0015] According to the present invention, it is possible to determine the defect type of a defect for each pixel of image data by utilizing a machine learning model while reducing the effort required for generating a marking image. [Brief description of the drawings]

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0018] [Summary of the Disclosure] 1 shows a defect type determination model 30 and a defect area determination model 50 according to an embodiment of the present invention. According to this embodiment, as shown in FIG. 1, defect type data is acquired for each pixel using two machine learning models: a defect type determination model 30 as a classification model trained to determine the type (defect type, for example, cracks, dents, powder, indentations, etc.) of a defect (for example, a flaw on the surface of a steel plate, an uneven portion, an abnormal portion of roughness, etc.) from image data (defect image) obtained by capturing an image of an inspection object (for example, a steel plate, a steel pipe, a bar, an aluminum material, etc.), and a defect area determination model 50 trained to determine a defect area (an inner area along the outer periphery of a defect that is occupied by a defect) based on the image data and a representative pixel obtained from a determination basis image that is the basis for the defect type determination by the defect type determination model 30.

[0019] The defect type determination model 30 according to the embodiment of the present invention can be realized by, for example, a DL model such as a convolutional neural network, and receives image data of an object to be inspected as an input, and outputs defect type data indicating the defect type of a defect captured on the object to be inspected as a determination result, as shown in Fig. 1. The defect type determination model 30 further outputs a determination basis image in a heat map format indicating pixels or image areas that are the basis for determining the type of the defect of the object to be detected.

[0020] On the other hand, the defect area determination model 50 according to an embodiment of the present invention can be realized, for example, by a DL model such as a convolutional neural network, and as shown in FIG. 1, receives as input image data input to the defect type determination model 30 and a representative pixel determined from the determination basis image output from the defect type determination model 30, and outputs defect area data indicating the defect area to be detected as the determination result.

[0021] Such a defect type determination model 30 (trained defect type determination model 30) can be generated from the training target defect type determination model 20, which is a classification model, by using a first training data set consisting of training images and defect type labels indicating the correct defect types of the defects captured in the training images.

[0022] On the other hand, the defect area determination model 50 (trained defect area determination model 50) can be generated from the defect area determination model 40 to be trained by using a second training data set consisting of a training image, a designation of an arbitrary pixel in the training image, and a defect area label that includes the arbitrary pixel in the training image and indicates an area (the correct answer) occupied by a defect appearing in the training image. In addition, since a trained model is publicly available, the trained defect area determination model 50 can also be generated by utilizing such a publicly available model.

[0023] Conventionally, in order to estimate the type of defect to be detected in image data for each pixel, it was necessary to use a machine learning model consisting of a segmentation model. In the process of training the machine learning model consisting of a segmentation model, a training data set created by manually marking the defect type label to be detected for each pixel on the training image was used, so the burden of the marking work was large. In addition, when dealing with defects, technical expertise on defects was required to determine the type of defect, and further, confidentiality from other companies was required.

[0024] Therefore, in the present invention, the assignment of defect type labels indicating the type of defect is performed by a highly specialized defect inspector, but rather than using a segmentation model which imposes a heavy burden of marking each pixel, a classification model which does not require marking and is lighter in burden is used, and a first training data set consisting of training images and defect type labels assigned on a pixel-by-pixel basis is used to train the defect identification and judgment model 20 to be trained.

[0025] On the other hand, since it is relatively easy for an ordinary person without specialized knowledge to determine the area where a defect exists and the confidentiality is low, a second training data set consisting of a training image, a specification of an arbitrary pixel within the defect, and a defect area label is used to train the training target defect area determination model 40.

[0026] The training images used in the second training dataset do not need to be images of the defects to be inspected (including images that show only defect-free non-defective parts) and can be any common image (of something other than a defect). This makes it easy to obtain images and makes it possible to outsource training to an external party, thereby reducing the effort required for model generation.

[0027] In addition, the trained defect area determination model 50 can be generated not by self-training but by reusing a publicly available machine learning model or by fine-tuning the model based on the model.

[0028] In this way, labeling of defect types that require expertise and confidentiality and labeling of defect areas that do not require expertise and confidentiality are performed using different methods, and two types of training data, namely, defect type labels indicating defect types and defect area labels indicating defect areas, are used to generate two different machine learning models, the trained defect type determination model 30 and the trained defect area determination model 50. This makes it possible to create training data at lower cost and estimate the defect type of the detection target on a pixel-by-pixel basis using the trained defect type determination model 30 and the trained defect area determination model 50.

[0029] [Information Processing System] Next, an information processing system according to an embodiment of the present disclosure will be described. Fig. 2 shows an information processing system 10 according to an embodiment of the present invention.

[0030] As shown in FIG. 2, the information processing system 10 includes a training data DB (training data database) 60 and a pseudo segmentation model generating device 100.

[0031] The training data DB60 is a database that stores a first training data set for training the defect type determination model 20 to be trained and a second training data set for training the defect area determination model 40 to be trained, and provides these training data sets to the pseudo segmentation model generating device 100.

[0032] The first training data set consists of training images that show the defects to be inspected (including images that show only defect-free non-defective parts) and defect type labels that indicate the correct defect type (e.g., cracks, dents, powder, indentations, etc.) of the defects shown in the training images (e.g., scratches on the steel plate surface, uneven areas, abnormal roughness areas, etc.).

[0033] On the other hand, the second training data set is composed of training images that are common to the first training data set, a designation of any pixel contained in a defect that appears in the training image, and a defect area label that indicates the defect area that is the area occupied by the defect that appears in the training image.

[0034] The pseudo segmentation model generating device 100 uses a first training data set and a second training data set acquired in advance from the training data DB 60 to train a defect type determination model 20 to be trained and a defect area determination model 40 to be trained.

[0035] Specifically, as described in more detail below, the pseudo segmentation model generating device 100 adjusts parameters of the defect type determination model 20 to be trained using a first training data set, and adjusts parameters of the defect area determination model 40 to be trained using a second training data set. Upon completing training of the defect type determination model 20 to be trained and the defect area determination model 40 to be trained, the pseudo segmentation model generating device 100 acquires a trained defect type determination model 30 and a trained defect area determination model 50. Note that, in the example shown in FIG. 2, one pseudo segmentation model generating device 100 trains both the defect type determination model 20 to be trained and the defect area determination model 40 to be trained, but the present invention is not limited thereto, and the defect type determination model 20 to be trained and the defect area determination model 40 to be trained may be trained by a plurality of different devices.

[0036] When the training of the defect type determination model 20 to be trained and the defect area determination model 40 to be trained is completed, the pseudo segmentation model generating device 100 acquires image data of an object to be inspected as an input using the trained defect type determination model 30 and the trained defect area determination model 50, and outputs defect type data indicating a defect type to be detected in the object to be inspected and defect area data indicating a defect area. Specifically, as described in detail below, the pseudo segmentation model generating device 100 first inputs the acquired image data to the trained defect type determination model 30, and acquires defect type data indicating a defect type to be detected from the trained defect type determination model 30. The pseudo segmentation model generating device 100 also inputs judgment basis data indicating a pixel or image area, etc., that is the basis for the defect type determination by the trained defect type determination model 30, and image data to the trained defect area determination model 50, and acquires defect area data indicating a defect area to be detected from the trained defect area determination model 50. The pseudo segmentation model generating device 100 outputs the thus acquired defect type data and defect area data as detection results.

[0037] In the explanation of this embodiment, the explanation is mainly given using examples in which the trained defect identification judgment model 30 and the trained defect area judgment model 50 are trained by the system itself, but it is also possible to obtain either or both of the trained defect identification judgment model 30 and the trained defect area judgment model 50 in a trained state from outside rather than training them by the system itself.

[0038] The pseudo segmentation model generating device 100 is a device that generates a pseudo deep learning model that performs inference on an image by segmentation. Fig. 4 shows a functional configuration of the pseudo segmentation model generating device 100 according to an embodiment of the present invention.

[0039] 4, the pseudo segmentation model generating device 100 includes an image acquiring unit 110, a defect type inference unit 120, a judgment basis image generating unit 130, a representative pixel determining unit 140, an area determining unit 150, a pixel information assigning unit 160, and an inference result output unit 170. Each of the functional units of the image acquiring unit 110, the defect type inference unit 120, the judgment basis image generating unit 130, the representative pixel determining unit 140, the area determining unit 150, the pixel information assigning unit 160, and the inference result output unit 170 may be realized by causing the processor 102 to execute a computer program stored in the storage device 101 of the pseudo segmentation model generating device 100.

[0040] The image acquiring unit 110 is a functional unit that acquires a defect image, which is an image showing a defect. Specifically, the image acquiring unit 110 generates image data of an inspection object (e.g., a steel plate surface, etc.) captured by a camera or the like, by itself, or acquires the image data from the training data DB 60 or from outside the information processing system 10, and provides the image data to the other functional units 120 to 170.

[0041] For example, when detecting the defect type and defect area of ​​defects present on the surface of a steel plate, the image acquisition unit 110 acquires defect images, which are images of defects on the surface of the steel plate and defect-free good parts, obtained by imaging the surface of the steel plate using a camera installed, for example, on the top of a conveying device that conveys the steel plate, by generating the defect images itself or by obtaining them from the training data DB 60 or from outside the information processing system 10.

[0042] The image acquisition unit 110 transmits the acquired defect image to the other functional units 120 to 170 of the pseudo segmentation model generating device 100.

[0043] The defect type inference unit 120 is a functional unit that infers the defect type of a defect captured in a defect image by using a deep learning model that performs inference on an image by classification. Specifically, the defect type inference unit 120 inputs the defect image acquired from the image acquisition unit 110 to the trained defect type determination model 30, and acquires defect type data of the defect of the inspection object captured in the input defect image from the trained defect type determination model 30.

[0044] For example, the trained defect type determination model 30 is acquired by training the defect type determination model 20 to be trained using a first training data set acquired from the training data DB 60. Here, the first training data set is composed of training images and defect type labels indicating the correct defect types (e.g., cracks, dents, powder, and indentations) of defects captured in the training images. The training images are images of an object to be inspected, and the defect type labels are correct labels indicating the correct defect types of defects on the surface of the object to be inspected. Since the trained defect type determination model 30 is a classification model, such defect type labeling is performed on an image-by-image basis by an expert such as a defect inspector.

[0045] Fig. 5 shows a defect type determination model training process according to an embodiment of the present invention. As shown in Fig. 5, in response to input of each input training image of the first training data set to the defect type determination model 20 to be trained, parameters of the defect type determination model 20 to be trained are adjusted according to the error between the output result output from the defect type determination model 20 to be trained and the corresponding output defect type label, using the backpropagation method. When training of the defect type determination model 20 to be trained is thus completed, a trained defect type determination model 30 can be obtained.

[0046] The defect type inference unit 120 inputs the defect image acquired by the image acquisition unit 110 into the trained defect type determination model 30 acquired in this manner, and obtains by inference the defect type of the defect on the steel plate surface shown in the defect image.

[0047] The judgment basis image generating unit 130 is a functional unit that generates a judgment basis image indicating which pixel of a defect image is likely to affect the inference of the defect type. Here, the judgment basis image is a heat map-like image indicating the degree to which each pixel or partial image area of ​​a defect image influenced the judgment when the trained defect type judgment model 30 judged the defect type of the defect image. Such a judgment basis image can be acquired based on any known technology such as CAM (Class Activation Mapping) or GradCAM. Typically, such a judgment basis image can be acquired as a judgment basis map in a two-dimensional heat map format having the same size as the input defect image.

[0048] The determination basis image generating unit 130 acquires a determination basis image from the trained defect type determination model 30 , and provides the acquired determination basis image data to the representative pixel determining unit 140 .

[0049] The representative pixel determination unit 140 is a functional unit that determines a representative pixel that can be regarded as a pixel that represents the presence of a defect, based on the luminance value of the defect image and the luminance value of the judgment basis image. Specifically, the representative pixel determination unit 140 acquires the defect image to be inspected from the image acquisition unit 110, and acquires the judgment basis image from the judgment basis image generation unit 130, and then adds the luminance value of the defect image to the luminance value of the judgment basis image to generate composite image data of the image data to be inferred and the judgment basis image data. Here, the luminance value can be the RGB-based pixel value of each pixel of the image data.

[0050] The representative pixel determination unit 140 determines the pixel with the maximum luminance value among the luminance values ​​of each pixel of the composite image data as the representative pixel. Such a representative pixel can prevent the trained defective area determination model 50 from mistakenly determining the background part of the image data to be inferred as a defective area.

[0051] The synthetic image data is not necessarily generated by simply adding together the luminance value of the image data to be inferred and the luminance value of the judgment basis image data, and for example, the representative pixel determination unit 140 may weight either the luminance value of the image data to be inferred or the luminance value of the judgment basis image data, add the weighted luminance value, and generate synthetic image data of the image data to be inferred and the judgment basis image data.The representative pixel determination unit 140 may also determine the pixel with the maximum luminance value among the luminance values ​​of each pixel of the synthetic image data as the representative pixel.

[0052] The area determination unit 150 is a functional unit that inputs a defect image and determines an area in the defect image corresponding to a defect by specifying a representative pixel using an object area determination model that determines an area corresponding to an object including a specified pixel in the image by specifying an arbitrary pixel in the image. Specifically, the area determination unit 150 inputs the defect image to a trained defect area determination model 50 corresponding to the object area determination model, and by specifying the representative pixel determined by the representative pixel determination unit 140 in the defect image, outputs by inference the object in which the representative pixel is contained, i.e., the defect area that is the area occupied by the defect including the representative pixel.

[0053] For example, the trained defect area determination model 50 is obtained by training the defect area determination model 40 to be trained using the training data DB 60 or a second training data set acquired from outside the information processing system 10.

[0054] FIG. 6 illustrates a training process for a defective area determination model according to an embodiment of the present invention.

[0055] When the defect area determination model 40 to be trained is realized as a deep learning model such as a neural network, as shown in FIG. 6, the parameters of the defect area determination model 40 to be trained are adjusted according to the error backpropagation method according to the error between the output result output from the defect area determination model 40 to be trained in response to input of the training image of the second training data set and the representative pixel obtained from the judgment basis image, and the corresponding defect area label. When the training of the defect area determination model 40 to be trained is completed in this manner, the trained defect area determination model 40 can be acquired. Alternatively, the trained defect area determination model 50 can be generated based on any known machine learning model that accepts image data and its additional information as input, such as Meta's SAM (Segment Anything Model), and outputs a detection area of ​​a detection target in pixel units. Furthermore, the trained defect area determination model 50 may be obtained from an external device by acquiring a model that has already been trained in a similar manner.

[0056] Here, the trained defective area determination model 50 can receive a defective image and a representative pixel as input, and output defect area data indicating a defective area including the representative pixel in the image data to be inferred.

[0057] The image information assignment unit 160 is a functional unit that assigns information corresponding to a defect type to each pixel in an area that the area determination unit 150 has determined to correspond to a defect, and assigns information corresponding to the absence of a defect to each pixel outside of the areas that the area determination unit 150 has determined to correspond to a defect.

[0058] The image information assignment unit 160 labels each pixel contained in an image area of ​​the defective image that is determined to be a defective area with the defect type determined by the trained defect type determination model 30 as pixel information, and labels image areas that are not determined to be defective areas (i.e., are not defective areas) with pixel information such as “no defect” or “normal.”

[0059] In this way, for each pixel in a defect image, pixels that correspond to defects are linked to information on the defect type, and pixels that correspond to non-defective products are linked to information indicating that they are not defective, making it possible to obtain estimation results for all pixels in the defect image as if the defect type had been estimated using a segmentation model.

[0060] The inference result output unit 170 is a functional unit that outputs the defect type and the information assigned by the pixel information assigning unit 160 as an inference result of the defect image. For example, the inference result output unit 170 can display image data indicating a defect area labeled with a defect type by the pixel information assigning unit 160 as the inference result to a user on the display of the pseudo segmentation model generating device 100 or the like.

[0061] [Method of generating pseudo-segmentation model] Next, a pseudo segmentation model generating method according to an embodiment of the present invention will be described. The pseudo segmentation model generating method is executed by the pseudo segmentation model generating device 100 described above, and more specifically, may be realized by one or more processors 102 of the pseudo segmentation model generating device 100 executing one or more programs or instructions stored in one or more storage devices 101. Fig. 7 is a flowchart showing the pseudo segmentation model generating method according to an embodiment of the present invention.

[0062] As shown in FIG. 7, in step S101, the image acquisition unit 110 of the pseudo segmentation model generating device 100 acquires a defect image which is an image showing a defect.

[0063] In step S102, the defect type inference unit 120 of the pseudo segmentation model generating device 100 infers the defect type of the defect shown in the defect image using a deep learning model that infers an image by classification.

[0064] In step S103, the determination basis image generating unit 130 of the pseudo segmentation model generating device 100 generates a determination basis image indicating which pixels of the defect image are likely to affect the inference of the defect type.

[0065] In step S104, the representative pixel determination unit 140 of the pseudo segmentation model generating device 100 determines a representative pixel that can be regarded as a pixel that represents the presence of a defect, based on the luminance value of the defect image and the luminance value of the judgment basis image.

[0066] In step S105, the area determination unit 150 of the pseudo segmentation model generating device 100 inputs a defect image and determines an area in the defect image corresponding to a defect by specifying a representative pixel using an object area determination model that determines an area corresponding to an object in the image that includes a specified pixel by specifying an arbitrary pixel in the image.

[0067] In step S106, the pixel information assignment unit 160 of the pseudo segmentation model generating device 100 assigns information corresponding to a defect type to each pixel in the defect image in the area determined in step S105 to correspond to a defect, and assigns information corresponding to the absence of a defect to each pixel other than the area determined in step S105 to correspond to a defect.

[0068] In step S107, the inference result output unit 170 of the pseudo segmentation model generating device 100 outputs the defect type and the information assigned in step S106 as an inference result for the defect image.

[0069] [Example] Fig. 8 shows an image input to the trained defect area determination model 50 according to the embodiment of the present invention. For example, when a defect image 70 shows a defect area 71 corresponding to a defect as shown in Fig. 8A, and a judgment basis image 80 shows a heat map area 81 equal to or greater than a predetermined threshold value and a pixel (representative pixel) 82 with a maximum value in the heat map area 81 as shown in Fig. 8B, a composite image 90 in which the representative pixel 82 is superimposed on the defect area 71 may be input to the trained defect area determination model 50 as shown in Fig. 8C. Such a trained defect area determination model 50 exists, which is publicly available, such as SAM by Meta.

[0070] Such a composite image 90 may be constructed by calculating the sum of the luminance value of each pixel of image 70 and the luminance value of each pixel of judgment basis image 80, or may be constructed by calculating a weighted sum of the luminance value of each pixel of image 70 and the luminance value of each pixel of judgment basis image 80.

[0071] According to the above-described embodiment, two different machine learning models, namely, the defect type label indicating a defect type requiring expertise and the defect area label indicating a defect area not requiring expertise, are used to generate two different models, namely, the defect type determination model 30 and the defect area determination model 50. By combining these, training data can be created with little effort, and the presence or absence of a defect and the defect type can be known for each pixel in a defect image as if a segmentation model was used.

[0072] [Hardware configuration] Fig. 3 shows a hardware configuration of a pseudo segmentation model generating device according to an embodiment of the present invention. As shown in Fig. 3, the pseudo segmentation model generating device 100 may be realized by a computing device such as a server or a personal computer (PC), and may have, for example, the hardware configuration shown in Fig. 3. That is, the pseudo segmentation model generating device 100 has a storage device 101, a processor 102, an interface device 103, and a communication device 104 that are interconnected via a bus B.

[0073] The programs or instructions for realizing the various functions and processes described below in the pseudo segmentation model generating device 100 may be downloaded from any external device via a network or the like, or may be provided from a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a flash memory.

[0074] Storage device 101 may be implemented by random access memory, flash memory, a hard disk drive, etc., and stores installed programs or instructions as well as files, data, etc. used in the execution of the programs or instructions. Storage device 101 may also include a non-transitory storage medium.

[0075] The processor 102 may be realized by one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), processing circuitry, etc., which may be composed of one or more processor cores, and performs various functions and processes of the pseudo segmentation model generating device 100 described below in accordance with programs, instructions, data such as parameters necessary to execute the programs or instructions, etc. stored in the memory device 101.

[0076] The interface device 103 realizes an interface with a user of the pseudo segmentation model generating device 100. For example, the user operates a keyboard, a mouse, or the like to operate a GUI (Graphical User Interface) displayed on a display or a touch panel, and transmits and receives various information, data, instructions, and the like between the pseudo segmentation model generating device 100 and the interface device 103.

[0077] The communication device 104 is realized by various communication circuits that execute communication processes with external devices, the Internet, and communication networks such as a LAN (Local Area Network).

[0078] However, the above-described hardware configuration is merely an example, and the pseudo segmentation model generating device 100 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0079] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure described in the claims. [Explanation of symbols]

[0080] 10. Information Processing Systems 20 Training target defect type determination model 30 Training target defect region judgment model 40 Trained defect classification model 50 trained defect area determination models 60 Training Data DB 100 Pseudo-segmentation model generator 110 Image acquisition unit 120 Defect type inference unit 130 Judgment basis image generation unit 140 Representative pixel determination unit 150 Area determination part 160 Pixel information assignment section 170 Inference result output section

Claims

1. A pseudo segmentation model generation device that pseudo-generates a deep learning model that performs inference on an image by segmentation, an image acquisition unit for acquiring a defect image which is an image showing a defect; a defect type inference unit that infers a defect type of a defect shown in the defect image by using a deep learning model that infers an image by classification; a determination basis image generating unit that generates a determination basis image indicating which pixel of the defect image is likely to affect the inference of the defect type; a representative pixel determination unit that determines a representative pixel that can be regarded as a pixel that represents the presence of the defect based on a luminance value of the defect image and a luminance value of the judgment basis image; an area determination unit that, by inputting the defect image and specifying the representative pixel, determines an area in the defect image corresponding to the defect using an object area determination model that, by specifying an arbitrary pixel in the image, determines an area corresponding to the object including the specified pixel in the image; a pixel information assigning unit that assigns information corresponding to the defect type to each pixel in an area determined by the area determining unit to correspond to the defect, and assigns information corresponding to the absence of a defect to each pixel other than the area determined by the area determining unit to correspond to the defect; an inference result output unit that outputs the defect type and the information assigned by the pixel information assigning unit as an inference result for the defect image; A pseudo segmentation model generating device comprising:

2. The representative pixel determination unit generating a sum image by adding together the luminance values ​​of the judgment basis image and the luminance values ​​of the target image; The pseudo segmentation model generating device according to claim 1 , wherein the representative pixel is determined based on a pixel having the highest brightness in the summed image.

3. The representative pixel determination unit weighting one of the luminance values ​​of the judgment grounds image and the luminance values ​​of the target image, and adding them together to generate a sum image; The pseudo segmentation model generating device according to claim 1 , wherein the representative pixel is determined based on a pixel having the highest brightness in the summed image.

4. A pseudo segmentation model generation method for pseudo-generating a deep learning model that performs inference on an image by segmentation, comprising: An image acquisition step of acquiring a defect image which is an image showing a defect; a defect type inference step of inferring a defect type of a defect shown in the defect image by using a deep learning model that infers an image by classification; a determination basis image generating step of generating a determination basis image indicating which pixels of the defect image are likely to affect inference of the defect type; a representative pixel determination step of determining a representative pixel that can be regarded as a pixel that represents the presence of the defect based on a luminance value of the defect image and a luminance value of the judgment basis image; an area determination step of inputting the defect image and determining an area in the defect image corresponding to the defect by specifying the representative pixel, using an object area determination model that determines an area corresponding to the object including the specified pixel in the image by specifying an arbitrary pixel in the image; a pixel information assigning step of assigning information corresponding to the defect type to each pixel in an area determined in the area determining step to correspond to the defect, and assigning information corresponding to the absence of a defect to each pixel other than the area determined in the area determining step to correspond to the defect; an inference result output step of outputting the defect type and the information assigned in the pixel information assigning step as an inference result for the defect image; A pseudo segmentation model generating method comprising:

5. The representative pixel determination step includes: generating a sum image by adding together the luminance values ​​of the judgment basis image and the luminance values ​​of the target image; The pseudo segmentation model generating method according to claim 4 , wherein the representative pixel is determined based on a pixel having the highest brightness in the summed image.

6. The representative pixel determination step includes: weighting one of the luminance values ​​of the judgment grounds image and the luminance values ​​of the target image, and adding them together to generate a sum image; The pseudo segmentation model generating method according to claim 4 , wherein the representative pixel is determined based on a pixel having the highest brightness in the summed image.

Images