Training information creation assistance device, image classification device, and program

Abstract

This training information creation assistance device comprises: a category score calculation unit that uses a segmentation model for an image which is composed of a plurality of pixels to calculate, for each combination of a pixel and a category, a category score indicating the degree of confidence that the pixel belongs to the category; an inference image determination unit that calculates an inference image on the basis of the category score; an adjacent category calculation unit that receives input of a category of interest and calculates, on the basis of the inference image and the category of interest, an adjacent category which is likely to be adjacent to the category of interest; and a correction-required score calculation unit that determines a correction-required score for at least one pixel on the basis of the category of interest, the adjacent category, and the category score.

Description

Teacher information creation support device, image classification device and program

[0001] The present invention relates to a training information creation support device, an image classification device, and a program, and relates to those for reducing the cost of creating training data in an image classification system with a learning function, for example.

[0002] Creating a pixel-by-pixel image classification model using deep learning requires training information, but the cost of providing training information is high. As a countermeasure, interactive segmentation is available, where the model learns collaboratively with the user by having the user indicate the areas and content of corrections to be made to the model's inference results. One possible method for selecting the areas to be corrected is to select areas with a low probability of inference from among areas inferred as the focus category, but this does not select areas that are not inferred as the focus category, so false negatives are not selected and accuracy is not improved.

[0003] International Publication No. WO 2020 / 129235 International Publication No. WO 2020 / 189269

[0004] In Patent Document 1, the tendency for inference probability to decrease is utilized to select areas of the model with low inference probability as candidates for correction areas, but misclassified areas unrelated to the target category may be selected, making it difficult to improve the accuracy of the target category.

[0005] In Patent Document 2, among the sections classified as the category of interest, sections with a low probability of inference are selected as candidates for correction sections, but sections that are not inferred to be in the category of interest are not selected, so false negative sections are not selected and accuracy is not improved.

[0006] For this reason, the present invention aims to present correction-required parts, including false negative parts, which are expected to improve the accuracy of the target category.

[0007] An example of a teacher information creation support device according to the present invention has: a category score calculation unit that uses a segmentation model to calculate, for an image consisting of a plurality of pixels, a category score indicating the degree of certainty that the pixel belongs to that category for each combination of pixel and category; an inference image determination unit that calculates an inference image based on the category score; an adjacent category calculation unit that accepts input of a focus category and calculates adjacent categories that are likely to be adjacent to the focus category based on the inference image and the focus category; and a correction-required score calculation unit that determines a correction-required score for at least one pixel based on the focus category, the adjacent categories, and the category score.

[0008] An example of an image classification device according to the present invention comprises: an image category inference unit that uses a segmentation model to calculate, for each combination of pixel and category, a category score indicating the degree of certainty that the pixel belongs to that category for an image consisting of multiple pixels, and calculates an inferred image based on the category score; and a teacher information creation GUI having a focus category setting unit, wherein the focus category setting unit accepts input of a focus category, the image classification device calculates areas requiring correction based on the focus category and the category score, and the teacher information creation GUI outputs the areas requiring correction and accepts the assignment of teacher information via a teacher information assignment unit.

[0009] An example of a program according to the present invention causes a computer to function as the teacher information creation support device described above.

[0010] An example of a program according to the present invention causes a computer to function as the image classification device described above.

[0011] According to the present invention, it is possible to present correction-required parts that will lead to improved accuracy against false negatives in a target category.

[0012] An example of a functional configuration of a teacher information creation support device according to a first embodiment of the present invention. An example of processing by an adjacent category calculation unit. An example of a processing result by the adjacent category calculation unit. An example of a correction-required score calculation unit. Another example of a method for calculating uncertainty. Another example of a method for calculating a correction-required score. Yet another example of a method for calculating a correction-required score. An example of a functional configuration of an image classification device according to a second embodiment of the present invention. An example of a functional configuration of an image classification device according to a third embodiment of the present invention. Another example of a functional configuration of an image classification device according to the third embodiment of the present invention. An example of a teacher information creation GUI. Another example of a teacher information creation GUI.

[0013]

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. [Embodiment 1] Fig. 1 shows an example of the functional configuration of a teacher information creation support device according to embodiment 1 of the present invention. First, to outline the functional configuration of Fig. 1, a teacher information creation support device 100 receives an input image D1, a segmentation model D2, and a focus category D4, and includes a category score calculation unit 101, an inference image determination unit 102, an adjacent category calculation unit 103, and a correction-required score calculation unit 104.

[0014] For an input image D1 consisting of a plurality of pixels, the category score calculation unit 101 uses a segmentation model D2 to calculate, for each combination of a pixel and a category, a category score D3 indicating the degree of certainty that the pixel belongs to that category. The category score D3 is a vector having values ​​indicating the likelihood of each category being identified for each pixel in the input image D1.

[0015] The inference image determination unit 102 calculates an inference image D5 based on the category score D3. The inference image D5 is an image in which a single category is identified for each pixel through inference. Specifically, the inference image D5 can be calculated by determining the category with the maximum value of the category score D3 for each pixel as the identification category.

[0016] The adjacent category calculation unit 103 receives the input of the focused category D4 and calculates an adjacent category D6 based on the inference image D5 and the focused category D4. An adjacent category means a category that is likely to be adjacent to the focused category (i.e., a category with a high adjacent likelihood), and a specific example will be described later.

[0017] The correction-needed score calculation unit 104 determines a correction-needed score D7 based on the focused category D4, the adjacent category D6, and the category score D3. The correction-needed score is calculated, for example, for a portion that is difficult for the segmentation model D2 to identify in relation to the focused category, and a specific example will be described later.

[0018] The teacher information creation support device 100 can be configured using, for example, a computer. The computer has a hardware configuration as a known computer, and includes, for example, a calculation means and a storage means. The calculation means includes, for example, a processor, and the storage means includes, for example, a storage medium such as a semiconductor memory device or a magnetic disk device. Some or all of the storage medium may be non-transitory storage media.

[0019] The computer may also include input / output means, which include, for example, input devices such as a keyboard and a mouse, output devices such as a display and a printer, and communication devices such as a network interface.

[0020] The storage means may store a program, and the processor executes the program, causing the computer to function as the teacher information creation support device 100 according to this embodiment or the device according to another embodiment.

[0021] Each component in FIG. 1 will now be described in detail.

[0022] The input image D1 is an image for which a correction score is to be calculated, and for at least one pixel (pixel or group of pixels) in this image, a correction score is calculated that is expected to improve the accuracy of the segmentation model D2.

[0023] The segmentation model D2 is a trained model that has undergone prior machine learning and has the ability to discriminate images. However, the prior training may be performed using a dataset that has similar characteristics to the input image D1, or may be performed using a dataset that is unrelated to the input image D1.

[0024] The segmentation model D2 calculates features for each pixel of the input image D1 and calculates a confidence level for classifying each category from the features. The confidence level may be a value indicating an inference probability normalized for each pixel using a softmax function, or it may be a logit function before being normalized using softmax.

[0025] The attention category D4 indicates a category that should be prioritized for accuracy improvement among the categories identified by the segmentation model, such as a category indicating the structure of the measurement target in the case of a semiconductor measurement image.

[0026] 2 shows an example of the processing of the adjacent category calculation unit 103. First, in processing step S201, the adjacent category calculation unit 103 extracts adjacent pixel groups.

[0027] First, the adjacent category calculation unit 103 extracts pixels in the inference image D5 that belong to the target category. Then, for each pixel inferred to belong to the target category, the unit 103 extracts pixels adjacent to that pixel as adjacent pixels (however, if an adjacent pixel also belongs to the target category, that pixel is not extracted as an adjacent pixel).

[0028] Next, in processing step S202, the adjacent category calculation unit 103 determines adjacent categories. Here, the adjacent category calculation unit 103 calculates the frequency of categories in the inference image D5 for the adjacent pixel groups, and determines the category with the highest frequency as the adjacent category to the focused category D4. In this way, a category that is likely to be adjacent to the focused category D4 is determined. Here, if there are multiple frequently occurring categories, it is also possible to determine multiple adjacent categories. In this way, it is possible to calculate the categories that are likely to be adjacent to the focused category D4.

[0029] In this embodiment, a category that is likely to be adjacent to the target category D4 refers to a category to which pixels adjacent to pixels belonging to the target category D4 in the inference image D5 belong with a high frequency. The definition of "high frequency" can be determined appropriately by those skilled in the art, but it refers to, for example, a case where the frequency exceeds a predetermined threshold.

[0030] An example of the processing result of the adjacent category calculation unit is shown in Fig. 3. Fig. 3 shows an example in which an inferred image in which each pixel is classified into category 1, 2, or 3 is input, and the target category is category 1.

[0031] First, in S201, a group of pixels adjacent to a location inferred to be category 1 is calculated as adjacent pixel group 300. Next, in S202, the frequency of the category to which adjacent pixel group 300 belongs is calculated, and category 2, which is the category with the highest frequency, is determined as the adjacent category.

[0032] For example, if a category indicates the structure of the measurement target in a semiconductor measurement image in which similar structures are repeated within the image, the number of categories adjacent to a specific category is limited, so the above method can be used to select the most frequent category as the adjacent category.

[0033] Furthermore, for example, if the white lines in a road image are set as the target category, the asphalt portion of the road is frequently registered as an adjacent category using the above method.

[0034] 4 shows an example of the correction need score calculation unit 104. First, in processing step S401, the correction need score calculation unit 104 extracts a category score related to the attention category D4 and a category score related to the adjacent category D6 for each pixel inferred to belong to the attention category D4 in the inference image D5 and each pixel inferred to belong to the adjacent category D6 in the inference image D5.

[0035] Then, in processing step S402, the correction score calculation unit 104 calculates uncertainty based on the extracted category scores as a value indicating the degree to which the segmentation model is insufficiently trained (e.g., includes insufficiently trained features). For example, the correction score calculation unit 104 calculates uncertainty as the smallness of the category scores for specified categories (the focus category D4 and the adjacent category D6) among the extracted category scores.

[0036] For example, if the category score is an inference probability, the uncertainty value is 1 - inference probability (the value obtained by subtracting the inference probability from 1). Generally, in a trained segmentation model, the inference probability of a pixel with sufficiently trained features is large, and the inference probability of a pixel with insufficiently trained features is small, so a location where 1 - inference probability is large (a location with large uncertainty) can be said to be a location where learning is insufficient.

[0037] By inputting the target category and adjacent categories as designated categories, the uncertainty of the part inferred to be the target category and the uncertainty of the part inferred to be the adjacent category are obtained. Then, based on these uncertainties, parts requiring correction with high uncertainty are identified in processing step S403, and a correction score for the identified parts requiring correction is calculated.

[0038] The correction-required portion may be, for example, a portion where the uncertainty is equal to or greater than a predetermined threshold. A specific method for calculating the correction-required score can be appropriately designed by a person skilled in the art, and may be, for example, the sum of the uncertainty related to the focus category and the uncertainty related to the adjacent category.

[0039] Areas with large uncertainty values ​​indicate areas that are difficult for the segmentation model to identify, and by prioritizing learning, it is expected that the segmentation model's identification accuracy will improve. This makes it possible to calculate the score requiring correction from candidates that include areas inferred from adjacent categories that may produce false negatives, in addition to areas inferred from the focus category that may produce false positives.

[0040] Another example of a method for calculating uncertainty is shown in Fig. 5. As shown in Fig. 5, a method for calculating uncertainty related to adjacent categories may be to use the magnitude of the category score for the target category as uncertainty at an inferred location belonging to an adjacent category.

[0041] 5, in processing step S501, the correction score calculation unit 104 extracts a group of category scores for each pixel inferred to belong to an adjacent category in the inference image. Then, in processing step S502, uncertainty (second uncertainty) is calculated based on the category score related to the target category from the extracted group of category scores. The second uncertainty can be, for example, the magnitude of the category score related to the target category.

[0042] Note that for a portion inferred to belong to the attention category, the same calculation method as in Fig. 4 can be used. That is, the correction score calculation unit 104 calculates uncertainty (first uncertainty) for each pixel inferred to belong to the attention category in the inference image, based on the category score related to the attention category.

[0043] In this case, the correction-needed score calculation unit 104 calculates the correction-needed score based on the first uncertainty and the second uncertainty.

[0044] In a false negative section, the input image may contain features of the target category. In other words, in such a section, even if the category with the highest category score is an adjacent category, the category score of the target category may also be high. Therefore, it is possible to calculate the correction score by limiting it to sections within the adjacent category that are more likely to be false negatives for the target category.

[0045] In the first embodiment, the above-described method makes it possible to calculate a score requiring correction that is expected to improve accuracy with respect to false positives as well as false negatives.

[0046] Another example of a method for calculating the correction need score is shown in Fig. 6. In the example of Fig. 6, in processing step S601, the correction need score calculation unit 104 smoothes the correction need score D7 for each pixel within a predetermined range including the pixel, and calculates the part where the value of the smoothed correction need score D8 is large as the part that needs to be corrected.

[0047] When a user makes a correction, it is more efficient to correct a large range at once rather than correcting only one isolated pixel, so by calculating the location with a high correction score within a predetermined range, it becomes possible to perform an efficient correction. A specific method for determining the predetermined range and a specific calculation for smoothing within the predetermined range can be designed appropriately by a person skilled in the art, but for example, two-dimensional convolution processing with a Gaussian function or simple averaging processing may be used.

[0048] 7 shows yet another example of a method for calculating a correction-needed score. In the example of FIG. 7, the correction-needed score calculation unit 104 identifies a portion with a high correction-needed score as a correction-needed portion D9 in processing step S701. The definition of a portion with a high correction-needed score can be determined appropriately by those skilled in the art, but it may be, for example, a portion with a correction-needed score equal to or greater than a threshold, or an area with a high deviation value of the correction-needed score.

[0049] By determining the parts requiring correction in this manner, the user can efficiently grasp the parts with high correction-requiring scores.

[0050] [Example 2] Example 2 proposes a GUI for creating teacher information using the correction-required score calculated in Example 1. Explanation of parts common to Example 1 may be omitted.

[0051] 8 shows an example of the functional configuration of an image classification device according to Example 2 of the present invention. The image classification device 800 also functions as a teacher information creation support device similar to Example 1. For example, the image classification device 800 may further have a configuration similar to that of the teacher information creation support device 100 in FIG.

[0052] First, an overview of the functional configuration of FIG. 8 will be given. The image classification device 800 receives an input image D801 and a segmentation model D802, and includes an image category inference unit 801 and a training information creation GUI 802.

[0053] For an input image D801 consisting of a plurality of pixels, the image category inference unit 801 uses a segmentation model D802 to calculate, for each combination of a pixel and a category, a category score D804 indicating the degree of certainty that the pixel belongs to that category, and calculates an inference image D803 based on the category score D804. The inference image D803 and the category score D804 are calculated in the same way as the inference image D5 and the category score D3 in FIG. 1, respectively.

[0054] The training information creation GUI 802 is a GUI intended for the user to create training information D805 for each pixel using an input image D801, an inference image D803, and a category score D804. The training information creation GUI 802 has a function of simultaneously displaying the input image D801 and the inference image D803, making it easier to find errors in the inference results, and of machine-detecting areas that need correction from the category score D804, making it easier to select the areas to be corrected.

[0055] 11 shows an example of a teacher information creation GUI 802. The teacher information creation GUI includes an input image display unit that displays an input image D801, an inference image display unit that displays an inference image D803, a teacher information assignment unit that has a function of assigning teacher information creation, a focus category setting unit that accepts input of a focus category, a correction-required part display unit that outputs (for example, displays) parts that need correction calculated based on the teacher information creation support device 100 shown in FIG. 1 from the input focus category, input image, and category score, an uncertainty display unit that similarly displays the calculated uncertainty, and a model learning start unit that accepts an operation to start learning the segmentation model.

[0056] The uncertainty display section and the correction-required section are displayed as reference information when adding teaching information, and may be omitted.

[0057] The image processing device may also include an automatic correction part setting unit that accepts an operation for automatically calculating correction parts.

[0058] The image classification device 800 can calculate the parts to be corrected based on the focused category and the category score in the same manner as the training information creation support device 100 of the first embodiment, for example.

[0059] The teacher information creation GUI 802 accepts the assignment of teacher information via a teacher information assignment unit. The teacher information assignment unit can set a category for each pixel using a click on a computer or a pen tablet. In the example of FIG. 11, the input image is displayed as the background, and the assigned teacher information is displayed superimposed on the foreground of the input image. However, the input image does not have to be displayed as the background. Furthermore, the GUI 802 may have a function for assigning the same category to multiple correction-requiring portions with a single operation.

[0060] The uncertainty display unit displays information to be used as a reference when assigning teacher information. The image classification device 800 calculates uncertainty, which indicates the degree to which the learning of the segmentation model is insufficient, in the same manner as the teacher information creation support device 100 of the first embodiment, and the uncertainty display unit displays this uncertainty. In the example of Figure 11, white areas indicate areas with high uncertainty.

[0061] Areas that are not selected as areas requiring correction but have a relatively high degree of uncertainty are often areas where the inference is incorrect, and even if they are not incorrect, the segmentation model is often insufficiently trained. For this reason, the user can refer to the display in the uncertainty display section and add training information to areas with a relatively high degree of uncertainty, thereby improving the accuracy of the segmentation model. However, if areas requiring correction are selected automatically, the uncertainty display section does not need to be displayed.

[0062] The uncertainty display unit may also separately display the uncertainty within the target category (first uncertainty) and the uncertainty within the adjacent category (second uncertainty) shown in Fig. 4. Furthermore, if there are multiple adjacent categories, the second uncertainty may be displayed in multiple regions.

[0063] The correction-required portion display section displays portions with high uncertainty as portions requiring correction. In the example of FIG. 11, the portions requiring correction are displayed as white areas. The user can refer to the portions requiring correction when adding teaching information. However, if portions requiring correction are selected automatically, the correction-required portion display section does not need to be displayed.

[0064] 12 shows another example of the training information creation GUI 802. In this example, the GUI in FIG. 11 is applied to detecting defects in structures during the semiconductor manufacturing process. As in FIG. 11, three categories are displayed, and in particular, the categories "Structure 1" and "Structure 2" represent appropriate structures of semiconductor structures, and the category "Defect" represents a structural defect.

[0065] Such a GUI allows a user to efficiently create teaching information that can be used to detect defects in semiconductor structures.

[0066] [Example 3] Example 3 proposes an image classification device for a segmentation model in the middle of learning in interactive segmentation, where learning of a segmentation model and user annotation are performed reciprocally. Explanation of parts common to Examples 1 and 2 may be omitted.

[0067] 9 shows an example of the functional configuration of an image classification device according to Example 3 of the present invention. The image classification device 900 first trains a segmentation model D802 using training information D805 calculated by the method shown in Example 2 and an input image.

[0068] A model learning unit 901 learns a segmentation model D802 using an input image D801 and training information D805.

[0069] The segmentation model D802 is trained in advance at the input stage. The advance training may be performed using a small amount of training information for the input image D801, or may be performed using a public dataset or the like.

[0070] In the image classification device 900, the model learning unit 901 learns a segmentation model D802, and then again creates training information D805 through processing by the image category inference unit 801, etc., and trains the segmentation model D802 using the training information D805. By repeating this processing until there are no more misclassifications in the inference image D803, it is possible to obtain a segmentation model D802 with high classification accuracy.

[0071] 10 shows another example of the functional configuration of an image classification device according to Example 3 of the present invention. The image classification device 1000 calculates uncertainty in the same manner as in Examples 1 and 2, and performs enhancement learning using the uncertainty for each pixel.

[0072] Areas with high uncertainty are areas where the segmentation model has not been sufficiently trained. The model training unit 1001 trains the segmentation model based on the uncertainty information D1001 so that the loss of pixels with high uncertainty is smaller than the loss of pixels with low uncertainty.

[0073] This type of learning is expected to improve accuracy. For example, by increasing the loss per pixel in proportion to the uncertainty and training the model according to that loss, it becomes possible to more strongly learn the features of areas with high uncertainty.

[0074] DESCRIPTION OF SYMBOLS 100... Teacher information creation support device 101... Category score calculation unit 102... Inference image determination unit 103... Adjacent category calculation unit 104... Correction required score calculation unit 300... Adjacent pixel group 800... Image classification device 801... Image category inference unit 802... Teacher information creation GUI 900... Image classification device 901... Model learning unit 1000... Image classification device 1001... Model learning unit D1... Input image D2... Segmentation model D3... Category score D4... Category of interest D5... Inference image D6... Adjacent category D7... Score requiring correction D8... Score requiring correction D9... Part requiring correction D801... Input image D802... Segmentation model D803... Inference image D804... Category score D805... Teacher information D1001... Uncertainty information S201... Processing step S202... Processing step S401...processing step S402...processing step S403...processing step S501...processing step S502...processing step S601...processing step S701...processing step

Claims

1. A teacher information creation support device comprising: a category score calculation unit that uses a segmentation model to calculate, for an image consisting of multiple pixels, a category score indicating the degree of certainty that each combination of pixel and category belongs to that category; an inference image determination unit that calculates an inference image based on the category score; an adjacent category calculation unit that accepts input of a focus category and calculates adjacent categories that are likely to be adjacent to the focus category based on the inference image and the focus category; and a correction required score calculation unit that determines a correction required score for at least one pixel based on the focus category, the adjacent categories, and the category score.

2. A teacher information creation support device as described in claim 1, wherein the correction score calculation unit extracts a category score related to the focus category and a category score related to the adjacent category for each pixel inferred to belong to the focus category in the inference image and each pixel inferred to belong to the adjacent category in the inference image, calculates uncertainty as a value indicating the degree to which learning of the segmentation model is insufficient based on each extracted category score, and calculates the correction score based on the uncertainty.

3. A teacher information creation support device as described in claim 1, characterized in that the correction score calculation unit calculates a first uncertainty for each pixel inferred to belong to the focus category in the inference image based on a category score related to the focus category, calculates a second uncertainty for each pixel inferred to belong to the adjacent category in the inference image based on a category score related to the focus category, and calculates the correction score based on the first uncertainty and the second uncertainty.

4. A teacher information creation support device as claimed in claim 1, characterized in that the correction-needed score calculation unit identifies the parts having a large correction-needed score as the parts requiring correction.

5. A teacher information creation support device as described in claim 1, characterized in that the correction score calculation unit smoothes the correction score for each pixel within a predetermined range including that pixel, and calculates the areas where the smoothed value is large as the areas requiring correction.

6. An image classification device comprising: an image category inference unit that uses a segmentation model to calculate, for each combination of pixel and category, a category score indicating the degree of certainty that the pixel belongs to that category, and calculates an inferred image based on the category score; and a teacher information creation GUI having a focus category setting unit, wherein the focus category setting unit accepts input of a focus category, the image classification device calculates areas needing correction based on the focus category and the category score, and the teacher information creation GUI outputs the areas needing correction and accepts the assignment of teacher information via a teacher information assignment unit.

7. An image classification device as described in claim 6, characterized in that said image classification device has a model learning unit that uses said teacher information to learn a segmentation model.

8. An image classification device as described in claim 7, characterized in that the image classification device calculates uncertainty indicating the degree to which the learning of the segmentation model is insufficient, and the model learning unit trains the segmentation model so that the loss of pixels with large uncertainty is smaller than the loss of pixels with small uncertainty.

9. An image classification device according to claim 6, characterized in that the image classification device calculates an uncertainty representing the degree to which the learning of the segmentation model is insufficient, and displays the uncertainty.

10. An image classification device as described in claim 6, characterized in that the image classification device calculates an adjacent category that is likely to be adjacent to the attention category based on the inference image and the attention category, calculates a first uncertainty for each pixel inferred to belong to the attention category in the inference image based on a category score related to the attention category, and calculates a second uncertainty for each pixel inferred to belong to the adjacent category in the inference image based on the category score related to the attention category, and the teacher information creation GUI displays the first uncertainty and the second uncertainty.

11. An image classification device according to claim 8, characterized in that the teaching information creation GUI displays areas where the uncertainty is large as areas requiring correction.

12. A program for causing a computer to function as the teacher information creation support device according to claim 1.

13. A program for causing a computer to function as the image classification device according to claim 6.

Images