Computer system and method for supporting correction of classification results

Abstract

To automatically correct results of semantic segmentation.SOLUTION: A computer system: receives an input of an input image; modifies a prediction model; performs prediction on the input image using the modified prediction model; executes processing of storing prediction results a plurality of times; calculates a probability distribution of each class for each pixel of the input image based on a plurality of prediction results; assigns an uncertainty level to results of classification of the pixel by performing statistical processing on the probability distribution of each class; determines, for each pixel, a class to which the pixel belongs based on the probability distribution of each class; generates a base image representing the class to which the pixel belongs; generates an uncertainty map representing the uncertainty level of each pixel of the input image; selects a pixel of the input image whose classification results needs to be corrected based on the uncertainty map; and corrects the classification results of the selected pixel of the input image using a plurality of prediction results.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present disclosure relates to techniques for correcting semantic segmentation results. [Background technology]

[0002] Artificial intelligence (AI) is increasingly being used in a variety of fields to classify images into multiple regions. For example, in the field of materials science, AI is used to extract features from tissue images and model the relationship with physical properties, which is expected to improve the efficiency of new material development. Semantic segmentation (SS) is widely used as an AI machine learning model to analyze tissue features.

[0003] It is important to evaluate the uncertainty of prediction results, which are the output of machine learning models, along with the accuracy of the prediction. A method described in Patent Document 1 is known as a method for quantifying uncertainty. Patent Document 1 describes the repeated application of Monte Carlo dropout to evaluate uncertainty. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-200677 Summary of the Invention [Problem to be solved by the invention]

[0005] The uncertainty map in Patent Document 1, which visualizes the uncertainty index, can limit the areas where the prediction is incorrect, but the task of manually correcting the prediction results requires a lot of work.

[0006] The present disclosure relates to techniques for modifying semantic segmentation results based on uncertainty in the results. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, a computer system includes a processor, a storage device connected to the processor, and an input / output interface connected to the processor, and holds definition information of a prediction model that makes predictions for class classification that determines a class to which a pixel of an image belongs from among a plurality of classes, the prediction model is a machine learning model that calculates the probability that each pixel of the image belongs to each class, the processor accepts input of an input image, transforms the prediction model, makes the prediction for the input image using the transformed prediction model, and executes a process of saving the prediction results a plurality of times, and calculates the probability distribution of each class for each pixel of the input image based on the plurality of prediction results. a distribution of the probability distributions of the classes for each pixel of the input image, and assigning an uncertainty level to the result of the class classification of the pixel of the input image by statistically processing the probability distribution of each class for each pixel of the input image; determining, for each pixel of the input image, the class to which the pixel belongs based on the probability distribution of each class; generating a base image representing the class to which each pixel of the input image belongs; generating an uncertainty map representing the uncertainty level for each pixel of the input image; selecting, based on the uncertainty map, pixels of the input image whose class classification results require correction; and correcting the base image by correcting the result of the class classification of the selected pixels of the input image using a plurality of the prediction results. [Effects of the Invention]

[0008] According to one aspect of the present disclosure, an image representing a classification result can be automatically corrected based on the uncertainty of the classification result (semantic segmentation). Problems, configurations, and advantages other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a functional configuration of a computer system according to a first embodiment. [Figure 2]FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer that configures the computer system of the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating the cooperation of functions in the learning phase of the computer system of the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a data structure of a teacher data DB according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a data structure of a model DB according to the first embodiment. [Figure 6A] FIG. 10 is a diagram illustrating an example of a data structure of a prediction result DB according to the first embodiment. [Figure 6B] FIG. 10 is a diagram illustrating an example of a data structure of a prediction result DB according to the first embodiment. [Figure 7A] FIG. 2 is a diagram illustrating an example of a data structure of an uncertainty assessment DB according to the first embodiment. [Figure 7B] FIG. 2 is a diagram illustrating an example of a data structure of an uncertainty assessment DB according to the first embodiment. [Figure 8A] FIG. 1 illustrates the characteristics of prediction in semantic segmentation. [Figure 8B] FIG. 1 illustrates the characteristics of prediction in semantic segmentation. [Figure 8C] FIG. 1 illustrates the characteristics of prediction in semantic segmentation. [Figure 9] 10 is a flowchart illustrating an example of a prediction process that the computer system of the first embodiment executes in a learning phase. [Figure 10] FIG. 10 is a diagram illustrating an example of a probability distribution of classes calculated by the computer system of the first embodiment. [Figure 11A] 10 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system of the first embodiment. [Figure 11B] 10 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system of the first embodiment. [Figure 11C] 10 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system of the first embodiment. [Figure 12A]FIG. 10 is a diagram illustrating an example of the correlation between the uncertainty index and the accuracy rate in the first embodiment. [Figure 12B] FIG. 10 is a diagram illustrating an example of the correlation between the uncertainty index and the accuracy rate in the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a method for calculating an evaluation index in the first embodiment. [Figure 14A] FIG. 10 is a diagram illustrating an example of boundary area processing executed by the computer system of the first embodiment. [Figure 14B] FIG. 10 is a diagram illustrating an example of boundary area processing executed by the computer system of the first embodiment. [Figure 14C] FIG. 10 is a diagram illustrating an example of boundary area processing executed by the computer system of the first embodiment. [Figure 15A] FIG. 2 is a diagram illustrating an example of a data structure of data generated by the computer system of the first embodiment. [Figure 15B] FIG. 2 is a diagram illustrating an example of a data structure of data generated by the computer system of the first embodiment. [Figure 16] 10 is a flowchart illustrating an example of an uncertainty map generation process executed by the computer system of the first embodiment. [Figure 17] FIG. 10 is a diagram illustrating cooperation of functions in a prediction phase of the computer system of the first embodiment. [Figure 18] 10 is a flowchart illustrating an example of a prediction process that is executed in a prediction phase by the computer system of the first embodiment. [Figure 19A] 10 is a flowchart illustrating an example of a class classification process executed by the computer system of the first embodiment. [Figure 19B] 10 is a flowchart illustrating an example of a class classification process executed by the computer system of the first embodiment. [Figure 19C] 10 is a flowchart illustrating an example of a class classification process executed by the computer system of the first embodiment. [Figure 20A] FIG. 2 is a diagram showing an example of an image output in the classification process of the first embodiment. [Figure 20B] FIG. 2 is a diagram showing an example of an image output in the classification process of the first embodiment. [Figure 20C] FIG. 2 is a diagram showing an example of an image output in the classification process of the first embodiment. [Figure 20D] FIG. 2 is a diagram showing an example of an image output in the classification process of the first embodiment. [Figure 20E] FIG. 2 is a diagram showing an example of an image output in the classification process of the first embodiment. [Figure 20F] FIG. 2 is a diagram showing an example of an image output in the classification process of the first embodiment. [Figure 21] FIG. 2 is a diagram illustrating an example of a GUI presented by the computer system of the first embodiment. [Figure 22] FIG. 2 is a diagram illustrating an example of a GUI presented by the computer system of the first embodiment. [Figure 23] FIG. 2 is a diagram illustrating an example of a GUI presented by the computer system of the first embodiment. [Figure 24] 10 is a flowchart illustrating an example of a class classification process executed in a prediction phase by a computer system according to a second embodiment. [Figure 25] FIG. 10 is a diagram showing an example of an uncertainty map according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.

[0011] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant explanations will be omitted.

[0012] In this specification, the terms "first," "second," "third," etc. are used to identify components and do not necessarily limit the number or order. [Example]

[0013] Fig. 1 is a diagram illustrating an example of a functional configuration of a computer system according to Example 1. Fig. 2 is a diagram illustrating an example of a hardware configuration of a computer that constitutes the computer system according to Example 1.

[0014] The computer system 100 is configured from a computer 200 as shown in Fig. 2. The computer 200 includes a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204. The computer 200 may also include input devices such as a mouse and a keyboard, and an output device such as a display.

[0015] The computer system 100 includes a learning unit 101, a prediction unit 102, an accuracy rate calculation unit 103, an uncertainty index calculation unit 104, an uncertainty level determination unit 105, and an input / output unit 106. The computer system 100 also stores a teacher data DB 110, a model DB 111, a prediction result DB 112, and an uncertainty assessment DB 113.

[0016] The training data DB 110 is a database that stores training data consisting of input data to be input to the machine learning model and correct answer data for prediction. In this embodiment, the input data is an image, and the correct answer data is an array of classes to which pixels belong. The model DB 111 is a database that stores information that defines the machine learning model. The prediction result DB 112 is a database that stores the results of predictions made using the machine learning model. The uncertainty assessment DB 113 is a database that stores information on assessments regarding the uncertainty of the results of class classification based on the prediction results.

[0017] It should be noted that with regard to each functional unit of the computer system 100, multiple functional units may be combined into one functional unit, or one functional unit may be divided into multiple functional units for each function. Furthermore, the databases held by the computer system 100 may be integrated into one database, or one database may be divided into multiple databases depending on the purpose of data management.

[0018] First, the processing in the learning phase will be described. Fig. 3 is a diagram showing the cooperation of functions in the learning phase of the computer system 100 of the first embodiment.

[0019] The learning unit 101 acquires the training data stored in the training data DB 110 as training data, and executes a learning process of a machine learning model for realizing semantic segmentation.

[0020] The machine learning model is a model that calculates the probability that a pixel belongs to each class, and is, for example, a convolutional neural network (CNN), SegNet, U-Net, or PSPNet. The machine learning method is, for example, deep learning. The present invention is not limited to the type of machine learning model. The present invention is not limited to the learning method of the machine learning model. The learning unit 101 stores the learning result in the model DB 111.

[0021] The prediction unit 102 acquires training data stored in the training data DB 110 as verification data and performs prediction using a machine learning model defined by information stored in the model DB 111. In this embodiment, an array of probabilities that pixels of the image belong to each class is output as a prediction result. The prediction unit 102 stores the prediction result in the prediction result DB 112.

[0022] In this embodiment, the prediction unit 102 uses a Monte Carlo dropout method to generate a machine learning model (hereinafter referred to as an MCD model) in which some of the nodes constituting the machine learning model are dropped out, and predicts validation data using the MCD model. Predictions are performed multiple times for one validation data set using different MCD models.

[0023] The accuracy rate calculation unit 103 uses the prediction result and the correct answer data included in the verification data to calculate the accuracy rate for the input data, the accuracy rate for each class, etc. The accuracy rate calculation unit 103 stores the processing results in the prediction result DB 112.

[0024] The uncertainty index calculation unit 104 calculates a plurality of uncertainty indexes using the prediction results. The uncertainty index calculation unit 104 stores the processing results in the uncertainty assessment DB 113. The calculated uncertainty indexes will be described later.

[0025] The uncertainty level determination unit 105 assigns an uncertainty level to the result of class classification of each pixel based on the prediction result for one verification data set using multiple uncertainty indices. The uncertainty level determination unit 105 stores the processing result in the uncertainty assessment DB 113.

[0026] The input / output unit 106 receives input to the computer system 100 and outputs various data.

[0027] FIG. 4 is a diagram illustrating an example of a data structure of the training data DB 110 according to the first embodiment.

[0028] The training data DB 110 stores a table 400 for managing training data. The table 400 stores entries including a data ID 401, input data 402, and correct answer data 403. One entry corresponds to one training data.

[0029] Data ID 401 is a field that stores the ID of the training data. Input data 402 is a field that stores input data to be input to the machine learning model. In this embodiment, an array of pixel values ​​of pixels that make up an image is stored as input data. Correct answer data 403 is a field that stores correct answer data that indicates the correct answer of the prediction. In this embodiment, an array of values ​​that indicates the class to which the pixel belongs is stored as correct answer data. Figure 4 shows a two-dimensional array that indicates whether the pixel belongs to the first class or the second class.

[0030] The data format of the teacher data DB 110 is not limited to the table format.

[0031] FIG. 5 is a diagram illustrating an example of a data structure of the model DB 111 according to the first embodiment.

[0032] The model DB 111 stores a table 500 that stores information that defines a machine learning model. One table 500 exists for one machine learning model. The table 500 stores entries including a layer ID 501, a layer type 502, and parameters 503. One entry exists for one layer.

[0033] Layer ID 501 is a field that stores the ID of a layer of a machine learning model. Layer type 502 is a field that stores the type of layer. Layer types include an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. Parameter 503 is a field that stores the parameters of the layer corresponding to layer ID 501. The layer parameters are, for example, weights.

[0034] The data format of the model DB 111 is not limited to the table format.

[0035] 6A and 6B are diagrams illustrating an example of the data structure of the prediction result DB 112 according to the first embodiment.

[0036] The prediction result DB 112 stores a table 600 and a table 610 .

[0037] The table 600 is a table for managing prediction results. The table 600 stores entries including a prediction result ID 601, a data ID 602, and a prediction result 603. One entry exists for one prediction result.

[0038] Prediction result ID 601 is a field that stores the ID of the prediction result. Data ID 602 ​​is a field that stores the ID of the input data on which prediction was performed. In the learning phase, the ID of the verification data is stored, and in the prediction phase, the ID of the input data itself is stored. Prediction result 603 is a field that stores the prediction result. In this embodiment, an array of the probability that a pixel belongs to each class is stored as the prediction result. In FIG. 6A, an array of the probability that a pixel belongs to each of the first and second classes is stored. The sum of the probabilities of each class of a pixel is 1.

[0039] Table 610 is a table for managing the accuracy rate of prediction results. Table 610 stores entries including a prediction result ID 611 and an accuracy rate 612. There is one entry for one prediction result.

[0040] The prediction result ID 611 is the same field as the prediction result ID 601. The accuracy rate 612 is a field that stores the accuracy rate. In this embodiment, the accuracy rate of each pixel, the accuracy rate of the entire input data, the accuracy rate of each class, etc. are stored. The accuracy rate of the entire input data can be calculated, for example, as the average accuracy rate of each pixel. The accuracy rate of a class can be calculated, for example, as the average accuracy rate of pixels assigned the same class in the correct data.

[0041] The information managed in the tables 600 and 610 may be managed in a data format other than the table format.

[0042] 7A and 7B are diagrams illustrating an example of the data structure of the uncertainty assessment DB 113 according to the first embodiment.

[0043] The uncertainty assessment DB 113 stores a table 700 and a table 710 .

[0044] Table 700 is a table for managing uncertainty indices for evaluating the uncertainty of the results of pixel classification based on predictions for input data. Table 700 stores entries including a data ID 701, a first uncertainty index 702, and a second uncertainty index 703. There is one entry for one piece of input data.

[0045] The data ID 701 is the same field as the data ID 602. The first uncertainty index 702 and the second uncertainty index 703 are fields for storing uncertainty indexes. Details of each uncertainty index will be described later.

[0046] Table 710 is a table for managing the uncertainty level assigned to input data. Table 710 stores entries including a data ID 711 and an uncertainty level 712. There is one entry for one piece of input data.

[0047] The data ID 711 is the same field as the data ID 602. The uncertainty level 712 is a field that stores the uncertainty level. In this embodiment, an array of the uncertainty levels of the pixels is stored.

[0048] The information managed in the tables 700 and 710 may be managed in a data format other than the table format.

[0049] Here, the characteristics of prediction in semantic segmentation will be described. Figures 8A, 8B, and 8C are diagrams for explaining the characteristics of prediction in semantic segmentation.

[0050] 8A, 8B, and 8C show the distribution of the probability that a pixel belongs to a class. Here, semantic segmentation for classification into three classes will be explained as an example.

[0051] As shown in FIG. 8A, if the variance of the probability distribution of a certain class is small and the most frequent value is close to 1, and the variance of the probability distribution of another class is small and the most frequent value is close to 0, the prediction result is sufficiently reliable.

[0052] As shown in FIG. 8B, if the probability distributions of each class overlap and the most frequent probability of each class is small (for example, less than 0.5), the prediction results are unreliable.

[0053] As shown in Figure 8C, when the probability distributions of two classes overlap, the mode of the probability distributions of the two classes is large (for example, greater than 0.5), and the mode of the probability distribution of the other class is approximately 0, it is highly likely that this is the boundary between the two classes, and it is not possible to uniquely determine which class it belongs to.

[0054] In this embodiment, the results of pixel classification are classified into the above three levels using two uncertainty indices calculated from the probability distribution.

[0055] Fig. 9 is a flowchart illustrating an example of a prediction process executed in the learning phase by the computer system 100 according to the first embodiment. Fig. 10 is a diagram illustrating an example of a probability distribution of classes calculated by the computer system 100 according to the first embodiment.

[0056] The prediction unit 102 acquires definition information of the machine learning model from the model DB 111 (step S101).

[0057] The prediction unit 102 starts a loop process of the verification data (step S102). Specifically, the prediction unit 102 acquires one piece of verification data from the training data DB 110.

[0058] The prediction unit 102 starts a prediction loop process (step S103). Specifically, the prediction unit 102 generates an MCD model using the Monte Carlo dropout method. In this loop process, prediction is performed a predetermined number of times. Note that, since it is only necessary to obtain the probability distribution of the classes, the model may be modified using a method other than the Monte Carlo dropout method.

[0059] The prediction unit 102 executes prediction of the verification data using the MCD model (step S104). The prediction unit 102 stores the prediction result in the table 600 of the prediction result DB 112.

[0060] The prediction unit 102 instructs the accuracy rate calculation unit 103 to calculate the accuracy rate. The instruction includes the ID of the prediction result.

[0061] The accuracy rate calculation unit 103 compares the prediction result with the correct data included in the verification data and calculates the accuracy rate (step S105). The accuracy rate calculation unit 103 stores the calculation result in table 610 of the prediction result DB 112. Thereafter, the accuracy rate calculation unit 103 notifies the prediction unit 102 of the completion of the processing.

[0062] The prediction unit 102 predicts whether the number of times the prediction has been executed is smaller than a predetermined number (step S106). If the number of times the prediction has been executed is greater than the predetermined number, the prediction unit 102 returns to step S103.

[0063] If the number of times the prediction has been performed is equal to or greater than the predetermined number, the prediction unit 102 instructs the uncertainty index calculation unit 104 to calculate an uncertainty index. The instruction includes the ID of the verification data.

[0064] The uncertainty index calculation unit 104 calculates the probability distribution of each class for each pixel (step S107). Specifically, the uncertainty index calculation unit 104 obtains the prediction result of the verification data from the table 600, and calculates the probability distribution of each class for each pixel as shown in FIG.

[0065] The uncertainty index calculation unit 104 calculates a first uncertainty index for each pixel based on the probability distribution of each class (step S108). Specifically, the following process is executed.

[0066] (S108-1) The uncertainty index calculation unit 104 selects a pixel.

[0067] (S108-2) The uncertainty index calculation unit 104 calculates the mode of the probability distribution of each class for the selected pixel.

[0068] (S108-3) The uncertainty index calculation unit 104 calculates the variance of the probability distribution of the class with the largest mode as a first uncertainty index.

[0069] (S108-4) The uncertainty index calculation unit 104 determines whether or not the processing has been completed for all pixels. If the processing has not been completed for all pixels, the uncertainty index calculation unit 104 returns to S108-1. If the processing has been completed for all pixels, the uncertainty index calculation unit 104 stores the calculation results in table 700 of the uncertainty assessment DB 113.

[0070] The uncertainty index calculation unit 104 calculates a second uncertainty index for each pixel based on the probability distribution of each class (step S109). Specifically, the following process is executed.

[0071] (S109-1) The uncertainty index calculation unit 104 selects a pixel.

[0072] (S109-2) The uncertainty index calculation unit 104 calculates the mode of the probability distribution of each class for the selected pixel.

[0073] (S109-3) The uncertainty index calculation unit 104 ranks the classes in descending order of the mode.

[0074] (S109-4) The uncertainty index calculation unit 104 calculates the sum of the most frequent values ​​of the classes ranked third and below as a second uncertainty index.

[0075] (S109-5) The uncertainty index calculation unit 104 determines whether or not the processing has been completed for all pixels. If the processing has not been completed for all pixels, the uncertainty index calculation unit 104 returns to S109-1. If the processing has been completed for all pixels, the uncertainty index calculation unit 104 stores the calculation results in table 700 of the uncertainty assessment DB 113.

[0076] After calculating the second uncertainty index, the uncertainty index calculation unit 104 notifies the prediction unit 102 of the completion of the processing. The prediction unit 102 determines whether the processing has been completed for all the verification data (step S110).

[0077] If the processing has not been completed for all the verification data, the prediction unit 102 returns to step S 102. If the processing has been completed for all the verification data, the prediction unit 102 ends the prediction processing.

[0078] 11A, 11B, and 11C are flowcharts illustrating an example of uncertainty level determination processing executed by the computer system 100 of the first embodiment. FIGS. 12A and 12B are diagrams illustrating an example of the correlation between the uncertainty index and the accuracy rate in the first embodiment. FIG. 13 is a diagram illustrating an example of a method for calculating an evaluation index in the first embodiment. FIGS. 14A, 14B, and 14C are diagrams illustrating an example of boundary region processing executed by the computer system 100 of the first embodiment. FIGS. 15A and 15B are diagrams illustrating an example of the data structure of data generated by the computer system 100 of the first embodiment.

[0079] The uncertainty level determination unit 105 executes the following process for one piece of verification data.

[0080] The uncertainty level determination unit 105 analyzes the correlation between the first uncertainty index and the accuracy rate for the verification data (step S201). Specifically, the following process is executed.

[0081] (S201-1) The uncertainty level determination unit 105 obtains the accuracy rate of each pixel from the table 610, and also obtains the first uncertainty index of each pixel from the table 700.

[0082] (S201-2) The uncertainty level determination unit 105 analyzes the correlation between the accuracy rate and the first uncertainty index. For example, the analysis result shown in FIG. 12A is obtained. Graph 1201 shows the correlation between the accuracy rate and the first uncertainty index. The dotted bar graph shows the number of correct pixels for the first uncertainty index. The white bar graph shows the number of incorrect pixels for the first uncertainty index.

[0083] The uncertainty level determination unit 105 calculates a threshold value Th1 based on the correlation between the first uncertainty index and the accuracy rate (step S202).

[0084] Specifically, the uncertainty level determination unit 105 uses correlation to calculate an uncertainty index at which the accuracy rate becomes a value designated by the user as a threshold value Th1. In Fig. 12A, the threshold value Th1 is 0.15.

[0085] The uncertainty level determination unit 105 analyzes the correlation between the second uncertainty index and the accuracy rate for the verification data (step S203). Specifically, the following process is executed.

[0086] (S203-1) The uncertainty level determination unit 105 obtains the accuracy rate of each pixel from the table 610, and also obtains the second uncertainty index of each pixel from the table 700.

[0087] (S203-2) The uncertainty level determination unit 105 analyzes the correlation between the accuracy rate and the second uncertainty index. For example, the analysis result shown in FIG. 12B is obtained. Graph 1202 shows the correlation between the accuracy rate and the second uncertainty index. The dotted bar graph shows the number of correct pixels for the second uncertainty index. The white bar graph shows the number of incorrect pixels for the second uncertainty index.

[0088] The uncertainty level determination unit 105 calculates a threshold value Th2 based on the correlation between the second uncertainty index and the accuracy rate (step S204). Specifically, the following process is executed.

[0089] (S204-1) The uncertainty level determination unit 105 calculates the change in the accuracy rate relative to the change in the second uncertainty index (the decreasing gradient of the accuracy rate).

[0090] (S204-2) The uncertainty level determination unit 105 calculates the maximum value of the second uncertainty index in the range where the gradient of decline in the accuracy rate is greatest as the threshold value Th2. In Fig. 12B, the gradient of decline in the range where the second uncertainty index is 0.1 and 0.2 is greatest, so the threshold value Th2 is 0.2.

[0091] As shown in FIG. 13, the uncertainty level determination unit 105 calculates, for each pixel, the difference (ΔC) between the mode (first value) of the class with the largest mode and the mode (second value) of the class with the second largest mode (step S205).

[0092] The uncertainty level determination unit 105 extracts, for each pixel, a combination of the class with the largest mode value and the class with the second largest mode value as the type of boundary (step S206).

[0093] For example, in the case of the probability distribution shown in Fig. 13, the boundary type extracted is class A / class B. Note that the boundary may be extracted only from pixels having a probability distribution relationship as shown in Fig. 8C.

[0094] The uncertainty level determination unit 105 starts a loop process of the boundary types (step S207). Specifically, the uncertainty level determination unit 105 selects one boundary type.

[0095] The uncertainty level determination unit 105 sets an initial value to the variable k (step S208). The initial value is, for example, 0.1. The initial value can be set arbitrarily.

[0096] The uncertainty level determination unit 105 starts loop processing of pixels (step S209). Specifically, the uncertainty level determination unit 105 selects one pixel of the verification data.

[0097] The uncertainty level determination unit 105 determines whether ΔC of the pixel is smaller than the variable k (step S210). If ΔC of each pixel in the prediction result is equal to or larger than the variable k, the uncertainty level determination unit 105 proceeds to step S212.

[0098] If ΔC of each pixel in the prediction result is smaller than the variable k, the uncertainty level determination unit 105 changes the pixel output to the class with the second most frequent value (step S211). After that, the uncertainty level determination unit 105 proceeds to step S212.

[0099] Normally, the output pixel class is determined to be the class with the largest mode, but for pixels where ΔC is smaller than the variable k, the mode is changed to the second most frequent class. For example, Figure 14A shows an image (classification result) generated by outputting the class with the largest mode. As shown in Figure 14B, when ΔC of a pixel in the black area (boundary) is smaller than 0.1, the output is changed to the class with the second most frequent mode, resulting in an image like that shown in Figure 14C. The dotted line indicates the boundary before the change.

[0100] In step S212, the uncertainty level determination unit 105 determines whether or not the process has been completed for all pixels (step S212). If the process has not been completed for all pixels, the uncertainty level determination unit 105 returns to step S209.

[0101] When the processing is completed for all pixels, the uncertainty level determination unit 105 calculates the accuracy rate by comparing the image generated from the output of each pixel with the image generated based on the correct data (step S213).

[0102] The uncertainty level determination unit 105 records the processing result in a table 1500 as shown in Fig. 15A. The table 1500 is stored in the uncertainty assessment DB 113. The table 1500 stores entries including a boundary type 1501, a variable k 1502, and an accuracy rate 1503. In step S212, the uncertainty level determination unit 105 adds an entry to the table 1500, sets the selected boundary type to the boundary type 1501 of the entry, sets the current value of the variable k to the variable k 1502, and sets the calculated accuracy rate to the accuracy rate 1503.

[0103] The uncertainty level determination unit 105 determines whether the variable k is smaller than 0.9 (step S213). 0.9 is the maximum value of the variable k. The maximum value can be set arbitrarily.

[0104] If the variable k is smaller than 0.9, the uncertainty level determination unit 105 updates the variable k (step S215), and then returns to step S209. For example, 0.1 is added to the value of the variable k.

[0105] If the variable k is equal to or greater than 0.9, the uncertainty level determining unit 105 refers to the table 1510 and determines the value of the variable k with the highest accuracy rate as the boundary type threshold value Th3 (step S213).

[0106] At this time, the uncertainty level determination unit 105 records the processing result in a table 1510 as shown in Fig. 15B. The table 1510 is stored in the uncertainty assessment DB 113. The table 1510 stores entries including a boundary type 1511, a ratio 1512, and a threshold value 1513. In step S213, the uncertainty level determination unit 105 adds an entry to the table 1510, sets the boundary type 1511 of the entry to the selected boundary type, and sets the threshold value 1513 to the value of the variable k with the highest accuracy rate.

[0107] The uncertainty level determination unit 105 determines whether or not the process has been completed for all boundary types (step S217). If the process has not been completed for all boundary types, the uncertainty level determination unit 105 returns to step S207.

[0108] When the process is completed for all boundary types, the uncertainty level determining unit 105 calculates the proportion of each boundary type (step S218).

[0109] Specifically, the uncertainty level determination unit 105 counts the number of pixels that output the second most frequent class for each boundary type. The uncertainty level determination unit 105 divides the number of pixels for each boundary type by the total number of pixels for each boundary type. The uncertainty level determination unit 105 sets the calculation result as the ratio 1512 for each entry in the table 1510.

[0110] The uncertainty level determination unit 105 starts loop processing of pixels (step S219). Specifically, the uncertainty level determination unit 105 selects one pixel of the verification data.

[0111] The uncertainty level determination unit 105 determines whether the first uncertainty index of the pixel is smaller than a threshold value Th1 (step S220).

[0112] If the first uncertainty index of the pixel is smaller than the threshold Th1, the uncertainty level determination unit 105 assigns an uncertainty level of "0" (step S221). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in the table 710. Thereafter, the uncertainty level determination unit 105 proceeds to step S228. The uncertainty level of "0" indicates that the result of classification is reliable.

[0113] If the first uncertainty index of the pixel is equal to or greater than the threshold value Th1, the uncertainty level determining unit 105 predicts whether the second uncertainty index of the pixel is greater than a threshold value Th2 (step S222).

[0114] If the second uncertainty index of the pixel is greater than the threshold value Th2, the uncertainty level determination unit 105 assigns an uncertainty level of "1" (step S223). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in the table 710. Thereafter, the uncertainty level determination unit 105 proceeds to step S228. The uncertainty level "1" indicates that the classification is unreliable.

[0115] If the second uncertainty index of the pixel is equal to or smaller than the threshold value Th2, the uncertainty level determining unit 105 calculates ΔC based on the probability distribution of each class of the pixel (step S224).

[0116] The uncertainty level determination unit 105 identifies the type of boundary based on the probability distribution of each class of pixels (step S225). Specifically, the uncertainty level determination unit 105 identifies the pair of the class with the largest mode and the class with the second largest mode as the type of boundary.

[0117] The uncertainty level determining unit 105 determines whether ΔC is smaller than the threshold value Th3 of the identified boundary type (step S226).

[0118] If ΔC is smaller than the threshold value Th3, the uncertainty level determination unit 105 assigns an uncertainty level of "2" (step S227). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in the table 710. Thereafter, the uncertainty level determination unit 105 proceeds to step S228. The uncertainty level of "2" indicates that the class cannot be determined because the pixel is located on the boundary between areas of different classes.

[0119] In step S228, the uncertainty level determination unit 105 determines whether or not the process has been completed for all pixels (step S228). If the process has not been completed for all pixels, the uncertainty level determination unit 105 returns to step S219. If the process has been completed for all pixels, the uncertainty level determination unit 105 ends the uncertainty level determination process.

[0120] The uncertainty level determining unit 105 may set, for example, "-1" to pixels to which no uncertainty level has been assigned.

[0121] FIG. 16 is a flowchart illustrating an example of the uncertainty map generation process executed by the computer system 100 of the first embodiment.

[0122] The input / output unit 106 receives the verification data and the selection of the uncertainty level from the user (step S301).

[0123] The input / output unit 106 generates an uncertainty map that displays the uncertainty level of the selected verification data (step S302). Specifically, the following process is executed.

[0124] (S302-1) The input / output unit 106 refers to the table 710 of the uncertainty assessment DB 113 and searches for an entry corresponding to the selected verification data.

[0125] (S302-2) The input / output unit 106 identifies pixels to which a specified uncertainty level has been assigned based on the searched entries.

[0126] (S302-3) The input / output unit 106 generates an image of the same size as the input data (image) included in the verification data, and sets RGB values ​​corresponding to the uncertainty level to the identified pixels of the image. This image is the uncertainty map. Note that the uncertainty map may be generated by superimposing the generated image on an image of class classification based on the input data or the prediction result.

[0127] The input / output unit 106 outputs the uncertainty map (step S303), and then the uncertainty map generation process ends.

[0128] It is possible to assign an uncertainty level to the classification results for input data contained in the training data, and present the uncertainty level as an image (uncertainty map). Users can refer to the uncertainty map to correct the correct data. By performing machine learning using the corrected training data, it is possible to improve the accuracy of predictions by the machine learning model and reduce uncertainty.

[0129] The following describes the processing in the prediction phase of the embodiment 1. Fig. 17 is a diagram showing the cooperation of functions in the prediction phase of the computer system 100 of the embodiment 1.

[0130] The prediction unit 102 acquires input data to be predicted, and performs prediction using a machine learning model defined by information stored in the model DB 111. The prediction unit 102 stores the prediction result in the prediction result DB 112.

[0131] The uncertainty index calculation unit 104 calculates a plurality of uncertainty indexes using the prediction results. The uncertainty index calculation unit 104 stores the processing results in the uncertainty assessment DB 113.

[0132] The uncertainty level determination unit 105 assigns an uncertainty level to the result of class classification of each pixel based on the prediction result of the input data to be predicted, using multiple uncertainty indices. The uncertainty level determination unit 105 stores the processing result in the uncertainty assessment DB 113.

[0133] The input / output unit 106 accepts input to the computer system 100 and outputs various data of the computer system 100. For example, the input / output unit 106 accepts input extraction conditions, searches the prediction result DB 112 for prediction results that satisfy the extraction conditions, and displays an image using the prediction results. At this time, the input / output unit 106 displays the input data and an uncertainty map. The input / output unit 106 also accepts an image selected by the user, and based on the image, modifies an image that represents the result of class classification of each pixel based on the initial prediction result, and displays the modified image.

[0134] FIG. 18 is a flowchart illustrating an example of the prediction process that the computer system 100 of the first embodiment executes in the prediction phase.

[0135] The prediction unit 102 acquires definition information of the machine learning model from the model DB 111 (step S401).

[0136] The prediction unit 102 starts a loop process of prediction (step S402). The process of step S402 is the same as the process of step S103.

[0137] The prediction unit 102 uses the MCD model to perform prediction of the input data (image) to be predicted (step S403). The process of step S403 is the same as the process of step S104.

[0138] The prediction unit 102 determines whether the number of times the prediction has been performed is smaller than a predetermined number (step S404). The process of step S404 is the same as the process of step S106.

[0139] If the number of times the prediction has been executed is equal to or greater than the predetermined number, the prediction unit 102 instructs the uncertainty index calculation unit 104 to calculate an uncertainty index. The instruction includes the ID of the input data.

[0140] The uncertainty index calculation unit 104 calculates the probability distribution of each class for each pixel (step S405). The process of step S405 is the same as the process of step S107.

[0141] The uncertainty index calculation unit 104 calculates a first uncertainty index for each pixel based on the probability distribution of each class (step S406). The process of step S406 is the same as the process of step S108.

[0142] The uncertainty index calculation unit 104 calculates a second uncertainty index for each pixel based on the probability distribution of each class (step S407). The process of step S407 is the same as the process of step S109.

[0143] In the uncertainty level determination process of the prediction phase, the processes from step S201 to step S218 are not executed. The threshold value used to assign the uncertainty level is the threshold value determined in the learning phase. For example, the average value of the threshold values ​​of each validation data is used.

[0144] 19A, 19B, and 19C are flowcharts illustrating an example of the classification process executed by the computer system 100 of the embodiment 1. Fig. 20A, 20B, 20C, 20D, 20E, and 20F are diagrams illustrating an example of an image output in the classification process of the embodiment 1.

[0145] The input / output unit 106 receives input data and a selection of an uncertainty level from the user (step S501).

[0146] The input / output unit 106 acquires the probability distribution of the class of each pixel of the input data, and also acquires information on the uncertainty level of the input data from the table 710 of the uncertainty assessment DB 113 (step S502).

[0147] The input / output unit 106 generates a base image that displays the results of class classification based on the probability distribution of the class for each pixel of the input data (step S503). Specifically, the input / output unit 106 generates an image (base image) that outputs the class with the largest mode in the probability distribution for each pixel. For example, a base image such as that shown in FIG. 20A is generated. FIG. 20A is an example of an image classified into three classes. The shaded area is a group of pixels classified into class C, the black area is a group of pixels classified into class B, and the white area is a group of pixels classified into class A.

[0148] The input / output unit 106 generates an uncertainty map that displays the uncertainty level of the selected input data (step S504). The processing in step S504 is the same as the processing in step S302. For example, uncertainty maps such as those shown in Figures 20B and 20C are generated. Figure 20B is an uncertainty map when uncertainty level "1" is selected, and Figure 20C is an uncertainty map when uncertainty level "2" is selected. In the following description, a region of pixels to which uncertainty level "1" is assigned will be referred to as a doubtful region, and a region of pixels to which uncertainty level "2" is assigned will be referred to as a boundary region.

[0149] The uncertainty level does not have to be input, in which case the input / output unit 106 generates an uncertainty map that allows each uncertainty level to be distinguished.

[0150] The input / output unit 106 outputs the base image and the uncertainty map (step S505), after which the input / output unit 106 transitions to a waiting state (step S506) and waits for an input from the user.

[0151] The user refers to the base image and the uncertainty map to determine whether or not the base image needs to be modified. If the user determines that the suspicious region needs to be modified, the user inputs a first modification request to the computer system 100. If the user determines that the boundary region needs to be modified, the user inputs a second modification request to the computer system 100. If the user determines that no modification is required, the user inputs an end request to the computer system 100.

[0152] When the input / output unit 106 receives an input from the user, it determines whether or not the input is a first modification request (step S507).

[0153] If the input is not a first revision request, the input / output unit 106 determines whether the input is a second revision request (step S508).

[0154] If the input is not a second modification request, the input / output unit 106 ends the classification process.

[0155] If the input is a first correction request, the correction process for the suspicious region is started. First, the input / output unit 106 presents a screen for inputting correction conditions and accepts the correction conditions from the user (step S509). Here, the correction conditions are the number of the target suspicious region and candidate images.

[0156] The input / output unit 106 searches the prediction result DB 112 for a prediction result based on the correction condition, and displays a candidate image based on the prediction result (step S510). Specifically, the following process is executed.

[0157] (S510-1) The input / output unit 106 identifies the class with the largest area in the suspicious region of the base image, that is, the class with the largest number of pixels.

[0158] (S510-2) The input / output unit 106 refers to the table 600 of the prediction result DB 112 and counts the number of pixels in the suspicious region that have the highest probability value of the identified class. The input / output unit 106 acquires a specified number of prediction results in descending order of the number of pixels.

[0159] (S510-3) The input / output unit 106 generates and displays a candidate image based on the obtained prediction result. Specifically, the input / output unit 106 generates the candidate image so as to output the class with the largest probability value for each pixel. For example, a candidate image such as that shown in FIG. 20D is generated.

[0160] The user selects a candidate image that is considered to be correct from among the candidate images, and the input / output unit 106 receives the candidate image selection from the user (step S511).

[0161] The input / output unit 106 modifies and displays the base image based on the selected candidate image (step S512). Then, the input / output unit 106 returns to step S506. Specifically, the input / output unit 106 changes the class to which the pixel included in the suspicious region of the base image belongs to the class with the largest probability value of the suspicious region of the selected candidate image. For example, if the candidate image shown in FIG. 20D is selected for a base image such as that shown in FIG. 20A, the base image is modified as shown in FIG. 20E.

[0162] If the input is a second modification request, the modification process of the boundary area is started. First, the input / output unit 106 identifies the type of boundary in the base image and calculates the proportion of each boundary type (step S513). The calculation process of the boundary proportion is the same as the process of step S218.

[0163] The input / output unit 106 acquires the threshold value Th3 for each boundary type from the table 1510 of the uncertainty assessment DB 113 (step S514). After that, the input / output unit 106 presents an interface for selecting a target boundary region.

[0164] When the input / output unit 106 receives a selection of a target boundary area from the user (step S515), it modifies and displays the display of the boundary area (step S516). After that, the input / output unit 106 transitions to a waiting state (step S517). In step S516, the following process is executed.

[0165] (S516-1) The input / output unit 106 identifies the boundary type of the selected boundary area. The type of boundary area can be identified based on the probability distribution of the classes of pixels included in the boundary area.

[0166] (S516-2) The input / output unit 106 selects one pixel included in the selected boundary region.

[0167] (S516-3) The input / output unit 106 determines whether ΔC of the selected pixel is smaller than the threshold value Th3 of the identified boundary type. If ΔC of the selected pixel is smaller than the threshold value Th3 of the identified boundary type, the input / output unit 106 changes the output of the pixel to the class with the second most frequent value. Note that the class may also be changed to the third or subsequent class with the most frequent value.

[0168] (S516-4) The input / output unit 106 determines whether or not the processing has been completed for all pixels included in the selected boundary region. If the processing has not been completed, the input / output unit 106 returns to S516-2.

[0169] (S516-5) When the processing is completed, the input / output unit 106 modifies the base image based on the processing results of each pixel included in the selected boundary region, and generates a candidate image. For example, the base image is modified as shown in FIG. 20F.

[0170] The user refers to the corrected base image and determines whether or not the boundary region needs to be corrected. If it is determined that the boundary region needs to be corrected, the user inputs a correction instruction including a new threshold value Th3. If it is determined that the boundary region does not need to be corrected, the user inputs a completion instruction.

[0171] When input / output unit 106 receives an input from the user, it determines whether or not the input is a correction instruction (step S518).

[0172] If the received input is a correction instruction, the input / output unit 106 returns to step S516. If the received input is a completion instruction, the input / output unit 106 returns to step S506.

[0173] 21, 22, and 23 are diagrams showing examples of GUIs presented by the computer system 100 of the first embodiment.

[0174] The GUI 2100 is presented by the input / output unit 106. The GUI 2100 includes a setting area 2101 and a display area 2102.

[0175] The setting area 2101 is an area for making various settings necessary for processing. The setting area 2101 includes input fields 2110 and 2111, a selection field 2112, and an operation button 2114.

[0176] The input field 2110 is a field for inputting the machine learning model to be used. For example, the name of the table 400 that defines the machine learning model is input. The input field 2111 is a field for inputting input data. For example, the name of the input data is input.

[0177] The selection field 2113 is a field for selecting the uncertainty level to be displayed on the uncertainty map. The selection field 2113 displays check boxes for selecting the uncertainty level to be displayed.

[0178] The operation button 2114 is an operation button for issuing an instruction to execute the classification process.

[0179] The display area 2102 is an area for displaying the processing results and includes a display column 2120 and operation buttons 2121, 2122, and 2123.

[0180] Display field 2120 displays input data 2131, a base image 2132, and an uncertainty map 2133. The uncertainty map 2133 is displayed as an image in which RGB values ​​are set according to the uncertainty level.

[0181] The operation button 2121 is an operation button for inputting a first modification request, the operation button 2122 is an operation button for inputting a second modification request, and the operation button 2123 is an operation button for inputting an end request.

[0182] When the operation button 2121 is operated on the GUI 2100, the input / output unit 106 presents a GUI 2200. The GUI 2200 includes a setting area 2201 and a display area 2202.

[0183] The setting area 2201 is an area for setting correction conditions, and includes a selection field 2210, an input field 2211, operation buttons 2212, and an extraction condition input field 2213.

[0184] The selection field 2210 is a field for selecting a target suspicious region. The input field 2211 is a field for inputting the number of candidate images. The operation button 2212 is an operation button for instructing the generation of candidate images. The extraction condition input field 2213 is a field for inputting extraction conditions for candidate images.

[0185] The display area 2202 is an area for displaying the correction results of the candidate image and the base image, and includes display fields 2220 and 2222, and operation buttons 2221 and 2223.

[0186] The display field 2220 is a field for displaying candidate images. The operation button 2221 is an operation button for instructing correction of the base image. The user selects a candidate image from the display field 2220 and operates the operation button 2221.

[0187] The display field 2222 is a field for displaying the correction results of the base image. The operation button 2223 is an operation button for instructing completion of the correction of the questionable region.

[0188] When the operation button 2122 is operated on the GUI 2100, the input / output unit 106 presents a GUI 2300. The GUI 2300 includes a setting area 2301 and a display area 2302.

[0189] The setting area 2301 includes a selection field 2310 , an input field 2311 , and an operation button 2312 .

[0190] The selection field 2310 is a field for selecting a target boundary region. The input field 2311 is a field for inputting a threshold value Th3. The input field 2311 may be left blank. When the input field 2311 is blank, the threshold value Th3 obtained from the uncertainty assessment DB 113 is used. The operation button 2312 is an operation button for instructing correction of the base image.

[0191] The display area 2302 is an area for displaying the correction result of the base image, and includes a display field 2320 and operation buttons 2321.

[0192] The display field 2320 is a field for displaying the correction result of the base image. The operation button 2321 is an operation button for instructing completion of the correction of the boundary area.

[0193] The computer system 100 of the first embodiment assigns an uncertainty level to the class classification result of each pixel based on the prediction result of the input data and displays an uncertainty map. By referring to the uncertainty map, the user can determine whether or not the base image needs to be modified. Furthermore, when the computer system 100 receives a correction instruction from the user, it can automatically correct the base image using the uncertainty map. This reduces the effort required to correct the class classification result based on the prediction result.

[0194] In processing based on the mode of the probability distribution, the median and mean of the probability distribution may be substituted.

[0195] The classification process may also be performed in the learning phase, which can reduce the effort required to correct the correct answer data in the training data. [Example]

[0196] In the second embodiment, the computer system 100 automatically modifies the base image based on the uncertainty level. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0197] The configuration of the computer system 100 of the second embodiment is the same as that of the first embodiment. The data structures of various databases of the second embodiment are the same as those of the first embodiment. The processing executed by the computer system 100 of the second embodiment in the learning phase is the same as that of the first embodiment.

[0198] The processing executed by the computer system 100 in the prediction phase in the second embodiment is partially different from that in the first embodiment. Specifically, the class classification processing is partially different. The other processing is the same as that in the first embodiment.

[0199] FIG. 24 is a flowchart illustrating an example of the class classification process executed in the prediction phase by the computer system 100 according to the second embodiment.

[0200] The processing from step S601 to step S604 is the same as the processing from step S501 to step S504.

[0201] The input / output unit 106 corrects the base image based on the uncertainty map (step S605). For example, the following process is executed.

[0202] (S605-1) The input / output unit 106 identifies the class with the largest area (number of pixels) in the classification results based on the prediction results. For example, in the case of the base image of FIG. 20A, class A (white area) is identified as the class with the largest area.

[0203] (S605-2) The input / output unit 106 searches the prediction result DB 112 for a prediction result in which the number of pixels whose output is the identified class is the largest in the suspicious region.

[0204] (S605-3) The input / output unit 106 corrects the suspicious region of the base image based on the output of the suspicious region of the searched prediction result. The correction method is the same as that in step S510.

[0205] (S605-4) The input / output unit 106 modifies the base image by changing the output of pixels in each boundary region whose ΔC is smaller than the boundary type threshold Th3 to the class with the second most frequent value. Note that the class may also be changed to the third or subsequent class with the most frequent value.

[0206] (S605-5) The input / output unit 106 calculates the accuracy rate of each class in the learning phase. For example, the input / output unit 106 calculates the average accuracy rate of each class based on the accuracy rate of each class in each entry of the table 610.

[0207] (S605-6) When a correction is made to a class whose accuracy rate is lower than a predetermined threshold, the input / output unit 106 displays a message recommending the user to check the corrected base image. For example, when a new class area is added to the suspicious area and the accuracy rate of the class is lower than the threshold, a warning message is displayed.

[0208] The input / output unit 106 displays the base image and the corrected base image (step S606), and the prediction process ends.

[0209] In the second embodiment, since the user does not need to perform an input operation, the effort required to correct the result of classification based on the prediction result can be further reduced. [Example]

[0210] The method of displaying the uncertainty map is different in Example 3. Hereinafter, Example 3 will be described, focusing on the differences from Example 1.

[0211] The configuration of the computer system 100 of the third embodiment is the same as that of the first embodiment. The data structures of various databases of the third embodiment are the same as those of the first embodiment. The processes other than the uncertainty map generation process executed by the computer system 100 in the learning phase and prediction phase of the third embodiment are the same as those of the first embodiment.

[0212] In the third embodiment, the uncertainty map generation process is partially different. Specifically, the process content of step S302 is different from that of the first embodiment.

[0213] (S302-1) The input / output unit 106 divides the input data into a plurality of partial images. The number of divisions is set in advance.

[0214] (S302-2) The input / output unit 106 selects one uncertainty level from the uncertainty levels designated by the user.

[0215] (S302-3) The input / output unit 106 selects a partial image.

[0216] (S302-4) The input / output unit 106 calculates the area ratio of the pixels to which the selected uncertainty level has been assigned. Specifically, the input / output unit 106 divides the number of pixels to which the uncertainty level has been assigned by the total number of pixels in the partial image.

[0217] (S302-5) The input / output unit 106 predicts whether the area ratio is greater than a threshold value Th4. The threshold value Th4 is a preset threshold value. If the area ratio is equal to or less than the threshold value Th4, the input / output unit 106 proceeds to S302-7.

[0218] (S302-6) If the area ratio is greater than the threshold value Th4, the input / output unit 106 registers the partial image and the uncertainty level in the display list.

[0219] (S302-7) The input / output unit 106 determines whether or not the processing has been completed for all partial images. If the processing has not been completed for all partial images, the input / output unit 106 returns to S302-3.

[0220] (S302-8) When processing has been completed for all partial images, the input / output unit 106 determines whether processing has been completed for all uncertainty levels specified by the user. If processing has not been completed for all uncertainty levels specified by the user, the input / output unit 106 returns to S302-2.

[0221] (S302-9) When the processing has been completed for all the uncertainty levels specified by the user, the input / output unit 106 generates an uncertainty map based on the display list.

[0222] FIG. 25 is a diagram illustrating an example of an uncertainty map according to the third embodiment.

[0223] As shown in FIG. 25, in the third embodiment, pixels according to the uncertainty level are displayed only in the partial images registered in the display list.

[0224] According to the third embodiment, by displaying only a partial image in which a large number of pixels are assigned a specified uncertainty level, it is possible to reduce the effort required for correction.

[0225] The uncertainty map may be output regardless of the area ratio.

[0226] It should be noted that the present invention can also provide a machine learning model that uses data other than images, such as text and spectra, as input.

[0227] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0228] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0229] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java (registered trademark).

[0230] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

[0231] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected. [Explanation of symbols]

[0232] 100 Computer Systems 101 Learning Department 102 Prediction Department 103 Accuracy rate calculation section 104 Uncertainty index calculation part 105 Uncertainty Level Judgment Unit 106 Input / output section 110 Teacher Data DB 111 Model DB 112 Prediction result DB 113 Uncertainty Assessment DB 200 calculator 201 processor 202 Main storage 203 Secondary storage device 204 Network Interface 2100, 2200, 2300 GUI

Claims

1. 1. A computer system comprising: a processor, a storage device connected to the processor, and an input / output interface connected to the processor; A prediction model is stored that performs predictions for class classification that determines the class to which an image pixel belongs from among a plurality of classes. The prediction model is a machine learning model that calculates the probability that each pixel of the image belongs to each class, The processor: Accepts input of an input image, Transforming the prediction model, performing the prediction on the input image using the transformed prediction model, and storing the prediction results; calculating a probability distribution of each class for each pixel of the input image based on the plurality of prediction results; assigning an uncertainty level to the result of the class classification of the pixel of the input image by statistically processing the probability distribution of each class for each pixel of the input image; determining, for each pixel of the input image, a class to which the pixel belongs based on the probability distribution of each class, and generating a base image representing the class to which each pixel of the input image belongs; generating an uncertainty map representing the uncertainty level for each pixel of the input image; selecting pixels of the input image that require correction of the classification result based on the uncertainty map; A computer system comprising: a computer that modifies the base image by modifying the class classification results of selected pixels of the input image using a plurality of the prediction results.

2. 2. The computer system of claim 1, The processor: assigning, as the uncertainty level, one of a first level representing that the result of the class classification of the pixel of the input image is reliable, a second level representing that the result of the class classification of the pixel of the input image is unreliable, and a third level representing that the pixel of the input image is located on the boundary of multiple classes and therefore the class to which it belongs cannot be uniquely determined; A computer system characterized in that at least one of the pixels of the input image to which the second level has been assigned and the pixels of the input image to which the third level has been assigned is selected as the pixels of the input image that require correction of the class classification result.

3. 3. The computer system according to claim 2, The processor: selecting an area formed by pixels of the input image to which the second level has been assigned; searching for the prediction result in which the class to which the largest number of pixels in the region belongs is different from the class to which the largest number of pixels in the region of the base image belongs; A computer system characterized by modifying the class to which the pixels included in the region of the base image belong based on the result of the class classification of the pixels of the input image included in the region based on the searched prediction result.

4. 4. The computer system according to claim 3, The computer system is characterized in that the processor presents an interface for selecting the region to be corrected and an interface for selecting the retrieved prediction result.

5. 3. The computer system according to claim 2, The processor: selecting an area formed by pixels of the input image to which the third level has been assigned; A computer system that changes a class to which the pixels included in the region of the base image belong.

6. 6. The computer system according to claim 5, The processor: generating the base image by determining, for each pixel of the input image, the class to which the most frequent value of the probability distribution is the largest as the class to which the pixel of the input image belongs; identifying a first probability distribution of a class having the largest mode value of the probability distribution and a second probability distribution of a class having the second largest mode value of the probability distribution for pixels included in the region of the base image; A computer system characterized by changing the class to which pixels included in the area of ​​the base image, where the difference between the most frequent value of the first probability distribution and the most frequent value of the second probability distribution is smaller than a predetermined value, belong to to a class other than the class with the largest most frequent value of the probability distribution.

7. 6. The computer system according to claim 5, The computer system is characterized in that the processor presents an interface for selecting the region to be modified.

8. 3. The computer system according to claim 2, A validation result of the learning phase for generating the predictive model is stored; the verification result includes information about the accuracy rate of each class in the prediction model; The computer system is characterized in that, when a correction is made to the corrected base image for a class whose accuracy rate is less than a threshold, the processor displays a display recommending confirmation of the corrected base image.

9. A method for supporting correction of a classification result executed by a computer system, comprising: The computer system a processor, a storage device connected to the processor, and an input / output interface connected to the processor; A prediction model is stored that performs predictions for class classification that determines the class to which an image pixel belongs from among a plurality of classes. The prediction model is a machine learning model that calculates the probability that each pixel of the image belongs to each class, The method for supporting correction of a classification result includes: a first step in which the processor accepts an input of an input image; a second step in which the processor executes a process of transforming the prediction model, making the prediction for the input image using the transformed prediction model, and storing the prediction result multiple times; a third step in which the processor calculates a probability distribution of each class for each pixel of the input image based on a plurality of the prediction results; a fourth step in which the processor statistically processes the probability distribution of each class for each pixel of the input image to assign an uncertainty level to the result of the class classification of the pixel of the input image; a fifth step in which the processor determines, for each pixel of the input image, a class to which the pixel belongs based on the probability distribution of each class, and generates a base image representing the class to which each pixel of the input image belongs; a sixth step in which the processor generates an uncertainty map representing the uncertainty level for each pixel of the input image; the processor selecting pixels of the input image that require modification of the classification result based on the uncertainty map; a seventh step in which the processor modifies the base image by modifying the class classification results of selected pixels of the input image using a plurality of the prediction results; A method for supporting correction of classification results, comprising:

10. The method for supporting correction of classification results according to claim 9, the fourth step includes a step of the processor assigning, as the uncertainty level, any one of a first level representing that the result of the class classification of the pixel of the input image is reliable, a second level representing that the result of the class classification of the pixel of the input image is unreliable, and a third level representing that the pixel of the input image is located on the boundary of multiple classes and therefore the class to which it belongs cannot be uniquely determined; a step of selecting, by the processor, as pixels of the input image that require correction of the class classification result, at least one of pixels of the input image to which the second level has been assigned and pixels of the input image to which the third level has been assigned.

11. The method for supporting correction of classification results according to claim 10, The seventh step includes: the processor selecting an area formed by pixels of the input image to which the second level has been assigned; the processor searches for the prediction result in which a class to which the largest number of pixels in the region belongs is different from a class to which the largest number of pixels in the region of the base image belongs; The processor modifies the class to which the pixel included in the region of the base image belongs based on the result of the class classification of the pixel of the input image included in the region based on the searched prediction result; A method for supporting correction of classification results, comprising:

12. The method for supporting correction of classification results according to claim 10, The seventh step includes: an eighth step in which the processor selects an area formed by pixels of the input image to which the third level has been assigned; a ninth step in which the processor changes the class to which the pixels included in the region of the base image belong; A method for supporting correction of classification results, comprising:

13. The method for supporting correction of classification results according to claim 12, comprising: the fifth step includes a step of generating the base image by the processor determining, for each pixel of the input image, a class to which the pixel of the input image belongs, the class to which the most frequent value of the probability distribution is maximum; The ninth step includes: The processor identifies, for pixels included in the region of the base image, a first probability distribution of a class having the largest mode value of the probability distribution and a second probability distribution of a class having the second largest mode value of the probability distribution; the processor changes a class to which pixels included in the region of the base image, in which a difference between the mode of the first probability distribution and the mode of the second probability distribution is smaller than a predetermined value, belong to a class other than the class in which the mode of the probability distribution is largest; A method for supporting correction of classification results, comprising:

14. The method for supporting correction of classification results according to claim 10, A validation result of the learning phase for generating the predictive model is stored; the verification result includes information about the accuracy rate of each class in the prediction model; The method for supporting correction of class classification results is characterized by including a step in which, when a correction is made to the corrected base image for a class whose accuracy rate is smaller than a threshold, the processor displays a display recommending confirmation of the corrected base image.

Images