Computer system, and method for supporting correction of class classification results

Abstract

This computer system: performs a process of receiving an input of an input image, transforming a prediction model, performing a prediction for the input image using the transformed prediction model, and storing the prediction result, a plurality of times; calculates the probability distribution of each class for each pixel of the input image on the basis of a plurality of prediction results; statistically processes the probability distribution of each class and thereby assigns uncertainty levels to pixel class classification results; determines the class to which each pixel belongs on the basis of the probability distribution of each class; generates a base image representing the class to which each pixel belongs; generates an uncertainty map representing the uncertainty level for each pixel of the input image; selects pixels of the input image, the class classification results of which need to be corrected, on the basis of the uncertainty map; and corrects the class classification results of the selected pixels of the input image using the plurality of prediction results.

Description

Computer system and method for supporting correction of classification results Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2024-19750, filed February 13, 2024, the contents of which are incorporated herein by reference.

[0002] The present disclosure relates to techniques for correcting semantic segmentation results.

[0003] Artificial intelligence (AI) is increasingly being used in a variety of fields to classify images into multiple regions. For example, in the field of materials science, AI is used to extract features from tissue images and model their relationship with physical properties, which is expected to improve the efficiency of new material development. Semantic segmentation (SS) is a widely used AI machine learning model for analyzing tissue features.

[0004] It is important to evaluate the uncertainty of prediction results, which are the output of a machine learning model, along with the accuracy of the prediction. A method described in Patent Literature 1 is known as a method for quantifying uncertainty. Patent Literature 1 describes a method for evaluating uncertainty by repeatedly applying Monte Carlo dropout.

[0005] Japanese Patent Application Laid-Open No. 2018-200677

[0006] The uncertainty map in Patent Document 1, which visualizes the uncertainty index, can limit the areas where the prediction is incorrect, but the task of manually correcting the prediction results requires a lot of work.

[0007] The present disclosure relates to techniques for modifying semantic segmentation results based on uncertainty in the results.

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a computer system includes a processor, a storage device connected to the processor, and an input / output interface connected to the processor, and holds definition information of a prediction model that makes predictions for class classification that determines a class to which a pixel of an image belongs from among a plurality of classes, the prediction model is a machine learning model that calculates the probability that each pixel of the image belongs to each class, the processor accepts input of an input image, transforms the prediction model, makes the prediction for the input image using the transformed prediction model, and executes a process of saving the prediction results a plurality of times, and calculates the probability distribution of each class for each pixel of the input image based on the plurality of prediction results. a distribution of the probability distributions of the classes for each pixel of the input image, and assigning an uncertainty level to the result of the class classification of the pixel of the input image by statistically processing the probability distribution of each class for each pixel of the input image; determining, for each pixel of the input image, the class to which the pixel belongs based on the probability distribution of each class; generating a base image representing the class to which each pixel of the input image belongs; generating an uncertainty map representing the uncertainty level for each pixel of the input image; selecting, based on the uncertainty map, pixels of the input image whose class classification results require correction; and correcting the base image by correcting the result of the class classification of the selected pixels of the input image using a plurality of the prediction results.

[0009] According to one aspect of the present disclosure, an image representing a classification result can be automatically corrected based on the uncertainty of the classification result (semantic segmentation). Problems, configurations, and advantages other than those described above will become apparent from the following description of the embodiments.

[0010] 1 is a diagram illustrating an example of a functional configuration of a computer system according to Example 1. FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer constituting the computer system according to Example 1. FIG. 3 is a diagram illustrating cooperation of functions in the learning phase of the computer system according to Example 1. FIG. 4 is a diagram illustrating an example of the data structure of a teacher data DB according to Example 1. FIG. 5 is a diagram illustrating an example of the data structure of a model DB according to Example 1. FIG. 6 is a diagram illustrating an example of the data structure of a prediction result DB according to Example 1. FIG. 7 is a diagram illustrating an example of the data structure of a prediction result DB according to Example 1. FIG. 8 is a diagram illustrating an example of the data structure of an uncertainty assessment DB according to Example 1. FIG. 9 is a diagram illustrating an example of the data structure of an uncertainty assessment DB according to Example 1. FIG. 10 is a diagram illustrating the characteristics of prediction in semantic segmentation. FIG. 11 is a diagram illustrating the characteristics of prediction in semantic segmentation. FIG. 12 is a flowchart illustrating an example of a prediction process executed by the computer system according to Example 1 in the learning phase. FIG. 13 is a diagram illustrating an example of a probability distribution of classes calculated by the computer system according to Example 1. FIG. 14 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system according to Example 1. FIG. 15 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system according to Example 1. FIG. 1 is a diagram showing an example of the correlation between the uncertainty index and the accuracy rate in Example 1. FIG. 2 is a diagram showing an example of the correlation between the uncertainty index and the accuracy rate in Example 1. FIG. 3 is a diagram showing an example of a method for calculating an evaluation index in Example 1. FIG. 4 is a diagram showing an example of boundary region processing performed by the computer system of Example 1. FIG. 5 is a diagram showing an example of boundary region processing performed by the computer system of Example 1. FIG. 6 is a diagram showing an example of boundary region processing performed by the computer system of Example 1. FIG. 7 is a diagram showing an example of the data structure of data generated by the computer system of Example 1. FIG. 8 is a diagram showing an example of the data structure of data generated by the computer system of Example 1. FIG. 9 is a flowchart explaining an example of uncertainty map generation processing performed by the computer system of Example 1. FIG. 10 is a diagram showing the coordination of functions in the prediction phase of the computer system of Example 1.1 is a flowchart illustrating an example of a prediction process executed by the computer system of Example 1 in a prediction phase. FIG. 1 is a flowchart illustrating an example of a class classification process executed by the computer system of Example 1. FIG. 1 is a flowchart illustrating an example of a class classification process executed by the computer system of Example 1. FIG. 1 is a diagram illustrating an example of an image output in the class classification process of Example 1. FIG. 1 is a diagram illustrating an example of an image output in the class classification process of Example 1. FIG. 1 is a diagram illustrating an example of an image output in the class classification process of Example 1. FIG. 1 is a diagram illustrating an example of an image output in the class classification process of Example 1. FIG. 2 is a diagram illustrating an example of an image output in the class classification process of Example 1. FIG. 3 is a diagram illustrating an example of a GUI presented by the computer system of Example 1. FIG. 4 is a diagram illustrating an example of a GUI presented by the computer system of Example 1. FIG. 5 is a flowchart illustrating an example of a class classification process executed by the computer system of Example 2 in a prediction phase. FIG. 6 is a diagram illustrating an example of an uncertainty map of Example 3.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the concept or spirit of the present invention.

[0012] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant explanations will be omitted.

[0013] In this specification, the terms "first," "second," "third," etc. are used to identify components and do not necessarily limit the number or order.

[0014] Fig. 1 is a diagram illustrating an example of the functional configuration of a computer system according to Example 1. Fig. 2 is a diagram illustrating an example of the hardware configuration of a computer that constitutes the computer system according to Example 1.

[0015] The computer system 100 is configured from a computer 200 as shown in Fig. 2. The computer 200 includes a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204. The computer 200 may also include input devices such as a mouse and a keyboard, and an output device such as a display.

[0016] The computer system 100 includes a learning unit 101, a prediction unit 102, an accuracy rate calculation unit 103, an uncertainty index calculation unit 104, an uncertainty level determination unit 105, and an input / output unit 106. The computer system 100 also stores a teacher data DB 110, a model DB 111, a prediction result DB 112, and an uncertainty assessment DB 113.

[0017] The training data DB 110 is a database that stores training data composed of input data to be input to the machine learning model and correct answer data for prediction. In this embodiment, the input data is an image, and the correct answer data is an array of classes to which pixels belong. The model DB 111 is a database that stores information that defines the machine learning model. The prediction result DB 112 is a database that stores the results of predictions made using the machine learning model. The uncertainty assessment DB 113 is a database that stores evaluation information regarding the uncertainty of the results of class classification based on the prediction results.

[0018] It should be noted that with regard to each functional unit of the computer system 100, multiple functional units may be combined into one functional unit, or one functional unit may be divided into multiple functional units for each function. Furthermore, the databases held by the computer system 100 may be integrated into one database, or one database may be divided into multiple databases depending on the purpose of data management.

[0019] First, the processing in the learning phase will be described. Fig. 3 is a diagram showing the cooperation of functions in the learning phase of the computer system 100 according to the first embodiment.

[0020] The learning unit 101 acquires the training data stored in the training data DB 110 as training data, and executes a learning process for a machine learning model to realize semantic segmentation.

[0021] The machine learning model is a model that calculates the probability that a pixel belongs to each class, and is, for example, a convolutional neural network (CNN), SegNet, U-Net, or PSPNet. The machine learning method is, for example, deep learning. The present invention is not limited to the type of machine learning model. The present invention is not limited to the learning method of the machine learning model. The learning unit 101 stores the learning results in the model DB 111.

[0022] The prediction unit 102 acquires training data stored in the training data DB 110 as verification data and performs prediction using a machine learning model defined by information stored in the model DB 111. In this embodiment, an array of probabilities that pixels of the image belong to each class is output as a prediction result. The prediction unit 102 stores the prediction result in the prediction result DB 112.

[0023] In this embodiment, the prediction unit 102 generates a machine learning model (hereinafter referred to as an MCD model) in which some of the nodes constituting the machine learning model are dropped out using a Monte Carlo dropout method, and predicts validation data using the MCD model. Predictions are performed multiple times for one validation data set using different MCD models.

[0024] The accuracy rate calculation unit 103 uses the prediction result and the correct answer data included in the verification data to calculate the accuracy rate for the input data, the accuracy rate for each class, etc. The accuracy rate calculation unit 103 stores the processing results in the prediction result DB 112.

[0025] The uncertainty index calculation unit 104 calculates a plurality of uncertainty indexes using the prediction results. The uncertainty index calculation unit 104 stores the processing results in the uncertainty assessment DB 113. The calculated uncertainty indexes will be described later.

[0026] The uncertainty level determination unit 105 assigns an uncertainty level to the result of class classification of each pixel based on the prediction result for one piece of verification data using multiple uncertainty indices. The uncertainty level determination unit 105 stores the processing result in the uncertainty assessment DB 113.

[0027] The input / output unit 106 receives input to the computer system 100 and outputs various data.

[0028] FIG. 4 is a diagram illustrating an example of a data structure of the teacher data DB 110 according to the first embodiment.

[0029] The teacher data DB 110 stores a table 400 for managing teacher data. The table 400 stores entries including a data ID 401, input data 402, and correct answer data 403. One entry corresponds to one piece of teacher data.

[0030] The data ID 401 is a field that stores the ID of the training data. The input data 402 is a field that stores the input data to be input to the machine learning model. In this embodiment, an array of pixel values ​​of pixels that make up an image is stored as the input data. The correct answer data 403 is a field that stores correct answer data that indicates the correct answer of the prediction. In this embodiment, an array of values ​​that indicates the class to which the pixel belongs is stored as the correct answer data. FIG. 4 shows a two-dimensional array that indicates whether the pixel belongs to the first class or the second class.

[0031] The data format of the teacher data DB 110 is not limited to the table format.

[0032] FIG. 5 is a diagram illustrating an example of a data structure of the model DB 111 according to the first embodiment.

[0033] The model DB 111 stores a table 500 that stores information that defines a machine learning model. One table 500 exists for one machine learning model. The table 500 stores entries including a layer ID 501, a layer type 502, and parameters 503. One entry exists for one layer.

[0034] The layer ID 501 is a field that stores the ID of a layer of a machine learning model. The layer type 502 is a field that stores the type of layer. Layer types include an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The parameter 503 is a field that stores the parameter of the layer corresponding to the layer ID 501. The layer parameter is, for example, a weight.

[0035] The data format of the model DB 111 is not limited to the table format.

[0036] 6A and 6B are diagrams illustrating an example of a data structure of the prediction result DB 112 according to the first embodiment.

[0037] The prediction result DB 112 stores a table 600 and a table 610 .

[0038] The table 600 is a table for managing prediction results. The table 600 stores entries including a prediction result ID 601, a data ID 602, and a prediction result 603. One entry exists for one prediction result.

[0039] The prediction result ID 601 is a field that stores the ID of the prediction result. The data ID 602 ​​is a field that stores the ID of the input data on which the prediction was made. In the learning phase, the ID of the verification data is stored, and in the prediction phase, the ID of the input data itself is stored. The prediction result 603 is a field that stores the prediction result. In this embodiment, an array of the probability that a pixel belongs to each class is stored as the prediction result. In FIG. 6A, an array of the probability that a pixel belongs to each of the first and second classes is stored. The sum of the probabilities of each class of a pixel is 1.

[0040] The table 610 is a table for managing the accuracy rate of the prediction results. The table 610 stores entries including a prediction result ID 611 and an accuracy rate 612. One entry exists for one prediction result.

[0041] The prediction result ID 611 is the same field as the prediction result ID 601. The accuracy rate 612 is a field for storing the accuracy rate. In this embodiment, the accuracy rate for each pixel, the accuracy rate for the entire input data, the accuracy rate for each class, etc. are stored. The accuracy rate for the entire input data can be calculated, for example, as the average of the accuracy rates for each pixel. The accuracy rate for a class can be calculated, for example, as the average of the accuracy rates for pixels assigned the same class in the correct data.

[0042] The information managed in the tables 600 and 610 may be managed in a data format other than the table format.

[0043] 7A and 7B are diagrams illustrating an example of a data structure of the uncertainty assessment DB 113 according to the first embodiment.

[0044] The uncertainty assessment DB 113 stores a table 700 and a table 710 .

[0045] Table 700 is a table for managing uncertainty indices for evaluating the uncertainty of pixel class classification results based on predictions for input data. Table 700 stores entries including a data ID 701, a first uncertainty index 702, and a second uncertainty index 703. One entry exists for one piece of input data.

[0046] The data ID 701 is the same field as the data ID 602. The first uncertainty index 702 and the second uncertainty index 703 are fields for storing uncertainty indexes. Details of each uncertainty index will be described later.

[0047] The table 710 is a table for managing the uncertainty level assigned to the input data. The table 710 stores entries including a data ID 711 and an uncertainty level 712. One entry exists for one piece of input data.

[0048] The data ID 711 is the same field as the data ID 602. The uncertainty level 712 is a field that stores the uncertainty level. In this embodiment, an array of the uncertainty levels of the pixels is stored.

[0049] The information managed in the tables 700 and 710 may be managed in a data format other than the table format.

[0050] Here, the characteristics of prediction in semantic segmentation will be described. Figures 8A, 8B, and 8C are diagrams illustrating the characteristics of prediction in semantic segmentation.

[0051] 8A, 8B, and 8C show the distribution of the probability that a pixel belongs to a class. Here, semantic segmentation for classification into three classes will be described as an example.

[0052] As shown in FIG. 8A, if the variance of the probability distribution of a certain class is small and the most frequent value is close to 1, and the variance of the probability distribution of another class is small and the most frequent value is close to 0, the prediction result is sufficiently reliable.

[0053] As shown in FIG. 8B, if the probability distributions of each class overlap and the most frequent probability of each class is small (for example, less than 0.5), the prediction results are unreliable.

[0054] As shown in Figure 8C, when the probability distributions of two classes overlap, the mode of the probability distributions of the two classes is large (for example, greater than 0.5), and the mode of the probability distribution of the other class is approximately 0, it is highly likely that this is the boundary between the two classes, and it is not possible to uniquely determine which class the data belongs to.

[0055] In this embodiment, the results of pixel classification are classified into the above three levels using two uncertainty indices calculated from the probability distribution.

[0056] Fig. 9 is a flowchart illustrating an example of a prediction process executed in the learning phase by the computer system 100 according to the first embodiment. Fig. 10 is a diagram illustrating an example of a probability distribution of classes calculated by the computer system 100 according to the first embodiment.

[0057] The prediction unit 102 acquires definition information of the machine learning model from the model DB 111 (step S101).

[0058] The prediction unit 102 starts a loop process of the verification data (step S102). Specifically, the prediction unit 102 acquires one piece of verification data from the training data DB 110.

[0059] The prediction unit 102 starts a prediction loop process (step S103). Specifically, the prediction unit 102 generates an MCD model using the Monte Carlo dropout method. In this loop process, prediction is performed a predetermined number of times. Note that, since it is only necessary to obtain the probability distribution of the classes, the model may be modified using a method other than the Monte Carlo dropout method.

[0060] The prediction unit 102 performs prediction of the verification data using the MCD model (step S104). The prediction unit 102 stores the prediction result in the table 600 of the prediction result DB 112.

[0061] The prediction unit 102 instructs the accuracy rate calculation unit 103 to calculate the accuracy rate. The instruction includes the ID of the prediction result.

[0062] The accuracy rate calculation unit 103 compares the prediction result with the correct data included in the verification data and calculates the accuracy rate (step S105). The accuracy rate calculation unit 103 stores the calculation result in table 610 of the prediction result DB 112. Thereafter, the accuracy rate calculation unit 103 notifies the prediction unit 102 of the completion of the processing.

[0063] The prediction unit 102 predicts whether the number of times the prediction has been executed is smaller than a predetermined number (step S106). If the number of times the prediction has been executed is smaller than the predetermined number, the prediction unit 102 returns to step S103.

[0064] If the number of times the prediction has been performed is equal to or greater than the predetermined number, the prediction unit 102 instructs the uncertainty index calculation unit 104 to calculate an uncertainty index. The instruction includes the ID of the verification data.

[0065] The uncertainty index calculation unit 104 calculates the probability distribution of each class for each pixel (step S107). Specifically, the uncertainty index calculation unit 104 obtains the prediction result of the verification data from the table 600, and calculates the probability distribution of each class for each pixel as shown in FIG.

[0066] The uncertainty index calculation unit 104 calculates a first uncertainty index for each pixel based on the probability distribution of each class (step S108). Specifically, the following process is executed.

[0067] (S108-1) The uncertainty index calculation unit 104 selects a pixel.

[0068] (S108-2) The uncertainty index calculation unit 104 calculates the mode of the probability distribution of each class for the selected pixel.

[0069] (S108-3) The uncertainty index calculation unit 104 calculates the variance of the probability distribution of the class with the largest mode as a first uncertainty index.

[0070] (S108-4) The uncertainty index calculation unit 104 determines whether or not processing has been completed for all pixels. If processing has not been completed for all pixels, the uncertainty index calculation unit 104 returns to S108-1. If processing has been completed for all pixels, the uncertainty index calculation unit 104 stores the calculation results in table 700 of the uncertainty assessment DB 113.

[0071] The uncertainty index calculation unit 104 calculates a second uncertainty index for each pixel based on the probability distribution of each class (step S109). Specifically, the following process is executed.

[0072] (S109-1) The uncertainty index calculation unit 104 selects a pixel.

[0073] (S109-2) The uncertainty index calculation unit 104 calculates the mode of the probability distribution of each class for the selected pixel.

[0074] (S109-3) The uncertainty index calculation unit 104 ranks the classes in descending order of the mode.

[0075] (S109-4) The uncertainty index calculation unit 104 calculates the sum of the most frequent values ​​of the classes ranked third and below as a second uncertainty index.

[0076] (S109-5) The uncertainty index calculation unit 104 determines whether or not processing has been completed for all pixels. If processing has not been completed for all pixels, the uncertainty index calculation unit 104 returns to S109-1. If processing has been completed for all pixels, the uncertainty index calculation unit 104 stores the calculation results in table 700 of the uncertainty assessment DB 113.

[0077] After calculating the second uncertainty index, the uncertainty index calculation unit 104 notifies the prediction unit 102 of the completion of the processing. The prediction unit 102 determines whether the processing has been completed for all the verification data (step S110).

[0078] If the processing has not been completed for all the verification data, the prediction unit 102 returns to step S102. If the processing has been completed for all the verification data, the prediction unit 102 ends the prediction process.

[0079] 11A, 11B, and 11C are flowcharts illustrating an example of uncertainty level determination processing executed by the computer system 100 of Example 1. FIGS. 12A and 12B are diagrams illustrating an example of the correlation between the uncertainty index and the accuracy rate in Example 1. FIG. 13 is a diagram illustrating an example of a method for calculating an evaluation index in Example 1. FIGS. 14A, 14B, and 14C are diagrams illustrating an example of boundary region processing executed by the computer system 100 of Example 1. FIGS. 15A and 15B are diagrams illustrating an example of the data structure of data generated by the computer system 100 of Example 1.

[0080] The uncertainty level determination unit 105 executes the following process for one piece of verification data.

[0081] The uncertainty level determination unit 105 analyzes the correlation between the first uncertainty index and the accuracy rate for the verification data (step S201). Specifically, the following process is executed.

[0082] (S201-1) The uncertainty level determination unit 105 obtains the accuracy rate of each pixel from the table 610, and also obtains the first uncertainty index of each pixel from the table 700.

[0083] (S201-2) The uncertainty level determination unit 105 analyzes the correlation between the accuracy rate and the first uncertainty index. For example, the analysis result shown in FIG. 12A is obtained. Graph 1201 shows the correlation between the accuracy rate and the first uncertainty index. The dotted bar graph shows the number of correct pixels for the first uncertainty index. The white bar graph shows the number of incorrect pixels for the first uncertainty index.

[0084] The uncertainty level determination unit 105 calculates a threshold value Th1 based on the correlation between the first uncertainty index and the accuracy rate (step S202).

[0085] Specifically, the uncertainty level determination unit 105 uses correlation to calculate an uncertainty index at which the accuracy rate becomes a value designated by the user, as a threshold Th1. In Fig. 12A, the threshold Th1 is 0.15.

[0086] The uncertainty level determination unit 105 analyzes the correlation between the second uncertainty index and the accuracy rate for the verification data (step S203). Specifically, the following process is executed.

[0087] (S203-1) The uncertainty level determination unit 105 obtains the accuracy rate of each pixel from the table 610, and also obtains the second uncertainty index of each pixel from the table 700.

[0088] (S203-2) The uncertainty level determination unit 105 analyzes the correlation between the accuracy rate and the second uncertainty index. For example, the analysis result shown in FIG. 12B is obtained. Graph 1202 shows the correlation between the accuracy rate and the second uncertainty index. The dotted bar graph shows the number of correct pixels for the second uncertainty index. The white bar graph shows the number of incorrect pixels for the second uncertainty index.

[0089] The uncertainty level determination unit 105 calculates the threshold value Th2 based on the correlation between the second uncertainty index and the accuracy rate (step S204). Specifically, the following process is executed.

[0090] (S204-1) The uncertainty level determination unit 105 calculates the change in the accuracy rate relative to the change in the second uncertainty index (the decreasing gradient of the accuracy rate).

[0091] (S204-2) The uncertainty level determination unit 105 calculates the maximum value of the second uncertainty index in the range where the gradient of decline of the accuracy rate is greatest as the threshold value Th2. In FIG. 12B, the gradient of decline of the second uncertainty index is greatest in the range between 0.1 and 0.2, so the threshold value Th2 is 0.2.

[0092] As shown in FIG. 13, the uncertainty level determination unit 105 calculates, for each pixel, the difference (ΔC) between the most frequent value (first value) of the class with the largest mode and the most frequent value (second value) of the class with the second largest mode (step S205).

[0093] The uncertainty level determining unit 105 extracts, for each pixel, a combination of the class with the largest mode value and the class with the second largest mode value as the type of boundary (step S206).

[0094] For example, in the case of the probability distribution shown in Fig. 13, the boundary type extracted is class A / class B. Note that the boundary may be extracted only from pixels having a probability distribution relationship as shown in Fig. 8C.

[0095] The uncertainty level determination unit 105 starts a loop process of boundary types (step S207). Specifically, the uncertainty level determination unit 105 selects one boundary type.

[0096] The uncertainty level determination unit 105 sets an initial value to the variable k (step S208). The initial value is, for example, 0.1. The initial value can be set arbitrarily.

[0097] The uncertainty level determination unit 105 starts a loop process of pixels (step S209). Specifically, the uncertainty level determination unit 105 selects one pixel of the verification data.

[0098] The uncertainty level determination unit 105 determines whether ΔC of the pixel is smaller than the variable k (step S210). If ΔC of each pixel in the prediction result is equal to or larger than the variable k, the uncertainty level determination unit 105 proceeds to step S212.

[0099] If ΔC of each pixel in the prediction result is smaller than the variable k, the uncertainty level determination unit 105 changes the output of the pixel to the class with the second most frequent value (step S211). After that, the uncertainty level determination unit 105 proceeds to step S212.

[0100] Normally, the output pixel class is determined to be the class with the largest mode, but for pixels where ΔC is smaller than the variable k, the class is changed to the class with the second largest mode. For example, FIG. 14A shows an image (classification result) generated by outputting the class with the largest mode. As shown in FIG. 14B, when ΔC of a pixel in a black area (boundary) is smaller than 0.1, the output is changed to the class with the second largest mode, resulting in an image as shown in FIG. 14C. The dotted line indicates the boundary before the change.

[0101] In step S212, the uncertainty level determination unit 105 determines whether or not the processing has been completed for all pixels (step S212). If the processing has not been completed for all pixels, the uncertainty level determination unit 105 returns to step S209.

[0102] When processing has been completed for all pixels, the uncertainty level determination unit 105 calculates the accuracy rate by comparing the image generated from the output of each pixel with the image generated based on the correct data (step S213).

[0103] The uncertainty level determination unit 105 records the processing result in a table 1500 as shown in FIG. 15A. The table 1500 is stored in the uncertainty assessment DB 113. The table 1500 stores entries including a boundary type 1501, a variable k 1502, and an accuracy rate 1503. In step S212, the uncertainty level determination unit 105 adds an entry to the table 1500, sets the selected boundary type to the boundary type 1501 of the entry, sets the current value of the variable k to the variable k 1502, and sets the calculated accuracy rate to the accuracy rate 1503.

[0104] The uncertainty level determination unit 105 determines whether the variable k is smaller than 0.9 (step S213). 0.9 is the maximum value of the variable k. The maximum value can be set arbitrarily.

[0105] If the variable k is smaller than 0.9, the uncertainty level determining unit 105 updates the variable k (step S215), and then returns to step S209. For example, the uncertainty level determining unit 105 adds 0.1 to the value of the variable k.

[0106] If the variable k is 0.9 or more, the uncertainty level determining unit 105 refers to the table 1510 and determines the value of the variable k with the highest accuracy rate as the boundary type threshold Th3 (step S213).

[0107] At this time, the uncertainty level determination unit 105 records the processing result in a table 1510 as shown in FIG. 15B. The table 1510 is stored in the uncertainty assessment DB 113. The table 1510 stores entries including a boundary type 1511, a ratio 1512, and a threshold value 1513. In step S213, the uncertainty level determination unit 105 adds an entry to the table 1510, sets the selected boundary type to the boundary type 1511 of the entry, and sets the value of the variable k with the highest accuracy rate to the threshold value 1513.

[0108] The uncertainty level determination unit 105 determines whether or not the processing has been completed for all boundary types (step S217). If the processing has not been completed for all boundary types, the uncertainty level determination unit 105 returns to step S207.

[0109] When the process has been completed for all boundary types, the uncertainty level determining unit 105 calculates the proportion of each boundary type (step S218).

[0110] Specifically, the uncertainty level determination unit 105 counts the number of pixels that output the second most frequent class for each boundary type. The uncertainty level determination unit 105 divides the number of pixels for each boundary type by the total number of pixels for each boundary type. The uncertainty level determination unit 105 sets the calculation result as the ratio 1512 for each entry in the table 1510.

[0111] The uncertainty level determination unit 105 starts a loop process of pixels (step S219). Specifically, the uncertainty level determination unit 105 selects one pixel of the verification data.

[0112] The uncertainty level determining unit 105 determines whether the first uncertainty index of the pixel is smaller than a threshold value Th1 (step S220).

[0113] If the first uncertainty index of the pixel is smaller than the threshold value Th1, the uncertainty level determination unit 105 assigns an uncertainty level of "0" (step S221). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in the table 710. Then, the uncertainty level determination unit 105 proceeds to step S228. The uncertainty level of "0" indicates that the classification result is reliable.

[0114] If the first uncertainty index of the pixel is equal to or greater than the threshold Th1, the uncertainty level determining unit 105 predicts whether the second uncertainty index of the pixel is greater than a threshold Th2 (step S222).

[0115] If the second uncertainty index of the pixel is greater than the threshold Th2, the uncertainty level determination unit 105 assigns an uncertainty level of "1" (step S223). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in the table 710. Then, the uncertainty level determination unit 105 proceeds to step S228. The uncertainty level of "1" indicates that the classification is unreliable.

[0116] If the second uncertainty index of the pixel is equal to or smaller than the threshold Th2, the uncertainty level determining unit 105 calculates ΔC based on the probability distribution of each class of the pixel (step S224).

[0117] The uncertainty level determination unit 105 identifies the type of boundary based on the probability distribution of each pixel class (step S225). Specifically, the uncertainty level determination unit 105 identifies the pair of the class with the largest mode and the class with the second largest mode as the type of boundary.

[0118] The uncertainty level determining unit 105 determines whether ΔC is smaller than the threshold value Th3 of the identified boundary type (step S226).

[0119] If ΔC is smaller than the threshold value Th3, the uncertainty level determination unit 105 assigns an uncertainty level of "2" (step S227). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in the table 710. Thereafter, the uncertainty level determination unit 105 proceeds to step S228. The uncertainty level of "2" indicates that the class of the pixel cannot be determined because the pixel is located on the boundary between areas of different classes.

[0120] In step S228, the uncertainty level determination unit 105 determines whether or not the process has been completed for all pixels (step S228). If the process has not been completed for all pixels, the uncertainty level determination unit 105 returns to step S219. If the process has been completed for all pixels, the uncertainty level determination unit 105 ends the uncertainty level determination process.

[0121] The uncertainty level determining unit 105 may set, for example, "-1" to pixels to which no uncertainty level has been assigned.

[0122] FIG. 16 is a flowchart illustrating an example of the uncertainty map generation process executed by the computer system 100 of the first embodiment.

[0123] The input / output unit 106 receives the verification data and the selection of the uncertainty level from the user (step S301).

[0124] The input / output unit 106 generates an uncertainty map that displays the uncertainty level of the selected verification data (step S302). Specifically, the following process is executed.

[0125] (S302-1) The input / output unit 106 refers to the table 710 of the uncertainty assessment DB 113 and searches for an entry corresponding to the selected verification data.

[0126] (S302-2) The input / output unit 106 identifies pixels to which the specified uncertainty level has been assigned based on the searched entries.

[0127] (S302-3) The input / output unit 106 generates an image of the same size as the input data (image) included in the verification data, and sets RGB values ​​corresponding to the uncertainty level to identified pixels of the image. This image is the uncertainty map. Note that the uncertainty map may be generated by superimposing the generated image on an image of class classification based on the input data or the prediction result.

[0128] The input / output unit 106 outputs the uncertainty map (step S303), and then ends the uncertainty map generation process.

[0129] It is possible to assign an uncertainty level to the classification results for input data contained in the training data and present the uncertainty levels as an image (uncertainty map). Users can refer to the uncertainty map to correct the correct data. By performing machine learning using the corrected training data, it is possible to improve the accuracy of predictions by the machine learning model and reduce uncertainty.

[0130] The following describes the processing in the prediction phase of Example 1. Fig. 17 is a diagram showing the cooperation of functions in the prediction phase of the computer system 100 of Example 1.

[0131] The prediction unit 102 acquires input data to be predicted, and performs prediction using a machine learning model defined by information stored in the model DB 111. The prediction unit 102 stores the prediction result in the prediction result DB 112.

[0132] The uncertainty index calculation unit 104 calculates a plurality of uncertainty indexes using the prediction results, and stores the processing results in the uncertainty assessment DB 113.

[0133] The uncertainty level determination unit 105 assigns an uncertainty level to the result of class classification of each pixel based on the prediction result of the input data to be predicted, using multiple uncertainty indices. The uncertainty level determination unit 105 stores the processing result in the uncertainty assessment DB 113.

[0134] The input / output unit 106 accepts input to the computer system 100 and outputs various data from the computer system 100. For example, the input / output unit 106 accepts input extraction conditions, searches the prediction result DB 112 for prediction results that satisfy the extraction conditions, and displays an image using the prediction results. At this time, the input / output unit 106 displays the input data and an uncertainty map. The input / output unit 106 also accepts an image selected by the user, and, based on the image, modifies an image that represents the class classification results of each pixel based on the initial prediction results, and displays the modified image.

[0135] FIG. 18 is a flowchart illustrating an example of a prediction process that the computer system 100 of the first embodiment executes in the prediction phase.

[0136] The prediction unit 102 acquires definition information of the machine learning model from the model DB 111 (step S401).

[0137] The prediction unit 102 starts a prediction loop process (step S402). The process of step S402 is the same as the process of step S103.

[0138] The prediction unit 102 uses the MCD model to perform prediction of the input data (image) to be predicted (step S403). The process of step S403 is the same as the process of step S104.

[0139] The prediction unit 102 determines whether the number of times the prediction has been performed is smaller than a predetermined number (step S404). The process of step S404 is the same as the process of step S106.

[0140] If the number of times the prediction has been executed is equal to or greater than the predetermined number, the prediction unit 102 instructs the uncertainty index calculation unit 104 to calculate an uncertainty index. The instruction includes the ID of the input data.

[0141] The uncertainty index calculation unit 104 calculates the probability distribution of each class for each pixel (step S405). The process of step S405 is the same as the process of step S107.

[0142] The uncertainty index calculation unit 104 calculates a first uncertainty index for each pixel based on the probability distribution of each class (step S406). The process of step S406 is the same as the process of step S108.

[0143] The uncertainty index calculation unit 104 calculates a second uncertainty index for each pixel based on the probability distribution of each class (step S407). The process of step S407 is the same as the process of step S109.

[0144] In the uncertainty level determination process of the prediction phase, the processes from step S201 to step S218 are not executed. The threshold value used to assign the uncertainty level is the threshold value determined in the learning phase. For example, the average value of the threshold values ​​of each validation data is used.

[0145] Figures 19A, 19B, and 19C are flowcharts illustrating an example of a classification process executed by the computer system 100 of Example 1. Figures 20A, 20B, 20C, 20D, 20E, and 20F are diagrams illustrating an example of an image output in the classification process of Example 1.

[0146] The input / output unit 106 receives input data and a selection of an uncertainty level from the user (step S501).

[0147] The input / output unit 106 acquires the probability distribution of the class of each pixel of the input data, and also acquires information on the uncertainty level of the input data from the table 710 of the uncertainty assessment DB 113 (step S502).

[0148] The input / output unit 106 generates a base image that displays the results of class classification based on the probability distribution of the class for each pixel of the input data (step S503). Specifically, the input / output unit 106 generates an image (base image) that outputs the class with the largest mode of the probability distribution for each pixel. For example, a base image such as that shown in FIG. 20A is generated. FIG. 20A is an example of an image classified into three classes. The shaded area represents a group of pixels classified into class C, the black area represents a group of pixels classified into class B, and the white area represents a group of pixels classified into class A.

[0149] The input / output unit 106 generates an uncertainty map that displays the uncertainty level of the selected input data (step S504). The processing of step S504 is the same as the processing of step S302. For example, uncertainty maps such as those shown in FIGS. 20B and 20C are generated. FIG. 20B is an uncertainty map when uncertainty level "1" is selected, and FIG. 20C is an uncertainty map when uncertainty level "2" is selected. In the following description, a region of pixels assigned with uncertainty level "1" will be referred to as a "questionable region," and a region of pixels assigned with uncertainty level "2" will be referred to as a "boundary region."

[0150] The uncertainty level does not have to be input. In this case, the input / output unit 106 generates an uncertainty map that allows each uncertainty level to be distinguished.

[0151] The input / output unit 106 outputs the base image and the uncertainty map (step S505), and then enters a waiting state (step S506) to wait for an input from the user.

[0152] The user refers to the base image and the uncertainty map to determine whether or not the base image needs to be modified. If the user determines that the suspicious region needs to be modified, the user inputs a first modification request to the computer system 100. If the user determines that the boundary region needs to be modified, the user inputs a second modification request to the computer system 100. If the user determines that no modification is required, the user inputs an end request to the computer system 100.

[0153] When the input / output unit 106 receives an input from the user, it determines whether the input is a first modification request (step S507).

[0154] If the input is not a first revision request, the input / output unit 106 determines whether the input is a second revision request (step S508).

[0155] If the input is not a second modification request, the input / output unit 106 ends the classification process.

[0156] If the input is a first correction request, the correction process for the suspicious region is started. First, the input / output unit 106 presents a screen for inputting correction conditions and accepts the correction conditions from the user (step S509). Here, the correction conditions are the number of the target suspicious region and candidate images.

[0157] The input / output unit 106 searches the prediction result DB 112 for a prediction result based on the correction condition, and displays a candidate image based on the prediction result (step S510). Specifically, the following process is executed.

[0158] (S510-1) The input / output unit 106 identifies the class with the largest area in the suspicious region of the base image, that is, the class with the largest number of pixels.

[0159] (S510-2) The input / output unit 106 refers to the table 600 in the prediction result DB 112 and counts the number of pixels in the suspicious region that have the highest probability value of the identified class. The input / output unit 106 acquires a specified number of prediction results in descending order of the number of pixels.

[0160] (S510-3) The input / output unit 106 generates and displays candidate images based on the obtained prediction results. Specifically, the input / output unit 106 generates candidate images so as to output the class with the largest probability value for each pixel. For example, a candidate image such as that shown in FIG. 20D is generated.

[0161] The user selects a candidate image that is believed to be correct from among the candidate images, and the input / output unit 106 accepts the selection of the candidate image from the user (step S511).

[0162] The input / output unit 106 modifies and displays the base image based on the selected candidate image (step S512). The input / output unit 106 then returns to step S506. Specifically, the input / output unit 106 changes the class to which the pixel included in the suspicious region of the base image belongs to the class with the highest probability value of the suspicious region of the selected candidate image. For example, if the candidate image shown in FIG. 20D is selected for a base image such as that shown in FIG. 20A, the base image is modified as shown in FIG. 20E.

[0163] If the input is a second modification request, the modification process of the boundary area is started. First, the input / output unit 106 identifies the boundary types in the base image and calculates the proportion of each boundary type (step S513). The process of calculating the boundary proportion is the same as the process in step S218.

[0164] The input / output unit 106 acquires the threshold value Th3 for each boundary type from the table 1510 of the uncertainty assessment DB 113 (step S514). After that, the input / output unit 106 presents an interface for selecting a target boundary region.

[0165] When the input / output unit 106 receives a selection of a target boundary area from the user (step S515), it modifies and displays the display of the boundary area (step S516). After that, the input / output unit 106 transitions to a waiting state (step S517). In step S516, the following process is executed.

[0166] (S516-1) The input / output unit 106 identifies the boundary type of the selected boundary area. The type of boundary area can be identified based on the probability distribution of the classes of pixels included in the boundary area.

[0167] (S516-2) The input / output unit 106 selects one pixel included in the selected boundary region.

[0168] (S516-3) The input / output unit 106 determines whether ΔC of the selected pixel is smaller than the threshold value Th3 of the identified boundary type. If ΔC of the selected pixel is smaller than the threshold value Th3 of the identified boundary type, the input / output unit 106 changes the output of the pixel to the class with the second most frequent value. Note that the class may also be changed to the third or subsequent class with the most frequent value.

[0169] (S516-4) The input / output unit 106 determines whether or not the processing has been completed for all pixels included in the selected boundary region. If the processing has not been completed, the input / output unit 106 returns to S516-2.

[0170] (S516-5) When the processing is completed, the input / output unit 106 modifies the base image based on the processing results of each pixel included in the selected boundary region, and generates a candidate image. For example, the base image is modified as shown in FIG. 20F.

[0171] The user refers to the corrected base image and determines whether or not the boundary region needs to be corrected. If it is determined that the boundary region needs to be corrected, the user inputs a correction instruction including a new threshold value Th3. If it is determined that the boundary region does not need to be corrected, the user inputs a completion instruction.

[0172] When the input / output unit 106 receives an input from the user, it determines whether the input is a correction instruction (step S518).

[0173] If the received input is a correction instruction, the input / output unit 106 returns to step S516. If the received input is a completion instruction, the input / output unit 106 returns to step S506.

[0174] 21, 22, and 23 are diagrams showing examples of a GUI presented by the computer system 100 of the first embodiment.

[0175] The GUI 2100 is presented by the input / output unit 106. The GUI 2100 includes a setting area 2101 and a display area 2102.

[0176] The setting area 2101 is an area for making various settings necessary for processing. The setting area 2101 includes input fields 2110 and 2111, a selection field 2112, and an operation button 2114.

[0177] The input field 2110 is a field for inputting the machine learning model to be used. For example, the name of the table 400 that defines the machine learning model is input here. The input field 2111 is a field for inputting input data. For example, the name of the input data is input here.

[0178] The selection column 2113 is a column for selecting the uncertainty level to be displayed on the uncertainty map. The selection column 2113 displays check boxes for selecting the uncertainty level to be displayed.

[0179] The operation button 2114 is an operation button for issuing an instruction to execute the classification process.

[0180] The display area 2102 is an area for displaying the processing results and includes a display field 2120 and operation buttons 2121, 2122, and 2123.

[0181] Display field 2120 displays input data 2131, a base image 2132, and an uncertainty map 2133. The uncertainty map 2133 is displayed as an image in which RGB values ​​are set according to the uncertainty level.

[0182] The operation button 2121 is an operation button for inputting a first modification request, the operation button 2122 is an operation button for inputting a second modification request, and the operation button 2123 is an operation button for inputting an end request.

[0183] When the operation button 2121 on the GUI 2100 is operated, the input / output unit 106 presents a GUI 2200. The GUI 2200 includes a setting area 2201 and a display area 2202.

[0184] The setting area 2201 is an area for setting correction conditions, and includes a selection field 2210, an input field 2211, operation buttons 2212, and an extraction condition input field 2213.

[0185] The selection field 2210 is a field for selecting a target suspicious region. The input field 2211 is a field for inputting the number of candidate images. The operation button 2212 is an operation button for instructing generation of candidate images. The extraction condition input field 2213 is a field for inputting extraction conditions for candidate images.

[0186] The display area 2202 is an area for displaying the correction results of the candidate image and the base image, and includes display fields 2220 and 2222 and operation buttons 2221 and 2223.

[0187] The display field 2220 is a field for displaying candidate images. The operation button 2221 is an operation button for instructing correction of the base image. The user selects a candidate image from the display field 2220 and operates the operation button 2221.

[0188] The display field 2222 is a field for displaying the correction result of the base image. The operation button 2223 is an operation button for instructing completion of the correction of the questionable region.

[0189] When the operation button 2122 is operated on the GUI 2100, the input / output unit 106 presents a GUI 2300. The GUI 2300 includes a setting area 2301 and a display area 2302.

[0190] The setting area 2301 includes a selection field 2310 , an input field 2311 , and an operation button 2312 .

[0191] The selection field 2310 is a field for selecting a target boundary region. The input field 2311 is a field for inputting a threshold value Th3. The input field 2311 may be left blank. When the input field 2311 is blank, the threshold value Th3 obtained from the uncertainty assessment DB 113 is used. The operation button 2312 is an operation button for instructing correction of the base image.

[0192] The display area 2302 is an area for displaying the result of modifying the base image, and includes a display field 2320 and operation buttons 2321.

[0193] The display field 2320 is a field for displaying the result of modifying the base image. The operation button 2321 is an operation button for instructing completion of the modification of the boundary area.

[0194] The computer system 100 of the first embodiment assigns an uncertainty level to the class classification result of each pixel based on the prediction result of the input data and displays an uncertainty map. By referring to the uncertainty map, the user can determine whether or not the base image needs to be modified. Furthermore, when the computer system 100 receives a modification instruction from the user, it can automatically modify the base image using the uncertainty map. This reduces the effort required to modify the class classification result based on the prediction result.

[0195] In processing based on the mode of the probability distribution, the median and mean of the probability distribution may be substituted.

[0196] The classification process may also be performed in the learning phase, which can reduce the effort required to correct the correct answer data in the training data.

[0197] In the second embodiment, the computer system 100 automatically modifies the base image based on the uncertainty level. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0198] The configuration of the computer system 100 of the second embodiment is the same as that of the first embodiment. The data structures of the various databases of the second embodiment are the same as those of the first embodiment. The processing executed by the computer system 100 of the second embodiment in the learning phase is the same as that of the first embodiment.

[0199] The processing executed by the computer system 100 in the prediction phase in the second embodiment is partially different from that in the first embodiment. Specifically, the classification processing is partially different. The other processing is the same as that in the first embodiment.

[0200] FIG. 24 is a flowchart illustrating an example of a class classification process executed in the prediction phase by the computer system 100 according to the second embodiment.

[0201] The processing from step S601 to step S604 is the same as the processing from step S501 to step S504.

[0202] The input / output unit 106 corrects the base image based on the uncertainty map (step S605). For example, the following process is executed.

[0203] (S605-1) The input / output unit 106 identifies the class with the largest area (number of pixels) in the classification results based on the prediction results. For example, in the case of the base image of FIG. 20A, class A (white area) is identified as the class with the largest area.

[0204] (S605-2) The input / output unit 106 searches the prediction result DB 112 for a prediction result in which the number of pixels whose output is the identified class is the largest in the suspicious region.

[0205] (S605-3) The input / output unit 106 corrects the suspicious region of the base image based on the output of the suspicious region of the searched prediction result. The correction method is the same as that in step S510.

[0206] (S605-4) The input / output unit 106 modifies the base image by changing the output of pixels in each boundary region whose ΔC is smaller than the boundary type threshold Th3 to the class with the second most frequent value. Note that the class may also be changed to the third or subsequent class with the most frequent value.

[0207] (S605-5) The input / output unit 106 calculates the accuracy rate of each class in the learning phase. For example, the input / output unit 106 calculates the average accuracy rate of each class based on the accuracy rate of each class in each entry of the table 610.

[0208] (S605-6) When a correction has been made to a class whose accuracy rate is lower than a predetermined threshold, the input / output unit 106 displays a message recommending that the user check the corrected base image. For example, when a new class area is added to the suspicious area and the accuracy rate of the class is lower than the threshold, a warning message is displayed.

[0209] The input / output unit 106 displays the base image and the corrected base image (step S606), and the prediction process ends.

[0210] In the second embodiment, since no input work by the user is required, the effort required to correct the results of classification based on the prediction results can be further reduced.

[0211] The method of displaying the uncertainty map is different in Example 3. Hereinafter, Example 3 will be described, focusing on the differences from Example 1.

[0212] The configuration of the computer system 100 of the third embodiment is the same as that of the first embodiment. The data structures of various databases of the third embodiment are the same as those of the first embodiment. The processes other than the uncertainty map generation process executed by the computer system 100 in the learning phase and the prediction phase of the third embodiment are the same as those of the first embodiment.

[0213] In the third embodiment, the uncertainty map generation process is partially different. Specifically, the process content of step S302 is different from that of the first embodiment.

[0214] (S302-1) The input / output unit 106 divides the input data into a plurality of partial images. The number of divisions is set in advance.

[0215] (S302-2) The input / output unit 106 selects one uncertainty level from the uncertainty levels designated by the user.

[0216] (S302-3) The input / output unit 106 selects a partial image.

[0217] (S302-4) The input / output unit 106 calculates the area ratio of the pixels to which the selected uncertainty level has been assigned. Specifically, the input / output unit 106 divides the number of pixels to which the uncertainty level has been assigned by the total number of pixels in the partial image.

[0218] (S302-5) The input / output unit 106 predicts whether the area ratio is greater than a threshold value Th4. The threshold value Th4 is a preset threshold value. If the area ratio is equal to or less than the threshold value Th4, the input / output unit 106 proceeds to S302-7.

[0219] (S302-6) If the area ratio is greater than the threshold value Th4, the input / output unit 106 registers the partial image and the uncertainty level in the display list.

[0220] (S302-7) The input / output unit 106 determines whether or not the processing has been completed for all the partial images. If the processing has not been completed for all the partial images, the input / output unit 106 returns to S302-3.

[0221] (S302-8) When processing has been completed for all partial images, the input / output unit 106 determines whether processing has been completed for all uncertainty levels specified by the user. If processing has not been completed for all uncertainty levels specified by the user, the input / output unit 106 returns to S302-2.

[0222] (S302-9) When the processing is completed for all the uncertainty levels specified by the user, the input / output unit 106 generates an uncertainty map based on the display list.

[0223] FIG. 25 is a diagram illustrating an example of an uncertainty map according to the third embodiment.

[0224] As shown in FIG. 25, in the third embodiment, pixels according to the uncertainty level are displayed only in the partial images registered in the display list.

[0225] According to the third embodiment, by displaying only a partial image in which a large number of pixels are assigned a specified uncertainty level, it is possible to reduce the effort required for correction.

[0226] The uncertainty map may be output regardless of the area ratio.

[0227] It should be noted that the present invention can also provide a machine learning model that uses data other than images, such as text and spectra, as input.

[0228] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0229] Furthermore, some or all of the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.

[0230] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java.

[0231] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or the storage medium.

[0232] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected.

Claims

1. A computer system comprising a processor, a storage device connected to the processor, and an input / output interface connected to the processor, wherein the computer system holds definition information of a prediction model that makes predictions for class classification that determines the class to which a pixel of an image belongs from a plurality of classes, the prediction model being a machine learning model that calculates the probability that each pixel of the image belongs to each class, wherein the processor: accepts input of an input image; transforms the prediction model, makes the prediction for the input image using the transformed prediction model, and executes a process of saving the prediction results a plurality of times; calculates a probability distribution of each class for each pixel of the input image based on the plurality of prediction results; assigns an uncertainty level to the result of the class classification of the pixel of the input image by statistically processing the probability distribution of each class for each pixel of the input image; determines the class to which each pixel of the input image belongs based on the probability distribution of each class, and generates a base image that represents the class to which each pixel of the input image belongs; generates an uncertainty map that represents the uncertainty level for each pixel of the input image; and selects pixels of the input image that require correction of the result of the class classification based on the uncertainty map. A computer system comprising: a computer that modifies the base image by modifying the class classification results of selected pixels of the input image using a plurality of the prediction results.

2. A computer system as described in claim 1, wherein the processor assigns as the uncertainty level one of a first level indicating that the result of the class classification of the pixel of the input image is reliable, a second level indicating that the result of the class classification of the pixel of the input image is unreliable, and a third level indicating that the pixel of the input image is located on the boundary of multiple classes and therefore the class to which it belongs cannot be uniquely determined, and selects at least one of the pixels of the input image assigned the second level and the pixels of the input image assigned the third level as pixels of the input image that require correction of the class classification result.

3. A computer system as described in claim 2, characterized in that the processor selects an area formed by pixels of the input image to which the second level has been assigned, searches for prediction results in which the class to which the largest number of pixels belong in the area is different from the class to which the largest number of pixels belong in the area of ​​the base image, and modifies the class to which the pixels included in the area of ​​the base image belong based on the result of the class classification of the pixels of the input image included in the area based on the searched prediction results.

4. A computer system according to claim 3, characterized in that the processor presents an interface for selecting the area to be corrected and an interface for selecting the searched prediction result.

5. A computer system according to claim 2, characterized in that the processor selects an area formed by pixels of the input image to which the third level has been assigned, and changes the class to which the pixels included in the area of ​​the base image belong.

6. A computer system according to claim 5, wherein the processor generates the base image by determining, for each pixel of the input image, the class to which the pixel of the input image belongs, which is the class having the largest mode of the probability distribution; for pixels included in the region of the base image, identify a first probability distribution having the largest mode of the probability distribution and a second probability distribution having the second largest mode of the probability distribution; and change the class to which pixels included in the region of the base image belong, where the difference between the mode of the first probability distribution and the mode of the second probability distribution is smaller than a predetermined value, to a class other than the class having the largest mode of the probability distribution.

7. A computer system according to claim 5, wherein said processor presents an interface for selecting said area to be modified.

8. A computer system as described in claim 2, characterized in that it retains verification results from the learning phase in which the predictive model is generated, the verification results including information regarding the accuracy rate of each class in the predictive model, and the processor, when a correction is made to the corrected base image for a class whose accuracy rate is less than a threshold, displays a message recommending confirmation of the corrected base image.

9. A method for supporting correction of classification results executed by a computer system, wherein the computer system has a processor, a storage device connected to the processor, and an input / output interface connected to the processor, and holds definition information of a prediction model that makes predictions for performing classification to determine a class to which a pixel of an image belongs from among a plurality of classes, the prediction model being a machine learning model that calculates the probability that each pixel of the image belongs to each class, and the method for supporting correction of classification results comprises: a first step in which the processor receives input of an input image; a second step in which the processor executes a process multiple times to transform the prediction model, make the prediction for the input image using the transformed prediction model, and save the prediction results; a third step in which the processor calculates a probability distribution of each class for each pixel of the input image based on the multiple prediction results; and a fourth step in which the processor assigns an uncertainty level to the classification result of the pixel of the input image by statistically processing the probability distribution of each class for each pixel of the input image. a fifth step in which the processor determines, for each pixel of the input image, the class to which the pixel belongs based on the probability distribution of each class, and generates a base image representing the class to which each pixel of the input image belongs; a sixth step in which the processor generates an uncertainty map representing the uncertainty level of each pixel of the input image; and a seventh step in which the processor selects, based on the uncertainty map, pixels of the input image that require correction of the classification result, and corrects the base image by correcting the classification result of the selected pixels of the input image using a plurality of the prediction results.

10. A method for assisting in the correction of classification results as set forth in claim 9, wherein the fourth step includes a step in which the processor assigns, as the uncertainty level, one of a first level indicating that the result of the classification of the pixel of the input image is reliable, a second level indicating that the result of the classification of the pixel of the input image is unreliable, and a third level indicating that the pixel of the input image is located on the boundary between multiple classes and therefore the class to which it belongs cannot be uniquely determined; and the seventh step includes a step in which the processor selects at least one of the pixels of the input image to which the second level has been assigned and the pixels of the input image to which the third level has been assigned as pixels of the input image for which the result of the classification of the class needs to be corrected.

11. A method for assisting in the correction of classification results as described in claim 10, wherein the seventh step comprises: a step by the processor selecting an area formed by pixels of the input image to which the second level has been assigned; a step by the processor searching for the prediction result in which the class to which the largest number of pixels belong in that area is different from the class to which the largest number of pixels belong in that area of ​​the base image; and a step by the processor correcting the class to which the pixels included in that area of ​​the base image belong, based on the result of the class classification of the pixels of the input image included in that area based on the searched prediction result.

12. A method for assisting in the correction of class classification results as described in claim 10, wherein the seventh step includes: an eighth step in which the processor selects an area formed by pixels of the input image to which the third level has been assigned; and a ninth step in which the processor changes the class to which the pixels included in said area of ​​the base image belong.

13. A method for supporting correction of classification results as set forth in claim 12, wherein the fifth step includes a step in which the processor generates the base image by determining, for each pixel of the input image, the class having the largest mode of the probability distribution as the class to which the pixel of the input image belongs; and the ninth step includes a step in which the processor identifies, for pixels included in the region of the base image, a first probability distribution having the largest mode of the probability distribution and a second probability distribution having the second largest mode of the probability distribution; and a step in which the processor changes the class to which pixels included in the region of the base image, where the difference between the mode of the first probability distribution and the mode of the second probability distribution is smaller than a predetermined value, belong to a class other than the class having the largest mode of the probability distribution.

14. A method for supporting correction of class classification results as described in claim 10, comprising: retaining verification results from a learning phase in which the predictive model is generated; the verification results including information regarding the accuracy rate of each class in the predictive model; and comprising a step in which, when a correction is made to the corrected base image for a class whose accuracy rate is less than a threshold, the processor displays a display recommending confirmation of the corrected base image.

Images