Teacher data generator, training device, region detector, program, teacher data generation method, region estimator training method, and region detection method

Abstract

To contribute to improving the accuracy of detecting a region which is the object of detection.SOLUTION: A training data generator 10 comprises: an acquisition unit 11 for acquiring a plurality of designated regions that each of a plurality of annotators designated to an image of interest as the region to be detected; an allocation unit 12 for allocating, as a label, data selected from data of three or more mutually different values representing the degree of likelihood of being a detection object to each region in the image of interest, on the basis of the plurality of designated regions; and a generation unit 13 for associating the image of interest with the label allocated to each region of the image of interest and thereby generating teacher data of a region estimator that estimates the region to be detected.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The disclosure of this specification relates to a teacher data generation device, a learning device, an area detection device, a program, a teacher data generation method, an area estimator learning method, and an area detection method. [Background technology]

[0002] Region estimators are known for detecting target regions from images. For example, region estimators that detect lesion regions from images have been proposed and are used in the medical field to support doctors in making diagnoses.

[0003] To train a region estimator to detect a region of a detection target, training data consisting of an image and a correct label that identifies the region of the detection target in the image is required. In order to accurately detect a region of a detection target using a region estimator, it is desirable that the correct label correctly identifies the region of the detection target, but it is not necessarily easy to correctly annotate the correct label (see Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour, “Deep learning with noisy labels: exploring techniques and remedies in medical image analysis”, arXiv preprint arXiv:1912.02911,2019. Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-described techniques, there is a risk that the knowledge of multiple annotators may not be fully utilized. In view of the above-described circumstances, an object of one aspect of the present invention is to improve the detection accuracy of a detection target region. [Means for solving the problem]

[0006] A training data generation device according to one embodiment of the present invention includes an acquisition unit that acquires a plurality of designated areas that each of a plurality of annotators has designated as a region to be detected in a target image; an assignment unit that assigns data selected from three or more different data values ​​that represent the degree of likelihood of the region being a detection target to each region in the target image based on the plurality of designated areas, as a label; and a generation unit that generates training data for a region estimator that estimates the region to be detected by associating the target image with the labels assigned to each of the regions of the target image. [Effects of the Invention]

[0007] According to the above aspect, a technique is provided that contributes to improving the detection accuracy of the detection target area. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a system according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of a functional configuration of a system according to an embodiment. [Figure 3] 10 is a flowchart illustrating the procedure of a teacher data generation process performed in a system according to an embodiment. [Figure 4] FIG. 10 is a diagram showing an example of labels assigned to each region of a target image in which biopsy regions have been annotated by multiple specialists. [Figure 5] 10 is a flowchart illustrating a procedure of a learning process performed in a system according to an embodiment. [Figure 6] FIG. 10 is a diagram showing a data flow in a learning process performed in a system according to an embodiment. [Figure 7] 10 is a flowchart illustrating a procedure for area detection processing performed in a system according to an embodiment. [Figure 8] FIG. 10 is a diagram showing a data flow in an area detection process performed in a system according to an embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a display of a biopsy region detected by a region detection process performed by a system according to an embodiment. [Figure 10] FIG. 10 is a diagram showing another example of labels assigned to each region of a target image in which biopsy regions have been annotated by multiple specialists. [Figure 11] FIG. 10 is a diagram showing yet another example of labels assigned to each region of a target image in which biopsy regions have been annotated by multiple specialists. [Figure 12] FIG. 10 is a diagram showing an example of labels assigned to each region of a target image in which biopsy regions have been annotated by multiple specialists. [Figure 13] FIG. 10 is a diagram showing yet another example of labels assigned to each region of a target image in which biopsy regions have been annotated by multiple specialists. [Figure 14] FIG. 10 is a diagram illustrating a configuration of a system according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Fig. 1 is a diagram illustrating an example of the configuration of a system according to an embodiment. Fig. 2 is a diagram illustrating an example of the functional configuration of a system according to an embodiment. System 1 shown in Figs. 1 and 2 is a system that creates training data for a machine learning model that detects a target region, causes the machine learning model to learn the target region using the created training data, and detects the target region using the trained machine learning model.

[0010] In the following, a specific example of system 1 will be described, taking as an example a medical diagnostic support system that detects a biopsy region as a detection target region from an image for a colposcopy examination. However, system 1 may be a medical diagnostic support system that detects any region, not just a biopsy region. Furthermore, system 1 is not limited to medical diagnostic support systems and can be applied to any application that detects a region from an image, and can also be used in industrial applications such as product inspection.

[0011] 1, the system 1 includes a camera 2, a CPU 3, a GPU 4, an input device 5, a display device 6, and a storage device 7. The camera 2 is an imaging device of the system 1, and is used to capture images of, for example, the cervix.

[0012] The CPU 3, GPU 4, input device 5, display device 6, and storage device 7 constitute the computer of System 1. The CPU 3 is the control unit of the computer of System 1 and controls the operation of System 1. The GPU 4, together with the CPU 3, constitutes the calculation unit of the computer of System 1. The input device 5, display device 6, and storage device 7 are, for example, a touch panel, a liquid crystal display, and a semiconductor memory, and constitute the input unit, display unit, and storage unit of the computer of System 1, respectively.

[0013] The system 1 operates as a teacher data generation device 10, a learning device 20, and an area detection device 30 by the CPU 3 and the GPU 4 executing a program stored in the storage device 7. That is, the system 1 includes the teacher data generation device 10, the learning device 20, and the area detection device 30.

[0014] The teacher data generating device 10, the learning device 20, and the area detecting device 30 included in the system 1 will be described in detail below.

[0015] Fig. 3 is a flowchart showing the procedure of the teacher data generation process performed in a system according to one embodiment. Fig. 4 is a diagram showing an example of labels assigned to each region of a target image in which biopsy regions have been annotated by multiple specialists. The configuration of the teacher data generation device 10 included in the system 1 and the process performed by the teacher data generation device 10 will be described below with reference to Figs. 2 to 4.

[0016] The training data generation device 10 generates training data for a machine learning model that detects regions suspected of containing lesions as biopsy regions in a target image captured by a camera 2. Specifically, the training data generation device 10 is a device that generates training data based on multiple designated regions in a target image that are designated as biopsy regions by multiple annotators, and includes an acquisition unit 11, an allocation unit 12, and a storage unit 13, as shown in FIG.

[0017] The acquisition unit 11 acquires multiple designated regions in the target image designated as biopsy regions by each of the multiple annotators described above. The assignment unit 12 assigns data indicating the degree of likelihood of each region in the target image being a detection target (biopsy region) as a label based on the multiple designated regions acquired by the acquisition unit 11. Here, the data assigned as a label is data selected from data of three or more different values, for example, a variable value of an ordinal scale, an interval scale, or a proportional scale selected from three or more different values. The storage unit 13 is a generation unit that associates the target image with the labels assigned to each region of the target image and generates training data for a machine learning model that estimates a biopsy region, and stores the generated training data in a storage device. Note that the machine learning model that estimates a biopsy region is an example of a region estimator that infers a detection target region.

[0018] The acquisition unit 11, allocation unit 12, and storage unit 13 are realized when the processor of the computer of the teacher data generation device 10 (CPU 3 and GPU 4 shown in FIG. 1) executes a program for generating teacher data stored in the storage device 7, and the teacher data generation process shown in FIG. 3 is performed. Note that the teacher data generation process shown in FIG. 3 is one example of a method for generating teacher data. Note that the process may be performed by a single processor, or multiple processors may perform the process jointly.

[0019] When the processor executes the program for generating training data, the processor first acquires a target image captured by the camera 2 from the storage device 7, and displays the target image on the display device 6 (step S1).

[0020] Thereafter, when a plurality of specialist doctors who are annotators use the input device 5 to make annotations specifying biopsy regions on the target image displayed on the display device 6, the processor accepts the designation operation (step S2) and acquires the plurality of designated regions specified by the annotation (step S3). Here, as shown in Fig. 4, an example will be described in which three specialist doctors designate designated regions SA1, SA2, and SA3, which are regions within the target image, on the target image.

[0021] The processor assigns a label to each region in the target image based on the multiple designated regions acquired in step S3 (step S4). Here, the processor may, for example, count the number of votes for each region by regarding each specialist's region designation as one vote for each unit region within the designated region designated by the region designation, and assign the counted number of votes for each region as a label for that region. That is, in step S4, the processor may determine the variable value (label) of each unit region based on the number of designated regions that include that unit region among the multiple designated regions. This type of calculation is desirable because it is easy to perform and also provides an appropriate value indicating the degree of likelihood of detection. Note that each unit region is, for example, one pixel, but is not limited to one pixel. A collection of multiple pixels, such as 2 x 2 pixels, may also be considered as one unit region.

[0022] 4 shows that each unit area within the overlapping area of ​​designated area SA1, designated area SA2, and designated area SA3 is assigned a label of "3," each unit area within the overlapping area of ​​any two of designated areas SA1, designated area SA2, and designated area SA3 is assigned a label of "2," each unit area within areas within designated area SA1, designated area SA2, and designated area SA3 that do not overlap with other designated areas is assigned a label of "1," and each unit area within areas not included in designated area SA1, designated area SA2, or designated area SA3 is assigned a label of "0." The labels shown in FIG. 4 are proportional scale variable values ​​selected from four values: "0," "1," "2," and "3," and represent the degree of likelihood of a biopsy area based on the evaluations of three specialists.

[0023] After completing the assignment of labels to each unit area, the processor associates the target image acquired in step S1 with the label assigned to the target image in step S4, generates training data, and stores it in the storage device 7 (step S5).

[0024] Thereafter, it is determined whether or not the assignment of labels to all target images has been completed (step S6). If not (step S6 NO), the processor acquires a new target image in step S1 and repeats the processes of steps S2 to S5. Then, when the assignment of labels to all target images has been completed (step S6 YES), the processor ends the training data generation process shown in FIG.

[0025] The teacher data generation device 10 that performs the teacher data generation process shown in Fig. 3 generates teacher data in which labels (also called correct labels) are expressed as three or more values, as shown in Fig. 4. By using such teacher data, it is possible to improve the performance of the machine learning model, specifically, the detection accuracy of the detection target area. Therefore, the teacher data generation device 10 can contribute to improving the detection accuracy of the detection target area by the machine learning model. Below, it will be explained how the detection accuracy is improved by expressing the labels as three or more values.

[0026] Labels included in training data for machine learning models for region detection are usually expressed as binary values, since they only need to distinguish between regions to be detected and regions that are not to be detected. This type of binary representation is accurate and the most efficient when the region to be detected can be annotated with high precision (for example, when the region to be detected is known). However, there are many cases where the region to be detected cannot be determined at the training data creation stage (for example, when the region to be detected is unknown). Annotation by multiple annotators on the same image, as described above, is typically performed in the latter case.

[0027] In the latter case, if information originally provided by multiple annotators, which is ternary or higher, is converted into binary information and assigned as a label, some of the information entered by the multiple annotators will be lost. For example, when three designated regions shown in Figure 4 are designated to generate training data for a machine learning model that detects regions that all annotators identify as biopsy regions, the region is binarized by dividing it into the region where the three designated regions overlap (the region marked with "3" in Figure 4) and the other regions (the regions marked with "0," "1," and "2" in Figure 4). In this case, the information distinguishing between "0," "1," and "2" shown in Figure 4 will be lost.

[0028] In contrast, the teacher data generation device 10 can provide teacher data that reflects the information input by multiple annotators without compromising it by directly assigning three or more values ​​as labels. This makes it possible to fully utilize the annotators' knowledge through learning using this teacher data, thereby improving the accuracy of region detection. Specifically, in the above example, by distinguishing between regions that were not supported by any annotators (regions "0" in FIG. 4) and regions that were supported by some annotators (regions "1" and "2" in FIG. 4), it becomes possible to more accurately detect the region that was supported by all annotators (region "3" in FIG. 4), which is the target of detection.

[0029] In this example, the region supported by all annotators (region "3" in FIG. 4) was set as the detection target region, but the fact that this detection target region supported by all annotators is distinguished from other regions is the same between the teacher data using three or more values ​​generated by the teacher data generation device 10 and the conventional teacher data using two values. This point deserves much attention, and suggests the possibility of improving the detection accuracy of the detection target itself by classifying and learning objects other than the detection target in terms of their relevance to the detection target when learning the detection target.

[0030] Fig. 5 is a flowchart showing the procedure of the learning process performed in a system according to an embodiment. Fig. 6 is a diagram showing the flow of data in the learning process performed in a system according to an embodiment. The configuration of the learning device 20 included in the system 1 and the process performed by the learning device 20 will be described below with reference to Figs. 2 and 5 to 6.

[0031] Learning device 20 is a learning device that causes a machine learning model to learn a biopsy region, which is a region suspected of being a lesion in a target image captured by camera 2, as a region to be detected. Specifically, learning device 20 is a device that trains a machine learning model, which is a region estimator that detects a biopsy region from a target image, using teacher data generated by teacher data generation device 10, and includes an input unit 21, an inference unit 22, and an adjustment unit 23, as shown in FIG.

[0032] The input unit 21 inputs a target image included in the training data generated by the training data generation device 10 to the inference unit 22. The inference unit 22 is a region estimator that estimates a biopsy region, which is a region to be detected. Specifically, the inference unit 22, which is a region estimator, is a deep neural network, which is a machine learning model for supervised learning, and is a regression model that outputs a value indicating the likelihood that each region of the input target image is a biopsy region as an inference result. The adjustment unit 23 adjusts the parameters of the inference unit 22 based on a comparison result between the value corresponding to each region in the target image output from the inference unit 22 and the label value corresponding to each region included in the training data. Specifically, the adjustment unit 23 adjusts the parameters of the inference unit 22, specifically the weights and biases between nodes (i.e., edges) of the deep neural network, based on the difference between the value corresponding to each region in the target image output from the inference unit 22 and the label value corresponding to each region included in the training data, so as to reduce the difference.

[0033] The input unit 21, inference unit 22, and adjustment unit 23 are realized by the processor of the computer of the learning device 20 (CPU 3 and GPU 4 shown in FIG. 1) executing a program for learning the detection target area stored in the storage device 7, and performing the learning process shown in FIG. 5. The processing may be performed by a single processor, or by multiple processors working together. The learning process shown in FIG. 5 is one example of a method for learning the detection target area.

[0034] When the processor executes a program for learning the detection target region, the processor first acquires a target image T1 constituting training data from the storage device 7 and inputs it to the inference unit 22 (deep neural network) (step S11), as shown in Fig. 6. As a result, the inference unit 22 outputs a biopsy region map M1 as an inference result.

[0035] The processor then acquires the biopsy region map M1 output by the inference unit 22 and the label T2 associated with the target image T1 input to the inference unit 22, and inputs them to the adjustment unit 23. As a result, the adjustment unit 23 adjusts the parameters of the inference unit 22 based on the difference between the value (output value) of each region of the biopsy region map M1 and the label T2 (step S12). More specifically, the adjustment unit 23 performs a loss calculation (calculation of the gradient of the loss function) and sequentially updates the weight and bias parameters in a direction that reduces the error (difference) between the output value and the label. Note that, for example, a mean square error is used as the loss function.

[0036] Thereafter, it is determined whether or not to end the learning (step S13). If not (step S13 NO), the processor repeats the processes of steps S11 and S12 using new training data (a set of target image and label). When the learning using the training data to be learned is completed (step S13 YES), the processor ends the learning process shown in FIG.

[0037] The learning device 20 performing the learning process shown in FIG. 5 uses training data with multi-valued labels (three or more values) such as those shown in FIG. 4 to train a machine learning model (inference unit 22) using a biopsy region as a detection target region. In other words, unlike a method in which an unknown biopsy region is defined by estimation based on the opinions of multiple annotators in a pre-learning stage and the result (training data with binarized labels) is used to learn a biopsy region, the learning device 20 regards the opinions of multiple annotators themselves as a likelihood of a biopsy region and learns a biopsy region using the likelihood of a biopsy region. This makes it possible to suppress information loss in the pre-learning stage, thereby enabling accurate inference of the annotators' opinions, and as a result, improving the detection accuracy of the actual biopsy region, which is the detection target region. Therefore, the learning device 20 can also contribute to improving the detection accuracy of the detection target region by the machine learning model.

[0038] Fig. 7 is a flowchart showing the procedure of area detection processing performed in a system according to an embodiment. Fig. 8 is a diagram showing the flow of data in area detection processing performed in a system according to an embodiment. Fig. 9 is a diagram showing an example display of a biopsy area detected in area detection processing performed in a system according to an embodiment. The configuration of area detection device 30 included in system 1 and the processing performed by area detection device 30 will be described below with reference to Figs. 2 and 7 to 9.

[0039] The area detection device 30 is a device that detects a biopsy area, which is an area suspected of containing a lesion, in a target image captured by the camera 2. Specifically, the area detection device 30 is a device that detects a biopsy area, which is an area to be detected, based on an inference result output from a machine learning model trained by the learning device 20, and includes an input unit 31, an inference unit 32, a classification unit 33, and an output unit 34, as shown in FIG.

[0040] Input unit 31 inputs a target image for detecting a biopsy region to inference unit 32, which is a region estimator trained by learning device 20. Inference unit 32 is a region estimator that estimates a biopsy region, which is a region to be detected, and the parameters of inference unit 22 have been adjusted through learning. In other words, it is a deep neural network, which is a machine learning model for supervised learning, and is a regression model that outputs, as an inference result, a value indicating the likelihood that each region of the input target image is a biopsy region.

[0041] The classification unit 33 classifies values ​​corresponding to each region in the target image output from the inference unit 32 into a number of types, the number of which is less than the number of possible values ​​of the labels included in the training data, based on a threshold. As an example, the classification unit 33 classifies the values ​​into two types: "1" indicating a biopsy region and "0" indicating a non-biopsy region. Alternatively, the classification unit 33 may classify the values ​​into three types: "2" indicating a first biopsy region that is certain to be a biopsy region, "1" indicating a second biopsy region that is suspected to be a biopsy region, and "0" indicating a non-biopsy region. The output unit 34 outputs the results of the classification of the values ​​corresponding to each region in the target image by the classification unit 33 as the detection result of the region of the target detected from the target image.

[0042] The input unit 31, inference unit 32, classification unit 33, and output unit 34 are realized when the processor of the computer of the area detection device 30 (CPU 3 and GPU 4 shown in FIG. 1) executes a program for detecting the area of ​​the detection target stored in the storage device 7, and performs the area detection process shown in FIG. 7. The process may be performed by a single processor, or by multiple processors working together. The area detection process shown in FIG. 7 is one example of a method for detecting the area of ​​the detection target.

[0043] When the processor executes a program for detecting a detection target region, the processor first acquires a target image T3 from the storage device 7 and inputs it to the inference unit 32 (deep neural network) (step S21), as shown in Fig. 8. As a result, the inference unit 32 outputs a biopsy region map M2 as an inference result.

[0044] The processor then acquires the biopsy region map M2 output by the inference unit 32 and inputs it to the classification unit 33. As a result, the classification unit 33 classifies the values ​​(output values) of each region of the biopsy region map M2 based on a threshold (step S22). Here, the classification is performed into a number of values ​​equal to or less than the number of possible values ​​of the labels of the training data.

[0045] For example, if the machine learning model of the inference unit 32 is trained using training data with labels consisting of four values ​​from 0 to 3, as shown in FIG. 4, the output values ​​will range from approximately 0 to 3, and these values ​​will be classified into three or fewer types. If a region determined to be a biopsy region by all three annotators is to be inferred as a biopsy region, and the remaining regions are to be inferred as non-biopsy regions, the threshold may be set to, for example, 2.5 (or a value greater than this). As a result, values ​​greater than or equal to 2.5 and less than 2.5 are classified into two values, for example, "1" and "0." Furthermore, if a region determined to be a biopsy region by all three annotators is to be inferred as a first biopsy region, and a region determined to be a biopsy region by two annotators is to be inferred as a second biopsy region, and the remaining regions are to be inferred as non-biopsy regions, the threshold may be set to, for example, 2.5 and 1.5. As a result, values ​​of 2.5 or greater, values ​​less than 2.5 and greater than 1.5, and values ​​less than 1.5 are classified into three values, for example, "2", "1", and "0".

[0046] Finally, the processor outputs the classification result as a biopsy region detection result R1 (step S23). The processor may output the biopsy region detection result R1, which is, for example, an image of the same size as the target image, to the display device 6 and display it on the display device 6. In other words, outputting the classification result as the biopsy region detection result R1 may mean displaying the classification result as an image.

[0047] Instead of or in addition to displaying the classification result on the display device 6, the processor may output the classification result as a biopsy region detection result R1 to the storage device 7 and store it in a non-volatile manner in the storage device 7. The processor may also process the target image using the biopsy region detection result R1 and output the processed image to the display device 6. In this case, for example, an image in which the biopsy region is marked on the target image may be displayed.

[0048] The biopsy region detection result R11 shown in Figure 9(a) is an example in which the biopsy region map M2 is classified into two values ​​in step S22, and the biopsy region TA is displayed separately from the rest. The biopsy region detection result R12 shown in Figure 9(b) is an example in which the biopsy region map M2 is classified into three values ​​in step S22, and the first biopsy region TA1, the second biopsy region TA2, and the rest are displayed separately.

[0049] The area detection device 30 performing the area detection process shown in FIG. 7 detects the detection target area by classifying the output results of a machine learning model (inference unit 32) trained using training data with multi-value labels (three or more values, as shown in FIG. 4) into categories equal to or less than the number of possible values ​​of the labels. That is, the area detection device 30 detects the detection target area using a machine learning model that has learned more classifications in the learning stage than the classifications distinguished in the detection stage (including inference by the machine learning model and the subsequent classification process). This allows the detection of the same classification (here, corresponding to the detection target area) with high accuracy, thereby enabling the detection of the detection target area with higher accuracy than conventional methods. Therefore, the area detection device 30 can also contribute to improving the detection accuracy of the detection target area using the machine learning model. Furthermore, displaying the detection results as an image allows users to easily understand the detection results.

[0050] The above-described embodiments are illustrative examples provided to facilitate understanding of the invention. The present invention is not limited to the above-described embodiments and should be understood to encompass various modifications and alternatives of the above-described embodiments. For example, it will be understood that the above-described embodiments can be embodied by modifying the components without departing from the spirit of the invention. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-described embodiments. Furthermore, those skilled in the art will understand that various embodiments can be implemented by deleting some components from all the components shown in the embodiments or by adding some components to the components shown in the embodiments. In other words, the above-described teacher data generation method, area estimator training method, area detection method, program, teacher data generation device, learning device, and area detection device can be variously modified and altered without departing from the scope of the claims.

[0051] In the above-described embodiment, an example was shown in which the number of votes from multiple annotators was directly assigned as a label when generating training data, but the method of assigning labels is not limited to this. When assigning labels, the variable value of each region may be determined based on a weighting calculation for each region according to at least one of the characteristics of the multiple annotators who specify the multiple designated regions, the classification result of the target image for which the multiple designated regions are specified, and the overlap of the multiple designated regions. For example, weighting may be performed based on various concepts such as the following: (1) Weighting according to the number of votes (2) Weighting according to the voter (annotator) (3) Weighting according to voting area (4) Weighting according to image type

[0052] Figure 10 shows an example of weighting according to the number of votes. In the example in Figure 10, the label value of an area that is unanimously determined to be a biopsy area is increased by one point to "4." In other words, the label value is calculated by changing the weighting when the number of votes is 0, 1, or 2 and when the number of votes is 3. According to this example, areas that are unanimously determined to be biopsy areas through learning are evaluated more highly than ever before. By maintaining the threshold used for classification, it is possible to further reduce the number of missed detections of areas that are unanimously determined to be biopsy areas during inference. This is expected to improve the accuracy of biopsy area detection.

[0053] FIG. 11 shows an example of weighting according to the voter. In the example of FIG. 11, the vote of the annotator who specified the designated area SA3 is weighted twice as much as the votes of other annotators to calculate the label value. According to this example, it is possible to assign labels by giving more weight to the opinion of a doctor who is familiar with the detection target (for example, a specialist or veteran doctor in that detection target) than to other doctors. In this case, too, it is expected that the detection accuracy of the biopsy region will be improved.

[0054] Another example of weighting according to voters is shown in Figure 12. In the example of Figure 12, the label value is calculated by excluding the votes of the annotator who specified the designated area SA4 that is most different from the designated areas specified by the other annotators (for example, the area of ​​the overlapping area is the smallest). This example has a high possibility of eliminating evaluations by inexperienced annotators or non-specialist annotators, and can improve detection accuracy by reducing sources of error.

[0055] Note that while Figure 12 shows an example in which annotators to be excluded from voting are selected on an image-by-image basis, annotators to be excluded from voting may also be selected based on votes for multiple images (designation of designated areas).

[0056] FIG. 13 shows an example of weighting according to the vote area. In the example of FIG. 13, when the area of ​​the partitioned region for each vote number is small, the weighting is increased to calculate the label value. This example makes it possible to avoid a situation where information on a small region is buried in information on a large region and is not sufficiently learned. This makes it possible to accurately detect even a small lesion as a biopsy region.

[0057] Furthermore, labels may be assigned with different weights depending on the image type. For example, if a pathological diagnosis has been performed on a target image in advance and a cancer grade has been assigned to the target image, the weights may be changed according to that level. This allows the system to learn to avoid overlooking biopsy regions for more severe cancers by setting a higher label value for target images with more severe cancers. Specifically, for example, if the grade of cervical intraepithelial neoplasia is CIN2 or higher, the label value may be calculated by increasing the weight of the region that is unanimously determined to be a biopsy region, as shown in the example of FIG. 10. However, if the grade is lower than CIN2, the label value may be calculated with equal weighting, as shown in the example of FIG. 4.

[0058] Furthermore, the labels calculated in the above-described embodiment are proportional scales with the non-biopsy region as the reference (0), but they may also be regarded as interval scales without a particular reference value, and interval scales may be used as labels. By using these scales as labels, inference by the machine learning model can be treated as a regression problem. Note that the labels may be variables on proportional scales, interval scales, or ordinal scales (e.g., high (2), medium (1), low (0), etc.). In this case, the inference is treated as ordinal regression.

[0059] In the above-described embodiment, an example was shown in which a regression model was used as the machine learning model constituting the inference unit, but the machine learning model may be one that learns without discarding information input by multiple annotators, and the labels of the training data may be one in which data selected from three or more different values ​​representing the degree of likelihood of being a detection target is assigned as a label to each unit region of the image included in the training data. Therefore, a classification model may be used instead of a regression model as the machine learning model, and in that case, for example, cross-entropy error is used as the loss function.

[0060] The classification model may be a U-Net structure, which is an image segmentation model that divides an image into multiple parts by classifying each unit region (e.g., pixel). This classification model may be capable of classifying images into three or more different classes that represent the degree of likelihood of the image being detected. In the case of learning regions designated by three annotators as described above, for example, a three-channel U-Net structure may be adopted for the machine learning model, in which classification results for class A, which is a region where one or more annotators voted (i.e., a region where the number of votes is one or more), are output to a first channel, classification results for class B, which is a region where two or more annotators voted (i.e., a region where the number of votes is two or more), are output to a second channel, and classification results for class C, which is a region where all three annotators voted (i.e., a region where the number of votes is three), are output to a third channel.

[0061] In this case, in the training data generation process shown in Figure 3, vectors in the same number as the number of channels can be generated for each unit region as labels constituting the training data for the machine learning model. Specifically, in step S4, labels can be generated and assigned to each unit region according to the number of votes as follows: Unit region with 0 votes: [0, 0, 0] Unit area with 1 vote: [1, 0, 0] Unit area with 2 votes: [1, 1, 0] Unit area with 3 votes: [1, 1, 1]

[0062] In this example, the first element of the label indicates whether it is class A, the second element of the label indicates whether it is class B, and the third element of the label indicates whether it is class C.

[0063] By generating labels as vectors with the same number of elements as the number of classes (three or more), the multi-label domain problem can be treated as a binary domain problem for each channel. Because each channel of the machine learning model influences the other channels, training these channels simultaneously allows for classification into classes with higher inference accuracy than training each channel's binary domain problem alone. Therefore, even when using a classification model, it is possible to detect target regions (e.g., regions with a vote count of 3) with greater accuracy, just as when using a regression model.

[0064] In order to confirm the improvement in inference accuracy for the above-mentioned multi-label region problem, the inventors of the present application had four oncologists annotate biopsy regions (tumor regions) for 46 images, and generated training data with labels that classify into three classes corresponding to votes of 1 or more, votes of 2 or more, and votes of 3 or more, based on the number of votes from 0 to 4, and training data with labels that classify into two classes corresponding to votes less than 3 and votes of 3 or more, and compared the detection accuracy of machine learning models obtained by learning based on these two types of training data.

[0065] Specifically, we used a neural network with a U-Net structure as the machine learning model, and trained it using two types of training data to detect regions where three or more annotators voted. We then evaluated the degree of agreement between regions where three or more annotators voted and the inferred regions. Evaluation using five-fold cross-validation showed that the average IoU (Intersection over Union) was 0.364 when training with training data containing labels for classification into three classes, and 0.331 when training with training data containing labels for classification into two classes, confirming that performance improves by classifying into three classes. These results suggest that higher accuracy can be achieved by training information that is classified into three or more values ​​at the time of training and then performing binary classification, rather than by training information that is classified into two values ​​at the time of training and then performing binary classification.

[0066] Note that the classification is not limited to this example. For example, a machine learning model may be trained so that an area where only one annotator votes (i.e., an area with one vote) is defined as class A and the classification results for class A are output on the first channel, an area where two annotators vote (i.e., an area with two votes) is defined as class B and the classification results for class B are output on the second channel, and an area where all three annotators vote (i.e., an area with three votes) is defined as class C and the classification results for class C are output on the third channel.

[0067] In this case, the labels can be generated as follows in step S4 of the training data generation process shown in Fig. 3. In this case too, the first element of the label indicates whether it is class A, the second element of the label indicates whether it is class B, and the third element of the label indicates whether it is class C. Unit region with 0 votes: [0, 0, 0] Unit area with 1 vote: [1, 0, 0] Unit domain with 2 votes: [0, 1, 0] Unit area with 3 votes: [0, 0, 1]

[0068] A new class X representing the region with 0 votes may be defined, and a machine learning model employing a 4-channel U-Net structure may be adopted. In this case, in step S4 of the training data generation process shown in Figure 3, a so-called one-hot vector may be generated as follows: In this example, the first, second, third, and fourth elements correspond to class X, class A, class B, and class C, respectively. Unit region with 0 votes: [1, 0, 0, 0] Unit area with 1 vote: [0, 1, 0, 0] Unit area with 2 votes: [0, 0, 1, 0] Unit area with 3 votes: [0, 0, 0, 1]

[0069] Furthermore, in the above-described embodiment, the system 1 includes the teacher data generation device 10, the learning device 20, and the area detection device 30. However, the system 1 may include these devices independently or within a single device. For example, the teacher data generation device 10 and the learning device 20 may be a single device, with the area detection device 30 being a different device. Alternatively, the learning device 20 and the area detection device 30 may be a single device, with the teacher data generation device 10 being a different device. Note that the area detection device 30 preferably includes an imaging device. This allows the device to perform inference on captured images and detect the area of ​​the detection target. Furthermore, as shown in FIG. 14, the teacher data generation device 10, the learning device 20, and the area detection device 30 may be a device separate from the information processing device 100 including the camera 2 that captures the target image, or may be a server 200 that receives and processes the target image via a network such as the Internet. In other words, results obtained by various processes (such as teacher data generation processing, learning processing, and area detection processing) performed on the server 200 may be received and displayed on a client terminal used by a user. [Explanation of symbols]

[0070] 10: Teacher data generation device, 11: Acquisition unit, 12: Allocation unit, 13: Storage unit, SA1 to SA4: Designated area, T1, T3: Target image, T2: Label

Claims

1. an acquisition unit that acquires a plurality of designated regions that are designated as detection target regions in a target image by a plurality of annotators; an assignment unit that assigns, as a label, data selected from three or more different data values ​​that represent a degree of likelihood of each region in the target image based on the plurality of designated regions; a generation unit that generates training data for a region estimator that estimates a region of a detection target by associating the target image with the label assigned to each region of the target image. Teacher data generation device.

2. 2. The teacher data generation device according to claim 1, The allocating unit allocates the label based on the number of designated areas including the area in question among the plurality of designated areas. Teacher data generation device.

3. 3. The teacher data generation device according to claim 2, The assignment unit assigns the labels based on a weighting calculation for each of the regions according to at least one of characteristics of the plurality of annotators who designate the plurality of designated regions, a classification result of the target image for which the plurality of designated regions are designated, and an overlap of the plurality of designated regions. Teacher data generation device.

4. 2. The teacher data generation device according to claim 1, The data selected from the three or more values ​​of the data is a vector that indicates the classification result into three or more different classes that represent the degree of likelihood of the detection target, and has the same number of elements as the number of the three or more classes. Teacher data generation device.

5. 2. The teacher data generation device according to claim 1, The data selected from the three or more value data is a variable value of an ordinal scale, an interval scale, or a proportional scale selected from three or more different values ​​that represent the degree of likelihood of the detection target. Teacher data generation device.

6. an input unit that inputs a target image included in the training data generated by the training data generation device according to claim 1 to a region estimator that estimates a region of a detection target; an adjustment unit that adjusts parameters of the region estimator based on a comparison result between a value corresponding to each region in the target image output from the region estimator and a label value corresponding to each region included in the training data. Learning device.

7. an input unit for inputting a target image to the region estimator trained by the learning device according to claim 6; an output unit that classifies the values ​​corresponding to each region in the target image output from the region estimator by the label, and outputs the result of classifying the values ​​corresponding to each region in the target image as a detection result of the region of the detection target detected from the target image. Area detection device.

8. an input unit for inputting a target image to the region estimator trained by the learning device according to claim 6; a classification unit that classifies values ​​corresponding to each region in the target image output from the region estimator into a number of types that is less than the number of possible values ​​of label values ​​included in training data for the region estimator, based on a predetermined threshold; an output unit that outputs the result of classifying the values ​​corresponding to each region in the target image as a detection result of the region of the detection target detected from the target image. Area detection device.

9. 8. The area detection device according to claim 7, The output unit displays the classification result as an image. Area detection device.

10. The area detection device according to claim 9, further comprising: an imaging device for capturing the target image; An area detection device characterized by:

11. On the computer, A process of acquiring a plurality of designated regions that each of a plurality of annotators has designated as a detection target region in a target image; a process of assigning data selected from three or more different values ​​representing a degree of likelihood of each region in the target image as a label based on the plurality of designated regions; and generating training data for a region estimator that estimates a region of a detection target by associating the target image with the label assigned to each region of the target image. program.

12. On the computer, a process of inputting a target image included in the training data generated by the training data generation device according to claim 1 to an area estimator that estimates an area of ​​a detection target; and adjusting parameters of the region estimator based on a comparison result between a value corresponding to each region in the target image output from the region estimator and a label value corresponding to each region included in the training data. program.

13. On the computer, a process of inputting a target image to a region estimator trained by the learning device according to claim 6; and a process of classifying the values ​​corresponding to each region in the target image output from the region estimator by the label, and outputting the result of classifying the values ​​corresponding to each region in the target image as a detection result of the region of the detection target detected from the target image. program.

14. A plurality of designated regions that each of a plurality of annotators has designated as a detection target region in a target image are acquired; assigning data selected from three or more different values ​​representing a degree of likelihood of being a detection target to each region in the target image based on the plurality of designated regions; By associating the target image with the labels assigned to each of the regions of the target image, training data for a region estimator that estimates the region of the detection target is generated. How to generate training data.

15. a target image included in the training data generated by the training data generation device according to claim 1 is input to an area estimator that estimates an area of ​​a detection target; The parameters of the region estimator are adjusted based on a comparison result between a value corresponding to each region in the target image output from the region estimator and a label value corresponding to each region included in the training data. How to train a region estimator.

16. a target image is input to a region estimator trained by the learning device according to claim 6; The values ​​corresponding to each region in the target image output from the region estimator are classified by the label, and the classified values ​​corresponding to each region in the target image are output as a detection result of the region of the detection target detected from the target image. Region detection methods.

Images