Machine learning model creation support device, operating method for machine learning model creation support device, and operating program for machine learning model creation support device

Abstract

This machine-learning model creation assistance device comprises a processor which: acquires a plurality of pieces of annotation information generated by a plurality of annotators adding a plurality of labels corresponding to a plurality of classes to areas of the same medical images; derives, for each class, community data that indicates the community of label addition methods by means of the plurality of annotators with respect to the plurality of pieces of annotation information; and generates decision annotation information to be used as correct answer data for a machine-learning model on the basis of the community data and a preset decision condition.

Description

[Technical Field]

[0001] The technology disclosed herein relates to a machine learning model creation support device, an operating method for a machine learning model creation support device, and an operating program for a machine learning model creation support device. [Background technology]

[0002] For example, machine learning models have been developed to recognize objects in medical images, such as recognizing tumors pixel by pixel in abdominal tomographic images taken with a CT (Computed Tomography) device. Such machine learning models require annotation information as ground truth data during the learning phase or accuracy evaluation phase. Annotation information is information generated by assigning a label corresponding to the class of the object to be recognized to the original image that is paired with the ground truth data. In the example of the abdominal tomographic image above, the annotation information is information generated by assigning the label "tumor" to the pixels of the tumor in the original abdominal tomographic image.

[0003] WO 2019 / 003485 describes allocating the generation of annotation information to multiple annotators. Specifically, the same original image is transmitted to each of the annotator terminals of multiple annotators, and multiple pieces of annotation information generated by the multiple annotators assigning labels to the original image are received from the multiple annotator terminals. Summary of the Invention

[0004] One embodiment of the technology disclosed herein provides a machine learning model creation support device, an operating method for the machine learning model creation support device, and an operating program for the machine learning model creation support device, which can easily obtain appropriate annotation information to be used as correct answer data for a machine learning model. [Means for solving the problem]

[0005] The machine learning model creation support device of the present disclosure includes a processor that acquires multiple pieces of annotation information generated by multiple annotators assigning multiple labels corresponding to multiple classes to areas of the same medical image, derives commonality data for each class for the multiple pieces of annotation information that indicates the commonality in how the multiple annotators assigned the labels, and generates confirmed annotation information to be used as correct answer data for the machine learning model based on the commonality data and predetermined confirmation conditions.

[0006] The operating method of the machine learning model creation support device disclosed herein includes acquiring multiple pieces of annotation information generated by multiple annotators assigning multiple labels corresponding to multiple classes to areas of the same medical image, deriving commonality data for each class for the multiple pieces of annotation information that indicates the commonality in how the multiple annotators assigned the labels, and generating confirmed annotation information to be used as correct answer data for the machine learning model based on the commonality data and predetermined confirmation conditions.

[0007] The operating program of the machine learning model creation support device disclosed herein causes a computer to perform processes including acquiring multiple pieces of annotation information generated by multiple annotators assigning multiple labels corresponding to multiple classes to areas of the same medical image, deriving commonality data for each class for the multiple pieces of annotation information that indicates the commonality in how the multiple annotators assigned the labels, and generating confirmed annotation information to be used as correct answer data for the machine learning model based on the commonality data and predetermined confirmation conditions. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a machine learning model creation support system. [Figure 2] FIG. 10 is a diagram showing medical images and annotation information transmitted and received between a machine learning model creation support server and an annotator terminal. [Figure 3]FIG. 10 is a diagram showing annotation information. [Figure 4] FIG. 1 is a block diagram showing a computer that constitutes a machine learning model creation support server. [Figure 5] FIG. 2 is a block diagram showing a processing unit of a processor of the machine learning model creation support server. [Figure 6] FIG. 10 is a diagram illustrating a determination condition and the processing of a derivation unit and a generation unit. [Figure 7] 10 is a flowchart showing a processing procedure of the machine learning model creation support server. [Figure 8] FIG. 10 is a diagram showing annotation information in which different class labels are assigned to the same region. [Figure 9] 10A and 10B are diagrams illustrating another example of a determination condition and the processing of the derivation unit and the generation unit. [Figure 10] 10A and 10B are diagrams illustrating another example of commonality data, determination conditions, and processing by the derivation unit and the generation unit. [Figure 11] FIG. 10 is a diagram showing a fourth embodiment in which display condition data is attached to a medical image. [Figure 12] FIG. 10 is a diagram showing an annotation information generation screen displayed on the display of the annotator terminal. [Figure 13] FIG. 10 is a diagram showing the annotation information generation screen when the generation end button is operated. [Figure 14] FIG. 10 is a diagram showing a fifth embodiment in which a human body region in which a human body is photographed is detected in a medical image, and commonality data is derived only for the detected human body region. [Figure 15] FIG. 10 is a diagram showing an annotation information generation screen in which a human body region is shaded. [Figure 16] FIG. 10 is a diagram showing annotator information. [Figure 17] FIG. 13 is a diagram showing a seventh embodiment in which different determination conditions are set for the center and periphery of a class region. [Figure 18] FIG. 13 is a diagram illustrating an eighth embodiment in which, in the definitive annotation information, the reliability indication value of the label on the periphery of the class area is set lower than that of the center of the class area. [Figure 19]FIG. 10 is a diagram showing how to assign labels when the class is blood vessels. [Figure 20] FIG. 13 is a diagram showing a ninth embodiment in which annotation information and finalized annotation information are transmitted to an annotator terminal. [Figure 21] FIG. 10 is a diagram showing an information comparison screen displayed on the display of the annotator terminal. DETAILED DESCRIPTION OF THE INVENTION

[0009] [First embodiment] 1, a machine learning model creation support system 2 includes a machine learning model creation support server (hereinafter abbreviated as support server) 10 and annotator terminals 11A, 11B, and 11C. Support server 10 and annotator terminals 11A to 11C are connected to each other so as to be able to communicate with each other via network 12. Network 12 is, for example, the Internet or a WAN (Wide Area Network).

[0010] The support server 10 is, for example, a server computer or a workstation, and is an example of a "machine learning model creation support device" according to the technology of the present disclosure. The annotator terminal 11A has a display 13A and an input device 14A, the annotator terminal 11B has a display 13B and an input device 14B, and the annotator terminal 11C has a display 13C and an input device 14C. The annotator terminal 11A is operated by an annotator 15A, the annotator terminal 11B is operated by an annotator 15B, and the annotator terminal 11C is operated by an annotator 15C. The annotator terminals 11A to 11C are, for example, personal computers, tablet terminals, etc. The annotators 15A to 15C are, for example, doctors, and are requested by the support server 10 to generate annotation information 21 (see FIG. 2). Note that, unless there is a need to particularly distinguish between them, the annotator terminals 11A to 11C will be collectively referred to as the annotator terminals 11. Similarly, the displays 13A to 13C, the input devices 14A to 14C, and the annotators 15A to 15C may be collectively referred to as the display 13, the input device 14, and the annotator 15. The input device 14 is, for example, at least one of a keyboard, a mouse, a touch panel, a microphone, and a gesture recognition device.

[0011] As an example, as shown in FIG. 2, the support server 10 transmits the same medical image 20 to the annotator terminals 11A to 11C. Here, an axial abdominal tomographic image captured by a CT device is shown as an example of the medical image 20. The medical image 20 is an original image for assigning a label according to a class, which is a subject to be recognized based on a preset task. Note that the same medical image 20 refers to medical images 20 captured using the same medical imaging device (also called a modality), such as a CT device, the same patient, and the same capture date and time.

[0012] The annotator terminal 11 displays a medical image 20 on a display 13. The annotator terminal 11 receives input of labeling of the medical image 20 on a pixel-by-pixel basis from the annotator 15 via the input device 14. In this manner, annotation information 21A is generated by the annotator 15A in the annotator terminal 11A, annotation information 21B is generated by the annotator 15B in the annotator terminal 11B, and annotation information 21C is generated by the annotator 15C in the annotator terminal 11C. Note that, like the annotator terminals 11A to 11C, the annotation information 21A to 21C may be collectively referred to as annotation information 21.

[0013] In this example, the medical image 20 is an abdominal tomographic image, and therefore the annotation information 21 is generated for each slice of the abdominal tomographic image. In Fig. 2, the human body structure is depicted in the annotation information 21 to facilitate understanding, but the actual annotation information 21 does not include data on the human body structure, but only data on the assigned labels (the same applies to Fig. 3 and the like below). More specifically, the annotation information 21 is information in which pairs of label types and position coordinates of pixels in the medical image 20 to which the labels are assigned are registered.

[0014] The annotator terminal 11 transmits the annotation information 21 to the support server 10. The support server 10 receives the annotation information 21 from the annotator terminal 11.

[0015] The three classes set in the task in this example are liver, tumor in the liver, and bleeding site in the tumor. Therefore, as an example, as shown in FIG. 3, the annotation information 21 includes a first region 25 labeled as liver, a second region 26 labeled as tumor, and a third region 27 labeled as bleeding. Note that the second region 26 is not designated if the annotator 15 determines that no tumor is present. Similarly, the third region 27 is not designated if the annotator 15 determines that no bleeding site is present.

[0016] 4, the computer constituting the support server 10 includes a storage 30, a memory 31, a processor 32, a communication unit 33, a display 34, and an input device 35. These are interconnected via a bus line 36.

[0017] The storage 30 is a hard disk drive built into a computer constituting the support server 10 or connected via a cable or network. Alternatively, the storage 30 is a disk array consisting of multiple hard disk drives. The storage 30 stores control programs such as an operating system, various application programs (hereinafter abbreviated as APs (Application Programs)), and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0018] The memory 31 is a work memory for the processor 32 to execute processing. The memory 31 is, for example, a random access memory (RAM) such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The processor 32 loads a program stored in the storage 30 into the memory 31 and executes processing according to the program. In this way, the processor 32 comprehensively controls each part of the computer. The processor 32 is, for example, a central processing unit (CPU). The memory 31 is an example of a "memory" according to the technology of the present disclosure. The storage 30, or the storage 30 and the memory 31, may be defined as an example of a "memory" according to the technology of the present disclosure.

[0019] The communication unit 33 is a network interface that controls the transmission of various information via the network 12, etc. The display 34 displays various screens. The various screens are provided with operation functions using a GUI (Graphical User Interface). The computer that constitutes the support server 10 accepts input of operation instructions from an input device 35 via the various screens. The input device 35 is at least one of a keyboard, a mouse, a touch panel, a microphone, a gesture recognition device, etc.

[0020] As an example, as shown in FIG. 5, an operating program 40 is stored in the storage 30. The operating program 40 is an AP for causing a computer constituting the support server 10 to function as a "machine learning model creation support device" according to the technology of the present disclosure. In other words, the operating program 40 is an example of an "operating program for a machine learning model creation support device" according to the technology of the present disclosure. In addition to the operating program 40, the storage 30 also stores a medical image 20, annotation information 21, a determination condition 41, and determined annotation information 42. Note that although only one medical image 20 is depicted, in reality, multiple medical images 20 are stored in the storage 30. The same applies to the annotation information 21 and the determined annotation information 42.

[0021] When the operating program 40 is started, the processor 32 works in cooperation with the memory 31, etc. to function as a read / write (hereinafter abbreviated as RW (Read Write)) control unit 50, an image transmission unit 51, an information receiving unit 52, a derivation unit 53, and a generation unit 54.

[0022] The RW control unit 50 controls the storage of various information in the storage 30 and the reading of various information from the storage 30. For example, the RW control unit 50 reads the medical image 20 from the storage 30 and outputs the read medical image 20 to the image transmission unit 51. The RW control unit 50 also reads the determination condition 41 from the storage 30 and outputs the read determination condition 41 to the generation unit 54.

[0023] Information about the annotator terminal 11 that transmits the medical image 20 is registered in advance in the storage 30. The image transmission unit 51 transmits the medical image 20 from the RW control unit 50 to the annotator terminal 11 that has been registered in advance.

[0024] The information receiving unit 52 receives the annotation information 21 from the annotator terminal 11. As a result, the support server 10 acquires the annotation information 21. The information receiving unit 52 outputs the received annotation information 21 to the RW control unit 50. The RW control unit 50 stores the annotation information 21 in the storage 30. Note that although FIG. 5 shows the information receiving unit 52 receiving the annotation information 21A to 21C simultaneously, in reality the annotation information 21A to 21C are received at different times. The information receiving unit 52 outputs the annotation information 21 to the RW control unit 50 every time it receives the annotation information 21, and the RW control unit 50 stores the annotation information 21 in the storage 30 every time the annotation information 21 is input from the information receiving unit 52.

[0025] When the annotation information 21A to 21C is stored in the storage 30, the RW control unit 50 reads the annotation information 21A to 21C from the storage 30, and outputs the read annotation information 21A to 21C to the derivation unit 53 and the generation unit .

[0026] The derivation unit 53 derives commonality data 60. The commonality data 60 is data indicating commonality in the way the three annotators 15A to 15C assign labels to the three pieces of annotation information 21A to 21C. The derivation unit 53 outputs the derived commonality data 60 to the generation unit .

[0027] The generation unit 54 generates the finalized annotation information 42 based on the commonality data 60 from the derivation unit 53, and the annotation information 21A to 21C and the finalization condition 41 from the RW control unit 50. The finalized annotation information 42 is annotation information that is ultimately used as correct answer data for the machine learning model. The generation unit 54 outputs the finalized annotation information 42 to the RW control unit 50. The RW control unit 50 stores the finalized annotation information 42 from the generation unit 54 in the storage 30.

[0028] The confirmed annotation information 42 is used as correct answer data together with the original medical image 20 in the learning phase or accuracy evaluation phase of the machine learning model. In the learning phase, the medical image 20 is input to the machine learning model. Next, the output annotation information output from the machine learning model is compared with the confirmed annotation information 42, and the loss of the machine learning model is calculated. The machine learning model is then updated according to the loss. The smaller the difference between the output annotation information and the confirmed annotation information 42, the smaller the loss. Therefore, the smaller the difference between the output annotation information and the confirmed annotation information 42, the smaller the degree of updating. The input of the medical image 20, the output of the output annotation information, the calculation of the loss, and the updating are repeated while exchanging pairs of the medical image 20 and the confirmed annotation information 42, i.e., correct answer data. In this way, the machine learning model is trained.

[0029] In the accuracy evaluation phase, a medical image 20 is input to a machine learning model that has undergone a certain amount of learning. Then, the output annotation information output from the machine learning model is compared with the final annotation information 42 to calculate a loss, and the accuracy of the machine learning model is evaluated based on the loss. In the accuracy evaluation phase, the accuracy is simply evaluated, and no updates are performed. A machine learning model that is determined to have an accuracy equal to or higher than a predetermined accuracy in this accuracy evaluation phase is used in the practical phase. Note that the correct answer data used in the learning phase is also referred to as learning data, and the correct answer data used in the accuracy evaluation phase is also referred to as evaluation data.

[0030] 6, first region 25A is a region labeled as liver in annotation information 21A, first region 25B is a region labeled as liver in annotation information 21B, and first region 25C is a region labeled as liver in annotation information 21C. Also, second region 26A is a region labeled as tumor in annotation information 21A, second region 26B is a region labeled as tumor in annotation information 21B, and second region 26C is a region labeled as tumor in annotation information 21C. Furthermore, third region 27A is a region labeled as bleeding in annotation information 21A, third region 27B is a region labeled as bleeding in annotation information 21B, and third region 27C is a region labeled as bleeding in annotation information 21C.

[0031] The derivation unit 53 counts the number of annotators 15 who assigned each of the labels liver, tumor, and hemorrhage, and sets the counted number of people as the numerical value of the commonality data 60. The numerical value of the commonality data 60 for a pixel labeled by one annotator 15 is 1, and the numerical value of the commonality data 60 for a pixel labeled by two annotators 15 is 2. Furthermore, the numerical value of the commonality data 60 for a pixel labeled by three annotators 15 is 3. The derivation unit 53 derives the commonality data 60 for each of the classes liver, tumor, and hemorrhage. The number of annotators 15 who assigned labels is an example of the "numerical value related to the number of annotators who assigned labels" according to the technology of the present disclosure.

[0032] The confirmation condition 41 in this embodiment is to adopt the labels of regions where the numerical value of the commonality data 60 is 3, that is, regions that have been labeled by all the annotators 15. Therefore, the generation unit 54 generates confirmed annotation information 42 including a first region 25X labeled as liver by three annotators 15, a second region 26X labeled as tumor by three annotators 15, and a third region 27X labeled as bleeding by three annotators 15.

[0033] Next, the operation of the above configuration will be described with reference to the flowchart in Fig. 7. When the operating program 40 is started, the processor 32 of the support server 10 functions as an RW control unit 50, an image transmission unit 51, an information reception unit 52, a derivation unit 53, and a generation unit 54, as shown in Fig. 5.

[0034] First, the medical image 20 is read out from the storage 30 by the RW control unit 50. The read out medical image 20 is output from the RW control unit 50 to the image transmission unit 51. The medical image 20 is transmitted to the annotator terminal 11 by the image transmission unit 51.

[0035] In the annotator terminal 11, the annotator 15 generates annotation information 21 based on the medical image 20. As shown in FIG. 3, the annotation information 21 is generated by assigning three labels corresponding to three classes, namely, liver, tumor, and bleeding, to regions of the same medical image 20. The annotation information 21 is transmitted from the annotator terminal 11 to the support server 10.

[0036] In the support server 10, the information receiving unit 52 receives the annotation information 21 from the annotator terminal 11. As a result, the annotation information 21 is acquired (step ST100). The annotation information 21 is output from the information receiving unit 52 to the RW control unit 50, and is stored in the storage 30 by the RW control unit 50.

[0037] The RW control unit 50 reads the annotation information 21 from the storage 30. The read annotation information 21 is output from the RW control unit 50 to the derivation unit 53 and the generation unit 54.

[0038] 6, the derivation unit 53 derives commonality data 60 indicating commonality in how the multiple annotators 15A to 15C assign labels for each class of the multiple pieces of annotation information 21A to 21C (step ST110). The commonality data 60 is a count value of the number of annotators 15 who assigned the labels liver, tumor, and bleeding, respectively. The commonality data 60 is output from the derivation unit 53 to the generation unit 54.

[0039] 6, the generation unit 54 generates the finalized annotation information 42 based on the commonality data 60 and the finalization conditions 41 (step ST120). In this example, the finalized annotation information 42 includes a first region 25X, a second region 26X, and a third region 27X to which the three annotators 15 have assigned the labels of liver, tumor, and hemorrhage, respectively. The finalized annotation information 42 is output from the generation unit 54 to the RW control unit 50, and is stored in the storage 30 by the RW control unit 50.

[0040] As described above, the processor 32 of the support server 10 includes an information receiving unit 52, a derivation unit 53, and a generation unit 54. The information receiving unit 52 acquires the annotation information 21A to 21C by receiving it from the annotator terminals 11A to 11C. The annotation information 21A to 21C is generated by the annotators 15A to 15C assigning three labels corresponding to three classes to regions of the same medical image 20. The derivation unit 53 derives commonality data 60 for each class, which indicates the commonality of how the annotators 15A to 15C assigned the labels, for the annotation information 21A to 21C. The generation unit 54 generates final annotation information 42 to be used as correct answer data for the machine learning model, based on the commonality data 60 and a preset determination condition 41.

[0041] When the generation of annotation information 21 for the same medical image 20 is shared among multiple annotators 15, differences arise among the multiple pieces of annotation information 21 because the annotators 15 assign labels differently. In the technology disclosed herein, annotation information 21 is generated by assigning multiple labels corresponding to multiple classes, and therefore the differences among the multiple pieces of annotation information 21 become greater as the number of labels assigned increases.

[0042] Therefore, in the technology disclosed herein, commonality data 60 is derived for each class, and definitive annotation information 42 is generated based on the derived commonality data 60. Therefore, it is possible to easily obtain appropriate definitive annotation information 42 to be used as ground truth data for a machine learning model.

[0043] [Second embodiment] 8, the annotation information 63 of this embodiment includes a first region 65 labeled with a liver, a second region 66 labeled with liver and tumor, and a third region 67 labeled with liver, tumor, and bleeding. The second region 66 and the third region 67 are regions labeled with different classes.

[0044] As described above, in the second embodiment, the annotation information 63 is information in which labels of different classes are assigned to the same region. Therefore, the number of types of labels assigned to the same region increases, and the differences between the multiple pieces of annotation information 21 become even greater compared to the first embodiment. Therefore, the effect of easily obtaining appropriate final annotation information 42 to be used as ground truth data for a machine learning model can be further enhanced.

[0045] [3_1 embodiment] 9, the determination condition 70 of this embodiment specifies that a label be adopted for a region where the commonality data 60 has a value of 2 or more, i.e., a region where two or more annotators have assigned the label. Therefore, the generation unit 54 generates the determined annotation information 42 including a first region 25Y labeled as liver by two or more annotators 15, a second region (not shown) labeled as tumor by two or more annotators 15, and a third region (not shown) labeled as hemorrhage by two or more annotators 15. The "2" to "two" in the determination condition 70 are examples of a "threshold" according to the technology of the present disclosure.

[0046] [Third embodiment] As an example, as shown in FIG. 10 , the derivation unit 53 (not shown) of this embodiment derives commonality data 75 in which the proportion of annotators 15 who assigned a label is registered for each position coordinate of a pixel to which the label is assigned. The derivation unit 53 derives commonality data 75 in which the proportion of annotators who assigned a liver label, as shown in the figure, is registered. The derivation unit 53 also derives commonality data 75 in which the proportion of annotators 15 who assigned a tumor label, and commonality data 75 in which the proportion of annotators 15 who assigned a hemorrhage label, as not shown in the figure, is registered. To determine the proportion of annotators 15 who assigned a label, first, as in the first embodiment described above, the number of annotators 15 who assigned a label is counted. Then, the counted number is divided by the total number of annotators 15. For example, if the number of annotators 15 who assigned labels is eight and the total number of annotators is ten, the percentage of annotators 15 who assigned labels is (8 / 10) × 100 = 80%. Note that the percentage of annotators 15 who assigned labels is an example of the "numerical value related to the number of annotators who assigned labels" according to the technology of the present disclosure.

[0047] The determination condition 76 of this embodiment is to adopt a label of an area where the rate of annotators 15 who assigned the label is 90% or more. The generation unit 54 (not shown) of this embodiment determines whether to adopt a label for each position coordinate, as shown in Table 77, based on the commonality data 75 and the determination condition 76. Specifically, the generation unit 54 determines to adopt a label of a position coordinate where the rate of annotators 15 who assigned the label of the commonality data 75 is 90% or more of the determination condition 76. On the other hand, the generation unit 54 determines not to adopt (non-adopt) a label of a position coordinate where the rate of annotators 15 who assigned the label of the commonality data 75 is less than 90% of the determination condition 76. The generation unit 54 generates the final annotation information 42 based on this adoption / rejection result. Note that "90%" of the determination condition 76 is an example of a "threshold" according to the technology of the present disclosure.

[0048] In the case of the first embodiment described above, in which all annotators 15 adopt the labels of the regions to which they have assigned labels, the final annotation information 42 is inevitably influenced by the annotation information 21 to which the regions to which they have assigned labels are relatively small. In contrast, according to the third embodiment and the third embodiment, which use the determination conditions 70 and 76 that adopt the assigned labels only when the numerical value of the commonality data 60 or 75 is equal to or greater than a threshold, it is possible to generate the final annotation information 42 in which labels are assigned to a wider region, and which is not significantly influenced by the annotation information 21 to which the regions to which they have assigned labels are relatively small.

[0049] As can be seen from the description of the third embodiment, the number of annotators 15 may be two or more and is not limited to three. Therefore, the number of annotation information 21 may also be two or more and is not limited to three. Furthermore, although the number of annotators 15 and the determination conditions 41, 70, and 76 are fixed, this is not limiting. The number of annotators 15 and the determination conditions 41, 70, and 76 may be variable. For example, the support server 10 may be configured so that a user operating the support server 10 can change the settings of the number of annotators 15 and the determination conditions 41, 70, and 76.

[0050] [Fourth embodiment] As an example, as shown in FIG. 11, the image transmission unit 80 of this embodiment attaches display condition data 81 to a medical image 20 and transmits the medical image 20 to the annotator terminals 11A to 11C. The display condition data 81 includes a window level (WL), a window width (WW), and a slice position. The window level and window width are parameters related to the display gradation of the medical image 20. The window level is a central value of the display area of ​​the medical image 20 that is set for the pixel values ​​of the original image of the medical image 20. The window width is a numerical value indicating the width of the display area of ​​the medical image 20. The slice position indicates the position of a tomographic plane when the medical image 20 is a tomographic image as in this example.

[0051] An annotation information generation screen 85, as shown in FIG. 12 as an example, is displayed on the display 13 of the annotator terminal 11. The annotation information generation screen 85 has a task display area 86, a tool button group 87, an image display area 88, etc. The task display area 86 displays the contents of the set task. The tool button group 87 is made up of tool buttons for various tools that allow the annotator 15 to specify labels corresponding to the classes specified in the task. The various tools include, for example, a specified class switching tool, a line drawing tool, an area filling tool, an area erasing tool, etc.

[0052] A medical image 20 is displayed in an image display area 88. Annotation information 21 is generated by adding labels to the medical image 20 displayed in the image display area 88 using various tools. As shown by the dashed-two-dot line enclosure, the medical image 20 is initially displayed under display conditions according to the attached display condition data 81. FIG. 12 shows an example in which the medical image 20 is displayed under display conditions according to the display condition data 81 exemplified in FIG. 11.

[0053] A back button 89 and a display gradation change button 90 are provided below the image display area 88. The slice position can be changed by operating the back button 89. Furthermore, the window level and window width can be changed by operating the display gradation change button 90. In this way, the annotator 15 can freely change the slice position of the medical image 20. Therefore, the annotator 15 can also repeatedly review the medical image 20 at a specific slice position. Furthermore, the annotator 15 can freely change the display conditions of the medical image 20. Although not shown in the drawings, the medical image 20 in the image display area 88 can be translated and enlarged and reduced.

[0054] A temporary save button 91 and a generation end button 92 are also provided at the bottom of the annotation information generation screen 85. When the temporary save button 91 is operated, the annotation information 21 generated up to that point is temporarily saved in the storage of the annotator terminal 11. When the generation end button 92 is operated, a dialog box 95 pops up, as shown in FIG. 13 as an example. Furthermore, as indicated by the dashed-dotted line enclosure, the display conditions of the medical image 20 in the image display area 88 are set to the display conditions in accordance with the attached display condition data 81.

[0055] The dialog box 95 is a GUI for asking the annotator 15 whether or not to really end the generation of the annotation information 21. The dialog box 95 is provided with a Yes button 96 and a No button 97. When the Yes button 96 is operated, the generated annotation information 21 is sent to the support server 10. When the No button 97 is selected, the dialog box 95 is closed, and the state returns to one in which the generation of the annotation information 21 is possible.

[0056] As described above, in the fourth embodiment, the display condition data 81 is attached to the medical image 20. The display condition data 81 is data for displaying the medical image 20 under the same display conditions when multiple annotators 15 view the medical image 20. Therefore, as shown in Fig. 12 and Fig. 13, the display conditions can be made the same in the multiple annotator terminals 11. Therefore, it is possible to prevent differences in the labels assigned by the annotators 15 due to differences in the display conditions.

[0057] 12, by initially displaying the medical image 20 under display conditions in accordance with the display condition data 81, it is possible to prevent discrepancies in the recognition of each annotator 15 from occurring from the start of generation of the annotation information 21. Furthermore, as shown in Fig. 13, by displaying the medical image 20 under display conditions in accordance with the display condition data 81 when generation of the annotation information 21 is completed, it is possible to prevent discrepancies in the recognition of each annotator 15 from occurring at the final confirmation of the annotation information 21.

[0058] In this embodiment, the medical image 20 is displayed under the display conditions according to the display condition data 81 both when the generation of the annotation information 21 is started and when the generation of the annotation information 21 is ended, thereby making the display conditions of the medical image 20 the same, but this is not limiting. The medical image 20 may be displayed under the display conditions according to the display condition data 81 only either when the generation of the annotation information 21 is started or when the generation of the annotation information 21 is ended.

[0059] [Fifth embodiment] As an example, as shown in FIG. 14 , the processor of the support server of this embodiment functions as a detection unit 105 in addition to the processing units 50 to 54 (units other than the derivation unit 53 are not shown) of the first embodiment. The detection unit 105 uses body surface recognition technology to detect a human body region 106 in which a human body is captured in the medical image 20. The detection unit 105 outputs human body region information 107, which is the detection result of the human body region 106, to the derivation unit 53. The human body region information 107 is specifically the position coordinates of pixels in the medical image 20 that correspond to the human body region 106. The derivation unit 53 of this embodiment derives commonality data 60 only for the human body region 106.

[0060] As can be seen from the examples of liver, tumor, and bleeding, the class of the medical image 20 is set only for the human body region 106, so there is no need to derive commonality data 60 for regions other than the human body region 106. Therefore, in the fifth embodiment, the detection unit 105 detects the human body region 106 in which a human body is captured in the medical image 20, and the derivation unit 53 derives the commonality data 60 only for the detected human body region 106. This reduces the processing load on the derivation unit 53, and as a result, the generation of the final annotation information 42 can be accelerated. Note that the commonality data 60 may be replaced by the commonality data 75 of the above-described third embodiment.

[0061] 15, on the annotation information generation screen 85 displayed on the display 13 of the annotator terminal 11, a colored shading 115 indicated by hatching may be displayed on the human body region 106 detected by the detection unit 105 so that the human body region 106 can be distinguished from other regions. This alerts the annotator 15 not to erroneously label regions other than the human body region 106.

[0062] Furthermore, the annotation information generation screen 85 may be configured so that labels cannot be assigned to regions other than the human body region 106. This configuration also makes it possible to prevent labels from being assigned to regions other than the human body region 106 by mistake.

[0063] [Sixth embodiment] 16 as an example, the derivation unit 53 of this embodiment counts the number of annotators 15 when deriving the commonality data 60 according to the annotator information 120. In the annotator information 120, for each annotator ID (Identification Data) for identifying each annotator 15, the attributes of the annotator 15 and the count number of people when deriving the commonality data 60 are registered.

[0064] Attributes include years of service and qualifications. Qualifications include radiology training instructor and diagnostic radiologist. The count is determined according to pre-set rules: +0.5 for years of service of 20 or more, -0.5 for years of service of less than 5 years, and +0.5 for qualified individuals. For example, annotator 15 with annotator ID "AN0001" has 22 years of service and is qualified as a radiology training instructor, so the count is 1 + 0.5 + 0.5 = 2. On the other hand, annotator 15 with annotator ID "AN0101" has 3 years of service and does not have any qualifications, so the count is 1 - 0.5 = 0.5.

[0065] As described above, in the sixth embodiment, when deriving the commonality data 60, weighting is performed according to the attributes of the annotators 15. This makes it possible to increase the number of people in areas to which labels have been assigned by annotators 15 who are considered to have a relatively high level of accuracy in labeling, such as annotators 15 with a relatively long tenure and / or qualified annotators 15. In this way, when the confirmation condition is, as in the confirmation condition 70 of the third embodiment, that labels for areas in which the numerical value of the commonality data 60 is equal to or greater than a threshold are adopted, the probability that labels assigned by annotators 15 who are considered to have a relatively high level of accuracy in labeling will be adopted as labels for the final annotation information 42 increases. As a result, the reliability of the final annotation information 42 can be improved.

[0066] Note that the commonality data 75 of the above-described third embodiment may be used instead of the commonality data 60. In this case, too, the proportion of areas to which labels have been assigned by annotators 15 who are considered to have a relatively high level of accuracy in labeling will be relatively high, and therefore the probability that labels assigned by annotators 15 who are considered to have a relatively high level of accuracy in labeling will be adopted as labels for the final annotation information 42 will increase, and as a result, the reliability of the final annotation information 42 can be improved.

[0067] For example, if there are an even number of annotators 15, and the number of annotators 15 who assigned labels is half and half, and the number of annotators 15 who did not assign labels is half, the label assignment method of the annotator 15 who is thought to have a relatively high level of accuracy in assigning labels may be adopted. Specifically, if there are four annotators 15, two of whom assigned labels and two of whom did not assign labels, and the annotator 15 who is thought to have a relatively high level of accuracy in assigning labels is on the side that did not assign labels, the label will not be adopted as the label of the final annotation information 42.

[0068] The annotator 15's field of expertise may be included in the attributes. In this case, for example, if the task is related to the field of expertise, the count is increased. Also, the number of published papers by the annotator 15 may be included in the attributes. In this case, if the number of published papers is equal to or greater than a first threshold, the count is increased, and if the number of published papers is less than a second threshold, the count is decreased.

[0069] [Seventh embodiment] As an example, as shown in FIG. 17 , the generation unit 54 (not shown) of this embodiment generates the finalized annotation information 42 based on two finalization conditions 125A and 125B. The finalization condition 125A is applied to a center portion 126A of a class region 126. On the other hand, the finalization condition 125B is applied to a peripheral portion 126B of the region 126. The finalization condition 125A specifies that a label of a region where 70% or more of the annotators 15 have assigned a label is to be adopted. On the other hand, the finalization condition 125B specifies that a label of a region where 90% or more of the annotators 15 have assigned a label is to be adopted. Note that the “70%” of the finalization condition 125A and the “90%” of the finalization condition 125B are examples of “thresholds” according to the technology of the present disclosure. In other words, the threshold of the finalization condition 125B applied to the peripheral portion 126B is set higher than the threshold of the finalization condition 125A applied to the center portion 126A.

[0070] The central portion 126A and the peripheral portion 126B are selected, for example, as follows. First, the regions labeled by all the annotators 15 are determined, and the centers of the determined regions are set as the center of the central portion 126A. Note that the center is, for example, at least one of the centroid, the incenter, the circumcenter, and the orthocenter. Next, the regions labeled by all the annotators 15 are enlarged, for example, by 20% without moving the centers. Then, the regions labeled by all the annotators 15 are set as the central portion 126A, and the region bordered by the 20% enlarged region and the regions labeled by all the annotators 15 is set as the peripheral portion 126B. Alternatively, the centers of the regions labeled by each annotator 15 may be determined, and the centers of the determined centers may be further determined and set as the center of the central portion 126A. Note that, although the peripheral portion is set as the region enlarged by 20% without moving the center of the labeled region, this is not limited to this. For example, if the distance from the center to the outer edge of a labeled region is 100, the distance from the center to 80 may be the central portion, and the remaining distance from 80 to 100 may be the peripheral portion.

[0071] As described above, in the seventh embodiment, a determination condition 125A is applied to the central portion 126A of the class region 126, and a determination condition 125B is applied to the peripheral portion 126B. The determination conditions are different between the central portion 126A and the peripheral portion 126B. The determination condition 125B is more difficult to satisfy than the determination condition 125A. The peripheral portion 126B is a boundary with another region, and therefore is particularly prone to mislabeling. Therefore, by making the condition for the peripheral portion 126B more difficult to satisfy than that for the central portion 126A, the reliability of the confirmed annotation information 42 for the peripheral portion 126B can be ensured.

[0072] [Eighth embodiment] 18 as an example, the generation unit 54 (not shown) of this embodiment sets a numerical value (hereinafter referred to as reliability display value) representing the reliability of the label of the peripheral portion 131B in the final annotation information 130 lower than that of the central portion 131A of the class area 131. Specifically, the generation unit 54 sets the reliability display value of the central portion 131A to the maximum value of 1, and sets the reliability display value of the peripheral portion 131B to 0.8. The central portion 131A and the peripheral portion 131B are also selected in the same manner as the central portion 126A and the peripheral portion 126B in the seventh embodiment.

[0073] As described above, in the eighth embodiment, the generation unit 54 sets a lower reliability display value for the label of the peripheral portion 131B of the class region 131 than for the central portion 131A of the class region 131 in the definitive annotation information 130. Therefore, when training a machine learning model using the definitive annotation information 130, it is possible to reduce the frequency of so-called false positives, in which a region that is not a class is recognized as a class, particularly in the early stages of the training phase.

[0074] The reliability display value of the label in the final annotation information may be set according to the ratio of the annotators 15 who assigned the label, as explained in the above-mentioned embodiment 3_2. For example, the ratio of the annotators 15 who assigned the label is used as the reliability display value. Specifically, if the ratio is 80%, the reliability display value is set to 0.8, and if the ratio is 20%, the reliability display value is set to 0.2.

[0075] As an example, as shown in FIG. 19 , when the class is blood vessel, the generation unit 54 preferably assigns a label to an extended region 136 in the final annotation information that is larger than the region 135 that satisfies the finalization condition. The extended region 136 is, for example, a region that is larger by a set number of pixels than the region 135 that satisfies the finalization condition. The set number of pixels is, for example, 1. Since the boundaries of blood vessels are often unclear, there is a demand for a margin to be set so that the surrounding area of ​​the blood vessel can also be recognized as a blood vessel. This demand can be met. The extended region 136 may be an area that encompasses a distance of, for example, 120 from the outer edge of the region 135 that satisfies the finalization condition, assuming that the distance from the center to the outer edge is 100. The center of the region 135 that satisfies the finalization condition may be determined in the same manner as the center of the region to which all the annotators 15 in the seventh embodiment assigned a label.

[0076] [Ninth embodiment] 20 as an example, the processor of the support server of this embodiment functions as an information transmission unit 140 in addition to the processing units 50 to 54 of the first embodiment. The information transmission unit 140 transmits the annotation information 21 and the finalized annotation information 42 stored in the storage 30 to the annotator terminal 11. The annotation information 21 may include not only information generated by the annotator 15 of the annotator terminal 11 that transmits the annotation information, but also information generated by an annotator other than the annotator 15 of the annotator terminal 11 that transmits the annotation information.

[0077] In the annotator terminal 11 that has received the annotation information 21 and the confirmed annotation information 42, an information comparison screen 145 shown in Fig. 21 as an example is displayed on the display 13. The information comparison screen 145 has an annotation information display area 146 in which the annotation information 21 is displayed superimposed on the medical image 20, and a confirmed annotation information display area 147 in which the confirmed annotation information 42 is displayed superimposed on the medical image 20. The annotation information display area 146 and the confirmed annotation information display area 147 are arranged side by side on the left and right.

[0078] The annotation information display area 146 initially displays the annotation information 21 generated by the annotator 15. The medical image 20 superimposed on the annotation information 21 and the medical image 20 superimposed on the finalized annotation information 42 are initially displayed under the same display conditions according to the display condition data 81.

[0079] Figure 21 shows an example in which areas labeled with all of the following labels are displayed: liver, tumor, and bleeding. However, the annotation information 21 and the final annotation information 42 can also be displayed by class, for example, only areas labeled with liver.

[0080] Provided below the annotation information display area 146 and the confirmed annotation information display area 147 are back buttons 148 and 149, as well as display gradation change buttons 150 and 151, which have the same functions as the back button 89 and display gradation change button 90 on the annotation information generation screen 85. Therefore, it is possible to change the slice position, window level, and window width, just as in the case of the annotation information generation screen 85.

[0081] The information comparison screen 145 further includes a display mode switch button 152, a correction button 153, an information switch button 154, and an end button 155 at the bottom. When the display mode switch button 152 is operated, the annotation information 21 in the annotation information display area 146 is moved to the confirmed annotation information display area 147, and the annotation information 21 and the confirmed annotation information 42 are superimposed. In this case, the overlapping portions of the labels of the annotation information 21 and the confirmed annotation information 42 are displayed in a darker color than the non-overlapping portions. When the correction button 153 is operated, the screen transitions to the annotation information generation screen 85, where the annotation information 21 can be corrected. When the information switch button 154 is operated, the display in the annotation information display area 146 is switched from the annotation information 21 generated by the annotator 15 to the annotation information 21 generated by another annotator 15. When the end button 155 is operated, the display of the information comparison screen 145 is cleared.

[0082] As described above, in the ninth embodiment, the information transmitting unit 140 transmits the annotation information 21 and the finalized annotation information 42 to the annotator terminal 11 operated by the annotator 15. Therefore, as shown in FIG. 21 , the annotation information 21 and the finalized annotation information 42 can be displayed on the annotator 15 so that they can be compared. The annotator 15 can compare the annotation information 21 that he or she has generated with the finalized annotation information 42 and use this information to generate future annotation information 21. In addition, in some cases, the annotator 15 can also correct the annotation information 21 while referring to the finalized annotation information 42.

[0083] The display conditions for the medical image 20 superimposed on the annotation information 21 and the medical image 20 superimposed on the final annotation information 42 may be the display conditions that have been set most frequently among the display conditions set by each annotator 15 on the annotation information generation screen 85. Alternatively, the following may be performed: the support server 10 receives the window level and window width at the time when the generation of the annotation information 21 is completed from each annotator terminal 11 along with the annotation information 21. The support server 10 sets the window level and window width attached to the annotation information 21 that is closest to the final annotation information 42, from the received window levels and window widths, as the display conditions for the medical image 20 superimposed on the annotation information 21 and the medical image 20 superimposed on the final annotation information 42.

[0084] In the first embodiment, the medical images 20 are transmitted from the support server 10 to the annotator terminal 11, but this is not limiting. For example, an image management server that stores and manages the medical images 20 may be provided separately from the support server 10, and the medical images 20 may be transmitted from the image management server to the annotator terminal 11.

[0085] The medical image 20 is not limited to an abdominal tomographic image taken by the exemplified CT device. For example, it may be a head tomographic image taken by an MRI (Magnetic Resonance Imaging) device. Furthermore, the medical image is not limited to a three-dimensional image such as a tomographic image. For example, it may be a two-dimensional image such as a simple radiographic image. It may also be a PET (Positron Emission Tomography) image, a SPECT (Single Photon Emission Computed Tomography) image, an endoscopic image, an ultrasound image, an ophthalmoscopic image, or the like.

[0086] The classes to which labels are assigned are not limited to the exemplified liver, tumor, bleeding, and blood vessel. They may also include other organs such as the brain, eyeball, spleen, and kidney, bones such as vertebrae and ribs, anatomical regions of organs such as S1 to S10 of the lung, and the head, body, and tail of the pancreas, as well as other abnormal findings such as cysts, atrophy, ductal stenosis, or ductal dilation. They may also include pacemakers, artificial joints, and bolts for treating fractures.

[0087] The display condition data 81 in the fourth embodiment is set to a value adapted to the type of medical image 20, the organ class, or the patient's body type, etc. If the medical image 20 is a radiological image, the display condition data 81 is further set to a value adapted to the irradiated radiation dose.

[0088] Although labels are assigned to each pixel of the medical image 20, this is not limiting. For example, labels may be assigned to a rectangular frame (when the medical image 20 is a two-dimensional image) or a box frame (when the medical image 20 is a three-dimensional image) that surrounds an entire class such as a tumor. In this case, for example, the label assigned to the area where all the frames overlap is adopted as the label of the final annotation information 42.

[0089] In the above embodiments, the annotator 15 is described as a person such as a doctor, but is not limited to this. The annotator 15 may also be a machine learning model.

[0090] Various screens such as the annotation information generation screen 85 may be transmitted from the support server 10 to the annotator terminal 11 in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). In this case, the annotator terminal 11 reproduces various screens to be displayed on a web browser based on the screen data, and displays them on the display 13. Note that instead of XML, other data description languages ​​such as JSON (Javascript (registered trademark) Object Notation) may be used.

[0091] The hardware configuration of the computer constituting the support server 10 can be modified in various ways. For example, the support server 10 can be configured with multiple server computers separated as hardware in order to improve processing power and reliability. For example, the functions of the RW control unit 50, image transmission unit 51, and information reception unit 52, and the functions of the derivation unit 53 and generation unit 54 can be distributed and performed by two server computers. In this case, the support server 10 is configured with two server computers. Some or all of the functions of each processing unit of the support server 10 may be performed by the annotator terminal 11.

[0092] In this way, the hardware configuration of the computer of the support server 10 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, not only the hardware but also APs such as the operating program 40 can be duplicated or stored in multiple storages in order to ensure safety and reliability.

[0093] In each of the above embodiments, the following various processors can be used as the hardware structure of the processing units that perform various processes, such as the RW control unit 50, the image transmission units 51 and 80, the information reception unit 52, the derivation unit 53, the generation unit 54, the detection unit 105, and the information transmission unit 140. The various processors include a CPU, which is a general-purpose processor that executes software (operation program 40) and functions as various processing units, as well as programmable logic devices (PLDs), which are processors whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and / or dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which are processors having a circuit configuration designed specifically for performing specific processes.

[0094] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0095] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by client and server computers. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by System on Chip (SoC). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0096] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0097] [Appendix 1] The annotation information is preferably information in which different class labels are assigned to the same region.

[0098] [Appendix 2] The commonality data is a numerical value relating to the number of annotators who assigned labels for each of multiple classes, and the confirmation condition is preferably such that the assigned label is adopted only if the numerical value is equal to or greater than a threshold value.

[0099] [Appendix 3] It is preferable that the medical image be accompanied by display condition data for displaying the medical image under the same display conditions when multiple annotators view the medical image.

[0100] [Appendix 4] Preferably, the processor detects a human body region in the medical image where a human body is depicted, and derives commonality data only for the detected human body region.

[0101] [Appendix 5] When deriving the commonality data, the processor preferably assigns weights according to the attributes of the annotators.

[0102] [Appendix 6] The determination conditions are preferably different between the center and the periphery of the class area, and are more difficult to satisfy in the periphery than in the center.

[0103] [Appendix 7] Preferably, the processor sets a lower numerical value representing the reliability of labels at the periphery of a class region than at the center of the class region in the definitive annotation information.

[0104] [Appendix 8] Preferably, the processor transmits the annotation information and the finalized annotation information to an annotator terminal used by the annotator.

[0105] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs but also to storage media that non-temporarily store programs.

[0106] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0107] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0108] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a processor; The processor: Obtaining a plurality of pieces of annotation information generated by a plurality of annotators assigning a plurality of labels corresponding to a plurality of classes to regions of the same medical image, wherein the plurality of pieces of annotation information are assigned labels of different classes in regions where the classes overlap; deriving commonality data for each of the classes that indicates commonality in how the labels are assigned by the plurality of annotators for the plurality of pieces of annotation information; generating final annotation information to be used as correct answer data for a machine learning model based on the commonality data and a predetermined finalization condition; A machine learning model creation support device.

2. the commonality data is a numerical value relating to the number of the annotators who assigned the labels to each of the plurality of classes; The machine learning model creation support device according to claim 1 , wherein the determination condition is that the assigned label is adopted only if the numerical value is equal to or greater than a threshold value.

3. 3. The machine learning model creation support device according to claim 1, wherein the medical image is accompanied by display condition data for displaying the medical image under the same display conditions when a plurality of the annotators view the medical image.

4. The processor: detecting a human body region in which a human body is captured in the medical image; The machine learning model creation support device according to claim 1 , wherein the commonality data is derived only for the detected human body region.

5. The processor: The machine learning model creation support device according to claim 1 , wherein when deriving the commonality data, weighting is performed according to attributes of the annotators.

6. 6. The machine learning model creation support device according to claim 1, wherein the determination conditions are different between a center portion and a periphery portion of the class region, and the degree of difficulty in satisfying the conditions is higher in the periphery portion than in the center portion.

7. The processor: The machine learning model creation support device according to claim 1 , wherein in the definitive annotation information, a numerical value representing the reliability of the label in the peripheral area of ​​the class is set lower than that in the center of the class.

8. The processor: The machine learning model creation support device according to claim 1 , wherein the annotation information and the finalized annotation information are transmitted to an annotator terminal used by the annotator.

9. The processor: The machine learning model creation support device according to claim 1 , wherein an instruction to change the setting of the determination condition is accepted.

10. Obtaining a plurality of pieces of annotation information generated by a plurality of annotators assigning a plurality of labels corresponding to a plurality of classes to regions of the same medical image, wherein the plurality of pieces of annotation information are assigned labels of different classes in regions where the classes overlap; deriving commonality data for each of the classes, the commonality indicating commonality in how the labels are assigned by the plurality of annotators for the plurality of pieces of annotation information; and generating final annotation information to be used as correct answer data for a machine learning model based on the commonality data and predetermined finalization conditions; A method for operating a machine learning model creation support device, comprising:

11. Obtaining a plurality of pieces of annotation information generated by a plurality of annotators assigning a plurality of labels corresponding to a plurality of classes to regions of the same medical image, wherein the plurality of pieces of annotation information are assigned labels of different classes in regions where the classes overlap; deriving commonality data for each of the classes, the commonality indicating commonality in how the labels are assigned by the plurality of annotators for the plurality of pieces of annotation information; and generating final annotation information to be used as correct answer data for a machine learning model based on the commonality data and predetermined finalization conditions; An operating program for a machine learning model creation support device that causes a computer to execute processing including the steps of:

Images