Information processing device, method of operating the information processing device, operating program of the information processing device

Abstract

This information processing device is provided with a processor. The processor acquires information on the history of reference by an annotator to related medical information related to a generation source medical image which is the generation source of annotation information generated by the annotator as correct answer data for a machine learning model for analyzing medical images, and derives evaluation information indicating the quality of the annotation information on the basis of the reference history information.

Description

Technical Field

[0001] The technology of the present disclosure relates to an information processing apparatus, a method of operating the information processing apparatus, and an operation program of the information processing apparatus.

Background Art

[0002] For example, a machine learning model for analyzing medical images has been developed, such as recognizing tumors in abdominal tomographic images taken by a CT (Computed Tomography) apparatus on a pixel-by-pixel basis. In such a machine learning model, annotation information as correct data is required in the learning phase or the accuracy evaluation phase. The annotation information is information generated, for example, by attaching a label corresponding to the class of the subject to be recognized to the original medical image that is paired in the correct data. In the example of the above abdominal tomographic image, the annotation information is information generated by attaching the label "tumor" to the pixels of the tumor in the abdominal tomographic image that is the original medical image.

[0003] Japanese Unexamined Patent Application Publication No. 2020-086519 describes a medical image processing apparatus including an acquisition unit that acquires teacher data (correct data) related to a medical image, and a reliability assignment unit that assigns reliability information based on one or both of the creation status of the teacher data (required creation time, confidence in the attached label, etc.) or information related to the creator who created the teacher data (number of years of experience, qualifications, busyness, etc.) to the teacher data acquired by the acquisition unit.

[0004] Japanese Patent Publication No. 2015-162109 describes a task assignment server that assigns new tasks to experts from a group of workers capable of processing new tasks via a network, comprising: a worker DB (Data Base) that stores worker group information including at least information on tasks previously processed by each worker in the worker group; a cluster creation function unit that classifies each worker in the worker group into one of several clusters based on the worker group information; and a processing result analysis function unit that designates a worker belonging to one of the multiple clusters as an expert based on the processing results of a test task which is part of the new task. [Overview of the project]

[0005] One embodiment of the technology of this disclosure provides an information processing device, a method for operating the information processing device, and an operating program for the information processing device that can more accurately evaluate the quality of annotation information. [Means for solving the problem]

[0006] The information processing device of this disclosure includes a processor, which acquires reference history information of related medical information associated with the source medical image from which the annotation information was generated when the annotator generates annotation information as ground truth data for a machine learning model for analyzing medical images, and derives evaluation information representing the quality of the annotation information based on the reference history information.

[0007] The method of operating the information processing device of this disclosure includes obtaining reference history information by the annotator of related medical information associated with the source medical image from which the annotation information was generated when the annotator generates annotation information as ground truth data for a machine learning model for analyzing medical images, and deriving evaluation information representing the quality of the annotation information based on the reference history information.

[0008] The operating program for the information processing device of this disclosure causes a computer to perform the following processes: obtaining reference history information by the annotator of related medical information associated with the source medical image from which the annotation information was generated when the annotator generates annotation information as ground truth data for a machine learning model for analyzing medical images; and deriving evaluation information representing the quality of the annotation information based on the reference history information. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing an information processing system. [Figure 2] This diagram shows the information transmitted and received between the information processing server and the annotator terminal. [Figure 3] This figure shows the annotation information. [Figure 4] This is a diagram showing the annotation information generation screen. [Figure 5] This figure shows the annotation information generation screen where relevant medical information is displayed. [Figure 6] This figure shows the reference history information based on whether or not the related medical information reference button was operated. [Figure 7] This is a block diagram showing the computers that make up an information processing server. [Figure 8] This is a block diagram showing the processing unit of the processor in an information processing server. [Figure 9] This figure shows evaluation information based on the content of the reference history information. [Figure 10] This is a flowchart showing the processing procedure of the information processing server. [Figure 11] This diagram shows five types of related medical information. [Figure 12] This figure shows a second embodiment in which evaluation information is derived according to the number of related medical information items referenced by the annotator out of the five types of related medical information. [Figure 13] This figure shows how the information receiving unit receives the annotator's operation history information for the source medical image when annotation information is generated. [Figure 14] FIG. 2 is a diagram showing a 3_1st embodiment of deriving evaluation information based on operation history information in addition to reference history information. [Figure 15] FIG. 5 is a diagram showing a state where an information receiving unit receives operation history information of an annotator for a medical image of another device when annotation information is generated. [Figure 16] [[ID=VIII]]FIG. 8 is a flowchart showing a mode of registering an implementation in a display gradation change operation of operation history information only when the display gradation of the recommended range is changed. [Figure 17] FIG. 11 is a flowchart showing a 3_2nd embodiment of determining that an annotator has referred to related medical information when there is any operation on the related medical information by the annotator. [Figure 18] FIG. 14 is a diagram showing a state where an information receiving unit receives attribute information of an annotator from an annotator terminal. [Figure 19] FIG. 17 is a diagram showing a 4th embodiment of deriving evaluation information based on attribute information in addition to reference history information. [Figure 20] FIG. 20 is a block diagram showing a processing unit of a processor of an information processing server according to a 5th embodiment. [Figure 21] FIG. 23 is a diagram showing a correct data selection screen. [Figure 22] FIG. 26 is a diagram showing a 6th embodiment of transmitting overall summary information and individual summary information to an annotator terminal. [Figure 23] FIG. 29 is a diagram showing an evaluation summary display screen. [Figure 24] FIG. 32 is a diagram showing another example of individual summary information. [Figure 25] FIG. 35 is a diagram showing another example of an evaluation summary display screen. [Figure 26] FIG. 38 is a diagram showing a 7th embodiment of transmitting a source medical image, related medical information, and recommended information to an annotator terminal. [Figure 27] FIG. 41 is a diagram showing related medical information including a plurality of types of medical images of another device. [Figure 28] FIG. 44 is a diagram showing related medical information including a plurality of types of medical images of different dates and times.

Best Mode for Carrying Out the Invention

[0010] [First Embodiment] As an example, as shown in FIG. 1, the information processing system 2 includes an information processing server 10 and a plurality of annotator terminals 11. The information processing server 10 and the annotator terminals 11 are connected to be mutually communicable via a network 12. The network 12 is, for example, the Internet or a WAN (Wide Area Network).

[0011] The information processing server 10 is, for example, a server computer, a workstation, etc., and is an example of an "information processing device" according to the technology of the present disclosure. The annotator terminal 11 has a display 13 and an input device 14. The annotator terminal 11 is operated by an annotator 15. The annotator terminal 11 is, for example, a personal computer, a tablet terminal, etc. The annotator 15 is, for example, a doctor and is requested to generate annotation information 23 (see FIG. 2) from the information processing server 10. Note that the input device 14 is at least one of, for example, a keyboard, a mouse, a touch panel, a microphone, and a gesture recognition device.

[0012] As an example, as shown in FIG. 2, the information processing server 10 transmits a source medical image 20 to the annotator terminal 11. Here, the source medical image 20 is exemplified by an axial cross-sectional abdominal tomographic image taken by a CT device. The source medical image 20 is an image for attaching a label according to a class of a subject to be recognized based on a preset task.

[0013] Furthermore, the information processing server 10 transmits related medical information 21 to the annotator terminal 11 along with the source medical image 20. The related medical information 21 is medical information related to the source medical image 20 and includes a medical image from another device 22. The medical image from another device 22 is an image taken on the same day as the source medical image 20, for example, by a different medical imaging device (also called a modality). Here, an axial cross-sectional abdominal tomographic image taken by an MRI (Magnetic Resonance Imaging) device is given as an example of the medical image from another device 22.

[0014] The annotator terminal 11 displays the source medical image 20 on the display 13. The annotator terminal 11 receives input from the annotator 15 via the input device 14 to assign labels to the source medical image 20 at the pixel level. In this way, annotation information 23 is generated by the annotator 15 in the annotator terminal 11.

[0015] In this example, the source medical image 20 is an abdominal tomographic image, so annotation information 23 is generated for each tomographic plane of the abdominal tomographic image. Note that in Figure 2, human body structures are drawn in the annotation information 23 for the sake of understanding, but the actual annotation information 23 does not contain data on human body structures, but only data on the assigned labels (the same applies to Figure 3 and so on). More specifically, annotation information 23 is information that registers a pair of label type and the position coordinates of the pixels in the source medical image 20 to which the label was assigned.

[0016] Furthermore, the annotator terminal 11 displays a medical image 22 of the related medical information 21 on the display 13 in response to instructions from the annotator 15. The annotator terminal 11 generates reference history information 24 indicating whether or not the annotator 15 has referenced the related medical information 21.

[0017] The annotator terminal 11 sends annotation information 23 and reference history information 24 to the information processing server 10. The information processing server 10 receives the annotation information 23 and reference history information 24 from the annotator terminal 11.

[0018] The classes defined for the task in this example are liver, tumor within the liver, and bleeding site within the tumor. Therefore, as shown in Figure 3 as an example, the annotation information 23 includes a first region 25 labeled as liver, a second region 26 labeled as tumor, and a third region 27 labeled as bleeding. Naturally, the second region 26 is not specified if the annotator 15 determines that no tumor exists. Similarly, the third region 27 is not specified if the annotator 15 determines that no bleeding site exists.

[0019] The annotator terminal 11's display 13 shows, for example, an annotation information generation screen 30A, as shown in Figure 4. The annotation information generation screen 30A includes a task display area 31, a group of tool buttons 32, and a source medical image display area 33, etc. The task display area 31 displays the contents of the set task. The group of tool buttons 32 consists of tool buttons for various tools that allow the annotator 15 to specify labels according to the class specified in the task. Examples of these tools include a specified class switching tool, a line drawing tool, a region filling tool, and a region erasing tool.

[0020] The source medical image display area 33 displays the source medical image 20. Annotation information 23 is generated by assigning labels to the source medical image 20 displayed in this source medical image display area 33 using various tools.

[0021] Below the source medical image display area 33, there are forward / backward buttons 34 and a display grayscale change button 35. The forward / backward buttons 34 can be used to change the slice position of the source medical image 20. The slice position indicates the position of the tomographic plane when the source medical image 20 is a tomographic image, as in this example. The display grayscale change button 35 can be used to change the window level (WL) and window width (WW). The window level and window width are parameters related to the display grayscale of the source medical image 20. The window level is the center value of the display area of ​​the source medical image 20, set relative to the pixel values ​​of the original image of the source medical image 20. The window width is a numerical value indicating the width of the display area of ​​the source medical image 20.

[0022] Thus, the annotator 15 can freely change the slice position of the source medical image 20. Therefore, the annotator 15 can repeatedly review the source medical image 20 at a specific slice position. In addition, the annotator 15 can freely change the display grayscale of the source medical image 20. Furthermore, the source medical image 20 in the source medical image display area 33 can be translated, enlarged, and reduced (see Figure 13).

[0023] At the bottom of the annotation information generation screen 30A, there are further buttons: a related medical information reference button 36, a temporary save button 37, and a generation end button 38. When the temporary save button 37 is pressed, the annotation information 23 generated up to that point is temporarily saved to the storage of the annotator terminal 11. When the generation end button 38 is pressed, the generated annotation information 23 is sent to the information processing server 10.

[0024] When the related medical information reference button 36 is operated, the system transitions to the annotation information generation screen 30B, as shown in Figure 5, as an example. The annotation information generation screen 30B has a related medical information display area 40 to the right of the source medical image display area 33. In this example, the related medical information display area 40 displays the medical image 22 from another device. Below the related medical information display area 40, there are forward / back buttons 41 and display grayscale change buttons 42, which have the same functions as the forward / back buttons 34 and display grayscale change buttons 35 at the bottom of the source medical image display area 33. Therefore, the annotator 15 can freely change the slice position and display grayscale of the medical image 22 from another device. In addition, the medical image 22 from another device in the related medical information display area 40 can be translated, enlarged, and reduced.

[0025] At the bottom of the annotation information generation screen 30B, a related medical information close button 43 is provided instead of the related medical information reference button 36. When the related medical information close button 43 is pressed, the related medical information display area 40, etc., is erased, and the display returns to the annotation information generation screen 30A.

[0026] As an example, as shown in Figure 6, if the annotation terminal 11 displays the annotation information generation screen 30B, which includes the related medical information 21 (medical image 22 from another device) in response to the operation of the related medical information reference button 36, on the display 13 before the generation completion button 38 is operated, i.e., if the related medical information reference button 36 was operated or not, it sends reference history information 24 to the information processing server 10 indicating that the related medical information 21 was referenced. On the other hand, if the related medical information reference button 36 is not operated even once before the generation completion button 38 is operated, and the annotation information generation screen 30B including the medical image 22 from another device is not displayed on the display 13, i.e., if the related medical information reference button 36 was not operated or not, it sends reference history information 24 to the information processing server 10 indicating that the related medical information 21 was not referenced.

[0027] As an example, as shown in Figure 7, the computer comprising the information processing server 10 includes storage 50, memory 51, processor 52, communication unit 53, display 54, and input device 55. These are interconnected via a bus line 56.

[0028] Storage 50 is a hard disk drive built into the computer constituting the information processing server 10, or connected via cable or network. Alternatively, storage 50 is a disk array consisting of multiple hard disk drives installed in series. Storage 50 stores control programs such as the operating system, various application programs (hereinafter abbreviated as AP (Application Program)), and various data associated with these programs. A solid-state drive may be used instead of a hard disk drive.

[0029] Memory 51 is work memory for the processor 52 to execute processing. Memory 51 is, for example, RAM (Random Access Memory) such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The processor 52 loads the program stored in storage 50 into memory 51 and executes processing according to the program. In this way, the processor 52 comprehensively controls each part of the computer. The processor 52 is, for example, a CPU (Central Processing Unit). Memory 51 is also an example of "memory" related to the technology of this disclosure. Note that storage 50, or storage 50 and memory 51, may also be defined as an example of "memory" related to the technology of this disclosure.

[0030] The communication unit 53 is a network interface that controls the transmission of various types of information via the network 12, etc. The display 54 displays various screens. These screens are equipped with GUI (Graphical User Interface) operation functions. The computers constituting the information processing server 10 receive operation instructions from the input device 55 through the various screens. The input device 55 is at least one of the following: a keyboard, mouse, touch panel, microphone, and gesture recognition device.

[0031] As an example, as shown in Figure 8, the storage 50 stores an operating program 60. The operating program 60 is an application program (AP) that causes the computer constituting the information processing server 10 to function as an "information processing device" according to the technology of this disclosure. In other words, the operating program 60 is an example of an "operating program for an information processing device" according to the technology of this disclosure. In addition to the operating program 60, the storage 50 also stores the source medical image 20, related medical information 21, annotation information 23, and reference history information 24. Although only one source medical image 20, related medical information 21, annotation information 23, and reference history information 24 are depicted, in reality, multiple source medical images 20, related medical information 21, annotation information 23, and reference history information 24 are stored in the storage 50.

[0032] When the operating program 60 is started, the processor 52 works in cooperation with the memory 51 and the like to function as a read / write (hereinafter abbreviated as RW) control unit 65, an image transmission unit 66, an information reception unit 67, and an output unit 68.

[0033] The RW control unit 65 controls the storage of various information to the storage 50 and the reading of various information from the storage 50. For example, the RW control unit 65 reads the source medical image 20 and related medical information 21 from the storage 50 and outputs the read source medical image 20 and related medical information 21 to the image transmission unit 66.

[0034] The information of the annotator terminal 11 that transmits the source medical image 20 and related medical information 21 is pre-registered in the storage 50. The image transmission unit 66 transmits the source medical image 20 and related medical information 21 from the RW control unit 65 to the pre-registered annotator terminal 11.

[0035] The information receiving unit 67 receives annotation information 23 and reference history information 24 from the annotator terminal 11. The information processing server 10 then acquires the annotation information 23 and reference history information 24. The information receiving unit 67 outputs the received annotation information 23 and reference history information 24 to the RW control unit 65. The RW control unit 65 stores the annotation information 23 and reference history information 24 in the storage 50. The RW control unit 65 also reads the reference history information 24 from the storage 50 and outputs the read reference history information 24 to the output unit 68.

[0036] The derivation unit 68 derives evaluation information 70 representing the quality of the annotation information 23 based on the reference history information 24. The derivation unit 68 outputs the derived evaluation information 70 to the RW control unit 65. The RW control unit 65 stores the evaluation information 70 from the derivation unit 68 in the storage 50. At this time, the RW control unit 65 stores the evaluation information 70 in association with the annotation information 23 and the reference history information 24, as shown by the dashed line.

[0037] The annotation information 23 is used as ground truth data, along with the source medical image 20, during the training phase or accuracy evaluation phase of the machine learning model. In this example, the annotation information 23 used as ground truth data is selected based on the evaluation information 70. In other words, the derivation of the evaluation information 70 is part of the machine learning model creation process.

[0038] In the learning phase, the source medical image 20 is input to the machine learning model. Next, the output annotation information output by the machine learning model is compared with the annotation information 23, and the machine learning model's loss is calculated. The machine learning model is then updated according to the loss. The loss decreases as the difference between the output annotation information and the annotation information 23 decreases. Therefore, the degree of update also decreases as the difference between the output annotation information and the annotation information 23 decreases. This input of the source medical image 20, output of the output annotation information, calculation of the loss, and update are repeated while pairs of source medical image 20 and annotation information 23, i.e., ground truth data, are exchanged. Through this, the machine learning model is learned.

[0039] In the accuracy evaluation phase, the source medical image 20 is input to a machine learning model that has undergone a certain amount of training. Then, the output annotation information output from the machine learning model is compared with the annotation information 23, a loss is calculated, and the accuracy of the machine learning model is evaluated based on the loss. In the accuracy evaluation phase, only the accuracy is evaluated; no updates are made. Machine learning models that are determined to have an accuracy equal to or greater than a predetermined level in this accuracy evaluation phase are used in the practical application phase. The ground truth data used in the training phase is also called training data, and the ground truth data used in the accuracy evaluation phase is also called evaluation data.

[0040] As an example, as shown in Figure 9, if the reference history information 24 indicates that the related medical information 21 was referenced, that is, if the related medical information 21 was referenced, the derivation unit 68 derives evaluation information 70 indicating that the quality of the annotation information 23 is good. On the other hand, if the reference history information 24 indicates that the related medical information 21 was not referenced, that is, if the related medical information 21 was not referenced, the derivation unit 68 derives evaluation information 70 indicating that the quality of the annotation information 23 is poor. Thus, the evaluation information 70 indicates a higher quality rating when the annotator 15 references the related medical information 21 than when the annotator 15 did not reference the related medical information 21. Note that the ground truth data uses only the source medical image 20 and annotation information 23 associated with, for example, the evaluation information 70 indicating good quality.

[0041] Next, the operation of the above configuration will be explained with reference to the flowchart in Figure 10. When the operating program 60 is started, the processor 52 of the information processing server 10 functions as the RW control unit 65, the image transmission unit 66, the information receiving unit 67, and the output unit 68, as shown in Figure 8.

[0042] First, the RW control unit 65 reads the source medical image 20 and related medical information 21 from the storage 50. The read source medical image 20 and related medical information 21 are output from the RW control unit 65 to the image transmission unit 66. The source medical image 20 and related medical information 21 are then transmitted to the annotator terminal 11 by the image transmission unit 66.

[0043] In the annotator terminal 11, the annotation information generation screen 30A shown in Figure 4 is displayed on the display 13. Then, the annotator 15 generates annotation information 23 based on the source medical image 20. At this time, if the annotator 15 operates the related medical information reference button 36, the system transitions to the annotation information generation screen 30B shown in Figure 5, and a medical image 22 from another device is displayed as related medical information 21.

[0044] As shown in Figure 6, the annotator terminal 11 generates reference history information 24 indicating whether or not the annotator 15 has referenced the relevant medical information 21. The annotation information 23 and reference history information 24 are transmitted from the annotator terminal 11 to the information processing server 10.

[0045] In the information processing server 10, the information receiving unit 67 receives annotation information 23 and reference history information 24 from the annotator terminal 11. As a result, the reference history information 24 is acquired (step ST100). The annotation information 23 and reference history information 24 are output from the information receiving unit 67 to the RW control unit 65, and stored in the storage 50 by the RW control unit 65.

[0046] The RW control unit 65 reads the reference history information 24 from the storage 50. The read reference history information 24 is output from the RW control unit 65 to the output unit 68.

[0047] As shown in Figure 9, the derivation unit 68 derives evaluation information 70 representing the quality of the annotation information 23 based on the reference history information 24 (step ST110). The evaluation information 70 is output from the derivation unit 68 to the RW control unit 65 and stored in the storage 50 by the RW control unit 65.

[0048] As described above, the processor 52 of the information processing server 10 includes an information receiving unit 67 and an information derivation unit 68. The information receiving unit 67 acquires information by receiving reference history information 24 from the annotator terminal 11. The reference history information 24 represents the reference history of the annotator 15 to related medical information 21 associated with the source medical image 20 that was the basis for generating the annotation information 23, when the annotator 15 generates the annotation information 23 as ground truth data for a machine learning model for analyzing medical images. The derivation unit 68 derives evaluation information 70 representing the quality of the annotation information 23 based on the reference history information 24.

[0049] The quality of annotation information 23 is largely determined by whether or not related medical information 21 was referenced when generating the annotation information 23. Therefore, according to the technology of this disclosure, which acquires reference history information 24 of related medical information 21 and derives evaluation information 70 based on the reference history information 24, it is possible to evaluate the quality of annotation information 23 more accurately compared to the case where evaluation information 70 is derived without using reference history information 24.

[0050] As shown in Figure 9, the evaluation information 70 gives a higher quality rating of the annotation information 23 when the annotator 15 refers to the related medical information 21 than when the annotator 15 does not refer to the related medical information 21. Therefore, the quality of the annotation information 23 can be properly evaluated.

[0051] As shown in Figure 2, the related medical information 21 includes a medical image 22 taken by a different medical imaging device than the source medical image 20. Therefore, lesions such as tumors that are not depicted in the source medical image 20 can be detected in the medical image 22, and it is possible to assist in labeling classes that are unclear from the source medical image 20 alone. As a result, the quality of the annotation information 23 can be improved.

[0052] [Second Embodiment] As an example shown in Figure 11, this embodiment handles five types of related medical information 80A, 80B, 80C, 80D, and 80E. Related medical information 80A includes a medical image 22 from a different device, similar to the related medical information 21 in the first embodiment described above. Related medical information 80B includes a medical image 81 from a different date and time. The medical image 81 from a different date and time is an image taken by the same medical imaging device as the source medical image 20, at a different date and time than the source medical image 20, typically earlier than the source medical image 20. Related medical information 80C includes specimen test data 82. Specimen test data 82 is data representing the results of specimen tests performed on the patient of the source medical image 20. Specimen tests include blood tests, urine tests, stool tests, etc.

[0053] Related medical information 80D includes pathology diagnosis results 83. Pathology diagnosis results 83 are the results of a pathologist's microscopic observation and diagnosis of lesions such as tumors taken from the patient in the source medical image 20. Related medical information 80E includes CAD (Computer-Aided Diagnosis) processing results 84. CAD processing results 84 are the results obtained by applying a pre-built CAD processing program to the source medical image 20. CAD processing results 84 are, for example, a frame surrounding the area where a liver tumor is suspected to exist. These medical images from different dates and times 81, specimen test data 82, pathology diagnosis results 83, and CAD processing results 84 are switchably displayed in the related medical information display area 40 of the annotation information generation screen 30B shown in Figure 5. By referring to these medical images from different dates and times 81, specimen test data 82, pathology diagnosis results 83, and CAD processing results 84 in the related medical information display area 40, the annotator 15 can label more accurately and appropriately. As a result, the quality of annotation information 23 can be improved.

[0054] As an example, as shown in Figure 12, the reference history information 90 of this embodiment registers whether or not the annotator 15 referenced each related medical information 80A to 80E. The derivation unit 68 (not shown) of this embodiment derives evaluation information 92 using the evaluation level table 91. The evaluation level table 91 registers the evaluation level of the annotation information 23 corresponding to the number of related medical information referenced by the annotator 15. Specifically, if the number of related medical information referenced by the annotator 15 is 0, evaluation level 1 is registered, and if the number of related medical information referenced by the annotator 15 is 1, evaluation level 2 is registered. Also, if the number of related medical information referenced by the annotator 15 is 2 and 3, evaluation level 3 is registered, and if the number of related medical information referenced by the annotator 15 is 4 and 5, evaluation level 4 is registered. The evaluation level is from 4 to 1, with 4 being the highest and 1 being the lowest. The evaluation level may also be represented by A to D, etc.

[0055] Figure 12 illustrates a case where the annotator 15 references two of the five types of related medical information 80A to 80E: the medical image from another device 22 (related medical information 80A) and the pathology diagnosis result 83 (related medical information 80D). In this case, since the number of related medical information referenced by the annotator 15 is two, the derivation unit 68 derives evaluation level 3 as evaluation information 92.

[0056] Thus, in the second embodiment, the derivation unit 68 derives evaluation information 92 according to the number of related medical information 80A to 80E referenced by the annotator 15. Rather than a simple determination of whether or not the related medical information 21 in the first embodiment was referenced, a more complex determination is added, making it possible to derive more reliable evaluation information 92.

[0057] In this embodiment, the evaluation level was derived according to the number of related medical information items referenced, regardless of the type of the five related medical information items 80A to 80E, but this is not limited to this. The evaluation may be weighted according to the type of related medical information items 80A to 80E. For example, if related medical information based on diagnostic results of a different method than diagnosis related to medical images, such as specimen test data 82 and pathology diagnosis results 83, is referenced, it may be evaluated more highly than if at least one of image information such as medical images from another device 22 and medical images from a different date and time 81, and CAD processing results 84 showing image processing results is referenced. Specifically, if related medical information such as specimen test data 82 and pathology diagnosis results 83 is referenced, the number of related medical information items referenced by the annotator 15 is counted as +2 per item, and if image information such as medical images from another device 22 and medical images from a different date and time 81 is referenced, the number of related medical information items referenced by the annotator 15 is counted as +1 per item, and so on.

[0058] [Embodiment 3_1] As an example, as shown in Figure 13, in this embodiment, the annotator terminal 11 generates operation history information 100. The operation history information 100 is a history of various operations performed on the source medical image 20 displayed in the source medical image display area 33 of the annotation information generation screens 30A and 30B. The operation history information 100 indicates whether or not an enlargement operation was performed on the source medical image 20 in the source medical image display area 33, and whether or not a display grayscale change operation was performed on the source medical image 20 in the source medical image display area 33. Figure 13 illustrates the case where an enlargement operation was performed to enlarge the liver portion. Also, Figure 13 illustrates the case where a display grayscale change operation was performed, changing the window level from 130 to 60 and the window width from 250 to 100.

[0059] The annotator terminal 11 transmits the operation history information 100, along with the annotation information 23 and the reference history information 24, to the information processing server 10. The information receiving unit 67 in this embodiment acquires the operation history information 100 by receiving it along with the annotation information 23 and the reference history information 24.

[0060] As an example, as shown in Figure 14, in this embodiment, the reference history information 90 of the second embodiment described above is used. The derivation unit 68 (not shown) of this embodiment derives evaluation information 102 using the evaluation level table 101 based on the reference history information 90 and the operation history information 100. The evaluation level table 101 registers the evaluation level of annotation information 23 corresponding to the sum of the number of related medical information referenced by the annotator 15 and the number of operations performed on the source medical image 20. Specifically, if the sum is 0 and 1, evaluation level 1 is registered, and if the sum is 2 and 3, evaluation level 2 is registered. Also, if the sum is 4 and 5, evaluation level 3 is registered, and if the sum is 6 and 7, evaluation level 4 is registered. As with the second embodiment described above, the evaluation level is from 4 to 1, with 4 being the highest and 1 being the lowest.

[0061] In Figure 14, as in Figure 12 of the second embodiment described above, an example is shown in which the annotator 15 references two of the five types of related medical information 80A to 80E: the medical image 22 from another device (related medical information 80A) and the pathology diagnosis result 83 (related medical information 80D). Furthermore, Figure 14 also illustrates the case in which both the enlargement operation and the display grayscale change operation of the source medical image 20 are performed. In this case, the sum of the number of related medical information referenced by the annotator 15 and the number of operations performed on the source medical image 20 is 2 + 2 = 4, so the derivation unit 68 derives evaluation level 3 as evaluation information 102.

[0062] Thus, in the third-first embodiment, the information receiving unit 67 acquires the operation history information 100 of the annotator 15 for the source medical image 20 when the annotation information 23 was generated by receiving it from the annotator terminal 11. The derivation unit 68 derives evaluation information 102 based on the operation history information 100 in addition to the reference history information 90. Therefore, compared to the first embodiment described above, in which evaluation information 70 is derived using only the reference history information 24, it becomes possible to evaluate the quality of the annotation information 23 more accurately.

[0063] The operation history information 100 is information related to zoom operations and display grayscale change operations. Zoom operations are necessary to observe the area to be labeled in more detail. Display grayscale change operations are necessary to make the area to be labeled easier to see. Thus, by making the operation history information 100 information related to zoom operations and display grayscale change operations, which are essential for accurate labeling, it is possible to contribute to the accurate evaluation of the quality of the annotation information 23.

[0064] In Figure 13, operation history information 100 for the source medical image 20 is shown as an example, but it is not limited to this. As an example, as shown in Figure 15, operation history information 105 for the medical image 22 of another device displayed in the related medical information display area 40 of the annotation information generation screen 30B may be generated instead of, or in addition to, the operation history information 100. The operation history information 105, like the operation history information 100, indicates whether or not an enlargement operation was performed on the medical image 22 of another device in the related medical information display area 40, and whether or not a display grayscale change operation was performed on the medical image 22 of another device in the related medical information display area 40. In Figure 15, an example is shown in which an enlargement operation was performed to enlarge the liver portion. Also in Figure 15, an example is shown in which a display grayscale change operation was performed, changing the window level from 150 to 100 and the window width from 200 to 150.

[0065] The operation history information 100 and 105 only need to be information relating to at least one of the following: a zoom operation and a display grayscale change operation. In addition, the operation history information 100 and 105 may also include whether or not a translation operation was performed.

[0066] As an example, the configuration shown in Figure 16 may be implemented at the annotator terminal 11. Specifically, the annotator 15 performs a display grayscale change operation on the source medical image 20 (step ST200), and if a label corresponding to a certain class is assigned (step ST210), it is determined whether the changed display grayscale was within the recommended range for the assigned label (step ST220). If it is determined that the changed display grayscale was within the recommended range for the assigned label (YES in step ST220), "Implemented" is registered in the display grayscale change operation of the operation history information 100 (step ST230). On the other hand, if it is determined that the changed display grayscale was not within the recommended range for the assigned label (NO in step ST220), "Not Implemented" is registered in the display grayscale change operation of the operation history information 100 (step ST240). The recommended range for display grayscale is set in advance for each class. The recommended range of display gradation is set, for example, based on the average value of display gradation set by an unspecified number of annotators 15 when assigning labels corresponding to that class.

[0067] According to the embodiment shown in Figure 16, unless the display grayscale is changed to the recommended range, "executed" will not be registered in the display grayscale change operation in the operation history information 100. Therefore, even if a label is assigned with an inappropriate display grayscale, it will be considered that the display grayscale change operation was performed, preventing the quality evaluation of the annotation information 23 from becoming too high.

[0068] Furthermore, while it was stated that determining whether a display grayscale change operation has been performed is based on whether or not the display grayscale has been changed to the recommended range, which is a condition based on the source medical image 20 or related medical information 21, this is not the only way to determine whether a display grayscale change operation has been performed. Even if the conditions based on the source medical image 20 or related medical information 21 are not met, if any display grayscale change operation has occurred, it may be considered that a display grayscale change operation has been performed unconditionally. Also, in the case of a zoom operation, if a label is assigned to the area displayed by the zoom operation, it may be determined that a zoom operation has been performed, and if a label is not assigned to the area displayed by the zoom operation, it may be determined that a zoom operation has not been performed.

[0069] Alternatively, the same source medical image 20 may be sent to multiple annotator terminals 11 to generate annotation information 23 for multiple annotators 15, and the recommended range of display grayscale may be set based on the average value of the display grayscale set at that time. In addition, the recommended range of display grayscale may be set for each organ and / or for each disease.

[0070] [Embodiment 3_2] As an example, as shown in Figure 17, in this embodiment, if the annotator 15 performs any operation on the related medical information 21, it is determined that the annotator 15 has referenced the related medical information 21.

[0071] In Figure 17, the related medical information 21 is displayed in the related medical information display area 40 (step ST300). If the annotator 15 performs any operation on the related medical information 21 (YES in step ST310), it is determined that the annotator 15 has referenced the related medical information 21, and this fact is registered in the reference history information 90 (step ST320). On the other hand, if the annotator 15 does not perform any operation on the related medical information 21 (NO in step ST310), it is determined that the annotator 15 has not referenced the related medical information 21, and this fact is registered in the reference history information 90 (step ST330).

[0072] For example, if the operation history information for medical image 22 on another device indicates that an enlargement operation was performed on medical image 22 on another device, the annotator 15 will determine that medical image 22 on another device was referenced and register that fact in the reference history information. Alternatively, if the operation history information for medical image 81 at a different date and time indicates that a display grayscale change operation was performed on medical image 81 at a different date and time, and the changed display grayscale was within the recommended range of the assigned label, the annotator 15 may determine that medical image 81 at a different date and time was referenced and register that fact in the reference history information.

[0073] Depending on the annotator 15, it may simply display the medical image 22 from another device without performing any zoom operations, and may not refer to the medical image 22 from another device very much. Also, in the first embodiment described above, the medical image 22 from another device is displayed only when the related medical information reference button 36 is operated, but depending on the specifications, the medical image 22 from another device may be displayed automatically without waiting for an operation of the annotator 15, such as the operation of the related medical information reference button 36. For this reason, if the annotator 15 determines that it has referred to the medical image 22 from another device simply because it has been displayed, the accuracy of the quality evaluation of the annotation information 23 may be impaired. Therefore, in the third-second embodiment, the annotator 15 determines that it has referred to the related medical information 21 when there is some operation performed on the related medical information 21 by the annotator 15. This ensures the accuracy of the quality evaluation of the annotation information 23.

[0074] Furthermore, the operation history information is not limited to zoom operations and display grayscale changes for the related medical information 21. The annotator terminal 11 may measure the display time of the related medical information 21, and if the display time is above a preset threshold, it may be determined that the related medical information 21 has been viewed. Alternatively, the annotator terminal 11 may transmit the display time of the related medical information 21 to the information processing server 10, and the information processing server 10 may determine whether or not the related medical information 21 has been viewed. In addition, the annotator terminal 11 has a gaze detection function for the annotator 15, detects the gaze of the annotator 15 viewing the annotation information generation screen 30B, and transmits the time the gaze was placed on the related medical information display area 40 where the related medical information 21 is displayed to the information processing server 10 as operation history information associated with the related medical information 21. The information processing server 10 may determine that the related medical information 21 has been viewed if the time the gaze was placed, as recorded in the operation history information, is within a predetermined period (for example, a cumulative total of three minutes).

[0075] [Fourth Embodiment] As an example, as shown in Figure 18, the annotator terminal 11 transmits the annotator 15's attribute information 110, along with the annotation information 23 and the reference history information 90, to the information processing server 10. The information receiving unit 67 in this embodiment acquires the attribute information 110 by receiving it along with the annotation information 23 and the reference history information 90. The attribute information 110 includes the annotator 15's years of service, qualifications, and workload. Qualifications include, in addition to the example of a radiology training instructor, a specialist in diagnostic radiology, etc. The workload is obtained when the annotator 15 inputs their workload to the annotator terminal 11. The workload can be "busy," as well as "normal" and "quiet," as exemplified.

[0076] As an example, as shown in Figure 19, the derivation unit 68 (not shown) of this embodiment derives evaluation information 112 using the evaluation level table 91 and evaluation level increment / decrement table 111 of the second embodiment based on the reference history information 90 and attribute information 110. The evaluation level increment / decrement table 111 registers an increment / decrement value for the evaluation level for each item in the attribute information 110. Specifically, if the length of service is 20 years or more, an increment / decrement value of +1 is registered, and if the length of service is less than 5 years, an increment / decrement value of -1 is registered. Also, if the person is qualified, an increment / decrement value of +1 is registered, and if the workload is "busy", an increment / decrement value of -1 is registered.

[0077] In Figure 19, as in Figure 12 of the second embodiment described above, an example is shown in which the annotator 15 references two of the five types of related medical information 80A to 80E: medical image from another device 22 (related medical information 80A) and pathology diagnosis result 83 (related medical information 80D). Furthermore, Figure 19 illustrates the case of annotator 15 with 22 years of service, qualifications as a radiology training instructor, and a workload level of "busy". In this case, the number of related medical information referenced by the annotator 15 is 2, and the increase / decrease value of the evaluation level according to the attribute information 110 is +1 for 22 years of service, +1 for qualifications, and -1 for a workload level of "busy", so the derivation unit 68 derives evaluation level 3 as evaluation information 112.

[0078] Thus, in the fourth embodiment, the information receiving unit 67 acquires attribute information 110 of the annotator 15 by receiving it from the annotator terminal 11. The derivation unit 68 derives evaluation information 112 based on the attribute information 110 in addition to the reference history information 90. Therefore, compared to the first embodiment, in which evaluation information 70 is derived using only the reference history information 24, it becomes possible to evaluate the quality of the annotation information 23 more accurately.

[0079] According to the embodiment shown in Figure 19, annotation information 23 generated by annotators 15 with relatively long years of service, or by qualified annotators 15, receives a relatively high quality rating. On the other hand, annotation information 23 generated by annotators 15 with relatively short years of service, or by busy annotators 15, receives a relatively low quality rating. Annotators 15 with relatively long years of service, or qualified annotators 15, are considered to have relatively high accuracy in labeling. On the other hand, annotators 15 with relatively short years of service, or busy annotators 15, are considered to have relatively low accuracy in labeling. Therefore, it is possible to make a quality rating that is adapted to the attributes of the annotator 15.

[0080] Evaluation information may be derived based on the reference history information 90, operation history information 100 or 105, and attribute information 110 by applying the above-described third-first embodiment and / or third-second embodiment.

[0081] While it is stated that attribute information 110 is sent from the annotator terminal 11, this is not the only option. For example, attribute information 110 for each annotator ID (Identification Data) used to identify the annotator 15 is stored in the storage 50 of the information processing server 10. Then, the annotator terminal 11 sends the annotator ID. The information processing server 10 obtains the attribute information 110 corresponding to the annotator ID sent from the annotator terminal 11 by reading it from the storage 50.

[0082] The level of busyness can also be derived from the attendance sheet, schedule sheet, etc., of Annotator 15. For example, if an employee leaves work after 9:00 PM for two consecutive days, it can be judged as busy, and if they leave work on time, it can be judged as slow. Alternatively, weeks with regular meetings, lectures, academic conferences, etc., can be judged as busy.

[0083] The annotator's (15) area of ​​expertise may be included in attribute information 110. In this case, for example, if the task is related to the area of ​​expertise, the evaluation level will be increased by +1. The number of published papers by annotator (15) may also be included in attribute information. In this case, if the number of published papers is equal to or greater than the first threshold, the evaluation level will be increased by +1, and if the number of published papers is less than the second threshold, the evaluation level will be decreased by -1.

[0084] [Fifth Embodiment] As an example, as shown in Figure 20, the information processing server 120 of this embodiment includes storage 121, a processor 122, a display 123, and an input device 124. The storage 121 stores the source medical image 20, annotation information 23, evaluation information 92, and an operating program 130 (related medical information 21, reference history information 90, etc. are not shown), as well as a machine learning model 131. Note that evaluation information 92 may be evaluation information 102 or evaluation information 112.

[0085] When the operating program 130 is started, the processor 122, in cooperation with memory and other components (not shown), functions as a display control unit 135, an instruction receiving unit 136, and a learning unit 137, in addition to the processing units 65 to 68 (not shown) of the first embodiment described above.

[0086] Annotation information 23 and evaluation information 92 are input to the display control unit 135. The display control unit 135 controls the display of the correct data selection screen 140 (see Figure 21) on the display 123 based on the annotation information 23 and evaluation information 92. The instruction receiving unit 136 receives instructions for selecting correct data through the correct data selection screen 140 via the input device 124. The learning unit 137 provides the source medical image 20 and annotation information 23 selected on the correct data selection screen 140 to the machine learning model 131 as correct data and trains the machine learning model 131.

[0087] As an example, as shown in Figure 21, the correct answer data selection screen 140 displayed on the display 123 under the control of the display control unit 135 has an annotation information selection unit 141 and an evaluation level selection unit 142. The annotation information selection unit 141 and the evaluation level selection unit 142 are provided with checkboxes 143 and 144 for selectively selecting either of these selection units 141 and 142.

[0088] The annotation information selection section 141 lists the evaluation level, annotation information ID, generation date, and the name of the annotator 15 for each annotation information 23. The annotation information selection section 141 also includes checkboxes 145 and 146 for each annotation information 23, allowing the user to select whether to use it in the training phase or the accuracy evaluation phase of the machine learning model 131. The annotation information selection section 141 can be scrolled vertically.

[0089] The evaluation level selection section 142 lists the number of annotation information entries 23 for each evaluation level. The evaluation level selection section 142 also includes checkboxes 147 and 148 for selecting whether to use each evaluation level in the training phase or the accuracy evaluation phase of the machine learning model 131. The evaluation level selection section 142 may also display the number of referenced related medical information entries corresponding to each evaluation level.

[0090] The user developing the machine learning model 131 selects annotation information 23 to be used as ground truth data in the training and accuracy evaluation phases of the machine learning model 131 by appropriately selecting checkboxes 143 to 148 on the ground truth data selection screen 140. After selecting the annotation information 23, the user developing the machine learning model 131 selects the training start button 149 located in the lower right corner of the ground truth data selection screen 140. This allows the instruction to select ground truth data to be received by the instruction reception unit 136.

[0091] Figure 21 illustrates the case where checkbox 144 of the evaluation level selection unit 142 is selected. Figure 21 also shows the case where annotation information 23 with an evaluation level of 4 of the evaluation information 92 is selected for the accuracy evaluation phase, and annotation information 23 with evaluation levels of 1 to 3 of the evaluation information 92 is selected for the learning phase.

[0092] The instruction receiving unit 136 outputs a correct data selection instruction to the learning unit 137. In the learning phase and the accuracy evaluation phase, the learning unit 137 provides annotation information 23 according to the correct data selection instruction to the machine learning model 131 for training.

[0093] Thus, in the fifth embodiment, the instruction receiving unit 136 accepts the selection of annotation information 23 to be used as correct answer data. Therefore, the machine learning model 131 can be trained by providing the machine learning model 131 with the annotation information 23 desired by the development user of the machine learning model 131.

[0094] According to the annotation information selection unit 141, it is possible to make detailed selections for each annotation information 23 as to whether or not to use it as correct answer data. According to the evaluation level selection unit 142, it is possible to make selections for each evaluation level as to whether or not to use it as correct answer data. According to the evaluation level selection unit 142, the effort required to select correct answer data can be reduced compared to selecting for each annotation information 23 using the annotation information selection unit 141.

[0095] Note that the user terminal operated by the developer of the machine learning model 131 may be separate from the information processing server. The machine learning model 131 is stored in the user terminal's storage. In this case, the information processing server sends annotation information 23 and evaluation information 92, which form the basis of the correct answer data selection screen 140, to the user terminal. Based on the annotation information 23 and evaluation information 92 from the information processing server, the user terminal generates the correct answer data selection screen 140 and displays it on its display. The user terminal sends a correct answer data selection instruction to the information processing server, indicating the selection status of the correct answer data selection screen 140. The information processing server accepts the correct answer data selection instruction and sends the source medical image 20 and annotation information 23 corresponding to the correct answer data selection instruction to the user terminal. The user terminal provides the source medical image 20 and annotation information 23 from the information processing server to the machine learning model 131 for training.

[0096] The machine learning model 131 may be trained using the annotation information 23 for evaluation level 1, evaluation level 2, evaluation level 3, and evaluation level 4, respectively, as ground truth data. In this case, the machine learning model 131 with the highest accuracy among the machine learning models 131 that use the annotation information 23 for each evaluation level as ground truth data will be used in the practical phase. The reference history information corresponding to the evaluation level of the annotation information 23 used in the machine learning model 131 with the highest accuracy may be recommended to the annotator 15.

[0097] [Sixth Embodiment] As an example, as shown in Figure 22, the processor of the information processing server in this embodiment functions as an information transmission unit 160 in addition to the processing units 65 to 68 (not shown) of the first embodiment described above. The information transmission unit 160 transmits the overall summary information 161 and the individual summary information 162 to the annotator terminal 11.

[0098] The overall summary information 161 provides an overview of the evaluation information 92 for all annotation information 23 generated for use as ground truth data for a particular machine learning model. The individual summary information 162 provides an overview of the evaluation information 92 for annotation information 23 generated by the annotator 15 of the annotator terminal 11 to which the individual summary information 162 is sent, from among the annotation information 23 generated for use as ground truth data for a particular machine learning model. Specifically, the overall summary information 161 and the individual summary information 162 are information in which the number of annotation information 23 entries is registered for each evaluation level of the evaluation information 92. In other words, the overall summary information 161 and the individual summary information 162 are statistical information on the number of annotation information 23 entries. Note that the evaluation information 92 may also be evaluation information 102 or evaluation information 112.

[0099] Upon receiving the overall summary information 161 and the individual summary information 162, the annotator terminal 11 displays an evaluation summary display screen 170, as shown in Figure 23, as an example, on the display 13. The evaluation summary display screen 170 has an overall summary information display area 171 and an individual summary information display area 172. The overall summary information display area 171 displays the overall summary information 161 and the percentage of annotation information 23 for each evaluation level. The individual summary information display area 172 displays the individual summary information 162 and the percentage of annotation information 23 for each evaluation level. The evaluation summary display screen 170 is turned off when the confirmation button 173 located in the lower right corner is selected.

[0100] Thus, in the sixth embodiment, the information transmission unit 160 transmits to the annotator terminal 11 operated by the annotator 15 an overall summary information 161 showing an overview of the evaluation information 92 of all annotation information 23, and individual summary information 162 showing an overview of the evaluation information 92 of the annotation information 23 generated by each individual annotator 15. Therefore, the annotator 15 can compare the differences between the overall evaluation information 92 and their own. For example, if the proportion of evaluation levels 4 and 3 in the individual summary information 162 is higher than the proportion of evaluation levels 4 and 3 in the overall summary information 161, the annotator 15 can see that the annotation information 23 they generated is relatively highly rated, and can confirm that their method of assigning labels is not so far off the mark. Note that it is not limited to transmitting both the overall summary information 161 and the individual summary information 162; it is sufficient to transmit at least one of them to the annotator terminal 11.

[0101] As an example, as shown in the individual summary information 175 in Figure 24, the acceptance or rejection of annotation information 23 into the correct data may be registered for each evaluation level of evaluation information 92. In this case, as shown in the evaluation summary display screen 180 in Figure 25, as an example, the acceptance or rejection of annotation information into the correct data is displayed in the individual summary information display area 181. In addition, an acceptance rate display area 182 is provided below the individual summary information display area 181 to display the acceptance rate. The acceptance rate is obtained by dividing the number of annotation information 23 that were accepted into the correct data by the total number of annotation information 23 generated by the annotator 15 at the annotator terminal 11 to which the individual summary information 175 is sent. By sending the individual summary information 175, on which the acceptance or rejection of annotation information 23 into the correct data is registered, to the annotator terminal 11, the annotator 15 can find out to what extent the annotation information 23 that it generated has been accepted into the correct data.

[0102] Furthermore, at least one of the calculations for the proportion of annotation information 23 for each evaluation level and the adoption rate can be performed by at least one of the information processing server 120 or the annotator terminal 11. While the proportion is exemplified as the proportion of the total or the proportion of the annotation information 23 associated with individual annotators 15 to the total, it is not limited to these. For example, the proportion may represent the ratio of individual annotation information 23 to all annotation information 23. For example, in the case of evaluation level 4, the total number of annotation information 23 is 51, and the number generated by the target annotator 15 is 16, so 16 / 51 = 31% is the proportion of individual annotation information 23 to all annotation information 23.

[0103] [Seventh Embodiment] As an example, as shown in Figure 26, the image transmission unit 66 of this embodiment retransmits the source medical image 20 and related medical information 21 to the annotator terminal 11 of the annotator 15 that generated annotation information 23 with a relatively poor quality evaluation, such as an evaluation level of 1, in order to prompt correction of the annotation information 23. At this time, the image transmission unit 66 attaches recommendation information 190. The recommendation information 190 stores the things that the annotator 15 should do to improve the evaluation level. Specifically, the recommendation information 190 stores the recommended range of window level and window width when assigning labels for each class.

[0104] Upon receiving the source medical image 20, related medical information 21, and recommendation information 190, the annotator terminal 11 displays a message prompting correction of the annotation information 23, and an annotation information generation screen including the recommendation information 190, on the display 13.

[0105] Thus, in the seventh embodiment, the annotator 15 is prompted to revise the annotation information 23 with a relatively low quality rating. At the same time, recommendation information 190 is attached to the revised annotation information 23, and efforts are made to improve the quality rating of the revised annotation information 23. As a result, the annotation information 23 with a relatively low quality rating can be effectively revised into annotation information 23 of high quality.

[0106] Furthermore, the recommendation information 190 may include a note stating that at least two types of related medical information 21 should be referenced. Also, when initially sending the source medical image 20 and related medical information 21 to the annotator terminal 11 to request the generation of annotation information 23, the recommendation information 190 may also be sent at the same time.

[0107] Furthermore, the annotator terminal 11 may display related medical information 21 in parallel according to the recommended information 190. If the annotator terminal 11 receives recommended information 190 that includes a note to refer to two or more types of related medical information 21, when the generation completion button 38 is operated, it may determine whether or not two or more types of related medical information 21 have been referred to so far. If the number of referred related medical information 21 has not been two or more, the display of the related medical information display area 40 may be automatically switched to force the annotator 15 to refer to two or more types of related medical information 21.

[0108] In the first embodiment described above, the source medical image 20 and related medical information 21 are transmitted from the information processing server 10 to the annotator terminal 11, but this is not limited to this. For example, an image management server for storing and managing the source medical image 20 and a related medical information management server for storing and managing the related medical information 21 may be provided separately from the information processing server 10, and the source medical image 20 and related medical information 21 may be transmitted from the image management server and the related medical information management server to the annotator terminal 11.

[0109] The source medical image 20 is not limited to abdominal tomography images taken with the exemplified CT scanner. For example, it could be a head tomography image taken with an MRI (Magnetic Resonance Imaging) scanner. Furthermore, medical images are not limited to three-dimensional images such as tomography images. For example, it could be a two-dimensional image such as a simple radiograph. It could also be a PET (Positron Emission Tomography) image, a SPECT (Single Photon Emission Computed Tomography) image, an endoscopic image, an ultrasound image, or a fundus examination image.

[0110] Similarly, the medical images from other devices 22 are not limited to abdominal tomographic images taken with the MRI device shown as an example. Furthermore, as shown in the related medical information 195 in Figure 27 as an example, there may be multiple types of medical images from other devices 22. In this case, information on the type of medical imaging device should be attached to each medical image from other devices 22. In Figure 27, examples include medical images from other devices 22A taken with an MRI device, medical images from other devices 22B taken with a plain radiography device, and medical images from other devices 22C taken with an ultrasound device.

[0111] Depending on the type of medical imaging device used to acquire the medical image 22 from another device, the number of references to the relevant medical information referenced by the annotator 15 may be changed. For example, the count may be set to 1 when a medical image 22A acquired by an MRI device is referenced, and to 0.5 when a medical image 22B acquired by a plain radiography device and a medical image 22C acquired by an ultrasound device are referenced. This makes it possible to differentiate between medical images 22 from another device that contribute relatively more to the generation of annotation information 23 and those that contribute less.

[0112] As an example, there may be multiple types of medical images 81 taken at different times, as shown in related medical information 197 in Figure 28. In this case, information on the date and time of acquisition should be attached to each medical image 81 taken at a different time. Figure 28 shows examples of medical image 81A taken at a different time on March 24, 2021, and medical image 81B taken at a different time on March 17, 2021, one week earlier.

[0113] Similar to the case of medical images 22 from a separate device, the count of the number of related medical information referenced by the annotator 15 may be changed according to the date and time the medical image 81 from a different date and time was taken. For example, the count may be set to 1 when the most recently taken medical image 81A from a different date and time is referenced, and the count may be set to 0.5 when the medical image 81B taken one week before medical image 81A from a different date and time is referenced. In this way, it is possible to differentiate between medical images 81 from a different date and time that contribute relatively more to the generation of annotation information 23 and those that contribute less.

[0114] The classes to which labels are assigned are not limited to the liver, tumors, and hemorrhages exemplified above. They may also include other organs such as the brain, eyeballs, spleen, and kidneys; bones such as vertebrae and ribs; anatomical parts of organs such as S1-S10 of the lungs; the head, body, and tail of the pancreas; and other abnormal findings such as cysts, atrophy, ductal stenosis, or ductal dilation. They may also include pacemakers, artificial joints, and bolts used for fracture treatment. Furthermore, different classes of labels may be assigned to the same area, such as assigning two labels—tumor and hemorrhage—to a bleeding site in a tumor.

[0115] Labels are currently assigned to each pixel of the source medical image 20, but this is not limited to this. For example, labels may be assigned to a rectangular (if the source medical image 20 is a two-dimensional image) or box-shaped (if the source medical image 20 is a three-dimensional image) frame that encloses an entire class such as a tumor.

[0116] The label may be applied to the source medical image 20 itself, rather than to a region of the source medical image 20. For example, a label indicating the presence or absence of dementia may be applied to a cross-sectional image of the head taken by an MRI device.

[0117] Various screens, such as the annotation information generation screen 30A, may be sent from the information processing server 10 or 120 to the annotator terminal 11 in the form of web-distributed screen data created using a markup language such as XML (Extensible Markup Language). In this case, the annotator terminal 11 reproduces the various screens to be displayed on a web browser based on the screen data and displays them on the display 13. Note that other data description languages ​​such as JSON (Javascript® Object Notation) may be used instead of XML.

[0118] When sending screen data such as the annotation information generation screen 30A from the information processing server 10 or 120 to the annotator terminal 11 as described above, the operation history information 100 and 105 of the third embodiment described above may be generated on the information processing server 10 or 120 side.

[0119] The hardware configuration of the computers constituting the information processing server 10 can be modified in various ways. For example, the information processing server 10 can be composed of multiple server computers separated as hardware, in order to improve processing power and reliability. For example, the functions of the RW control unit 65, the image transmission unit 66, and the information reception unit 67, and the function of the output unit 68 can be distributed among two server computers. In this case, the information processing server 10 is composed of two server computers. Some or all of the functions of each processing unit 65 to 68 of the information processing server 10 may be handled by the annotator terminal 11.

[0120] Thus, the hardware configuration of the computer of the information processing server 10 can be appropriately changed according to the required performance, such as processing power, security, and reliability. Furthermore, not only the hardware, but also the application programs (APs) such as the operating program 60 can, of course, be duplicated or distributed and stored on multiple storage devices for the purpose of ensuring security and reliability.

[0121] In each of the above embodiments, for example, the hardware structure of the Processing Unit that performs various processes such as the RW control unit 65, image transmission unit 66, information reception unit 67, output unit 68, display control unit 135, instruction reception unit 136, learning unit 137, and information transmission unit 160 can be the following types of processors. The types of processors include a CPU, which is a general-purpose processor that executes software (operation program 60 or 130) and functions as various processing units, as well as a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), and / or a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to perform a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0122] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, and / or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0123] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, which then functions as multiple processing units, as exemplified by client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0124] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor elements.

[0125] [Note 1] It is preferable that the quality rating is higher when the annotator refers to relevant medical information than when the annotator does not.

[0126] [Note 2] The related medical information preferably includes at least medical images taken by a different medical imaging device than the source medical image.

[0127] [Note 3] There are multiple types of related medical information, and it is preferable for the processor to derive evaluation information according to the number of related medical information items that the annotator has referenced among the multiple types of related medical information.

[0128] [Note 4] Preferably, the processor acquires annotator operation history information for at least one of the source medical image and related medical information when generating annotation information, and derives evaluation information based on the operation history information in addition to the reference history information.

[0129] [Note 5] The operation history information preferably relates to at least one of the following: zoom operation and display grayscale change operation.

[0130] [Note 6] The processor preferably acquires attribute information of the annotator and derives evaluation information based on the attribute information in addition to the reference history information.

[0131] [Note 7] The processor preferably accepts the selection of annotation information to be used as ground truth data.

[0132] [Note 8] The processor preferably transmits to the annotator terminal operated by the annotator at least one of the following: overall summary information showing an overview of the evaluation information of all annotation information, and individual summary information showing an overview of the evaluation information of the annotation information generated by each individual annotator.

[0133] The technology of this disclosure can be appropriately combined with the various embodiments and / or variations described above. Furthermore, it is understood that various configurations can be adopted without departing from the spirit of the invention, and the invention is not limited to the embodiments described above. Moreover, the technology of this disclosure extends not only to programs but also to storage media for storing programs non-temporarily.

[0134] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0135] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0136] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. Equipped with a processor, The aforementioned processor, When an annotator generates annotation information as ground truth data for a machine learning model to analyze medical images, the annotator obtains the reference history information of related medical information associated with the source medical image from which the annotation information was generated. Based on the aforementioned reference history information, evaluation information representing the quality of the annotation information is derived. Information processing device.

2. The information processing apparatus according to claim 1, wherein the evaluation information is higher when the annotator refers to the related medical information than when the annotator does not refer to the related medical information.

3. The information processing device according to claim 1 or claim 2, wherein the related medical information includes at least a medical image taken by a medical image acquisition device other than the source medical image.

4. The aforementioned related medical information comes in multiple forms. The aforementioned processor, An information processing device according to any one of claims 1 to 3, which derives the evaluation information according to the number of the related medical information referenced by the annotator from among the multiple types of related medical information.

5. The aforementioned processor, When generating the annotation information, the annotator's operation history information is obtained for at least one of the source medical image and the related medical information. An information processing device according to any one of claims 1 to 4, wherein the evaluation information is derived based on the operation history information in addition to the reference history information.

6. The information processing apparatus according to claim 5, wherein the operation history information relates to at least one of the following: a zoom operation and a display grayscale change operation.

7. The aforementioned processor, Obtain the attribute information of the aforementioned annotator, An information processing device according to any one of claims 1 to 6, wherein the evaluation information is derived based on the attribute information in addition to the reference history information.

8. The aforementioned processor, An information processing device according to any one of claims 1 to 7, which accepts the selection of annotation information to be used as the correct answer data.

9. The processor is The information processing apparatus according to claim 8, which accepts the selection of annotation information to be used as correct answer data, according to the annotation information or evaluation information.

10. The aforementioned processor, The information processing apparatus according to any one of claims 1 to 8, which transmits to an annotator terminal operated by the annotator at least one of the following: overall summary information showing an overview of the evaluation information of all the annotation information, and individual summary information showing an overview of the evaluation information of the annotation information generated by each of the annotators.

11. The processor is The information processing device according to any one of claims 1 to 10, wherein the source medical image and the related medical information are retransmitted to the annotator terminal operated by the annotator, with the purpose of prompting the annotator to modify the annotation information by attaching recommendation information indicating recommended operations to improve the evaluation.

12. The processor is The information processing device according to any one of claims 1 to 11, wherein weighting is applied according to the type of related medical information in the derivation of the evaluation information.

13. A method for operating an information processing device equipped with a processor, The aforementioned processor, When an annotator generates annotation information as ground truth data for a machine learning model for analyzing medical images, the annotator obtains the reference history information of related medical information associated with the source medical image from which the annotation information was generated, and, Based on the aforementioned reference history information, evaluation information representing the quality of the annotation information is derived. A method for operating an information processing device, including the device itself.

14. When an annotator generates annotation information as ground truth data for a machine learning model for analyzing medical images, the annotator obtains the reference history information of related medical information associated with the source medical image from which the annotation information was generated, and, Based on the aforementioned reference history information, evaluation information representing the quality of the annotation information is derived. An operating program for an information processing device that causes a computer to perform a process that includes [specific details].

Images