Program, assistance method, and system
The system addresses the inconsistency in medical image annotation by generating a presentation screen for comparing multiple annotations, improving annotation quality and consistency, which is essential for reliable medical image analysis.
Patent Information
- Application Number
- PCT/JP2024/019507
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-05
AI Technical Summary
Medical image annotation can be inconsistent among different annotators, even for experienced doctors, leading to variations in labeling judgments, which complicates the annotation process and affects the quality of learning data for machine learning algorithms.
A program and system that generate a presentation screen to compare and display classification contents from multiple annotations on medical images, allowing users to visually recognize and compare label classifications across different annotators, including features like grid-based and frame-based annotations, and presenting comparison results to support annotation consistency.
The system provides useful information for annotators by facilitating the comparison of label classifications, improving annotation consistency and quality of learning data, thereby enhancing the reliability of medical image analysis.
Smart Images

Figure JP2024019507_05122024_PF_FP_ABST
Abstract
Description
Program, support method and system
[0001] The present invention relates to a program, a method, and a system for supporting annotation of medical images.
[0002] The data generation device in Patent Document 1 generates training data for a machine learning algorithm. The training data consists of multiple images and labels associated with each image. Depending on the context, the term "annotation" can refer to the act of labeling an image itself or a specified region within an image, or it can refer to a group of assigned labels.
[0003] International Publication No. 2021 / 206053
[0004] Even doctors with specialized knowledge may disagree on whether a medical image contains features such as pathological abnormalities. This can lead to differences in labeling decisions when annotating a medical image, for example, when multiple annotators perform annotations or when the same annotator performs annotations multiple times. The present inventors have realized that the consistency and / or differences in content between multiple annotations can be useful information for annotators.
[0005] The present invention provides a program, a support method, and a system that are configured to be able to present information that is useful when performing annotation.
[0006] According to the present disclosure, at least the following inventions are provided: [1] A program for supporting annotation of medical images, the program causing a computer to execute a presentation process for generating a presentation screen, the presentation screen presenting to a user classification content of at least one label associated with at least one region image in the medical image by at least one of a plurality of annotations. [2] The program according to [1], wherein the plurality of annotations include first and second annotations, the first annotation is made by a first user, and the second annotation is made by a second user or was made by the first user earlier than the first annotation, and the presentation process causes a computer to generate the presentation screen so as to present to the first user classification content of the at least one label associated with the at least one region image by the first and second annotations while the first user is making the first annotation or after the first annotation is completed. [3] The program according to [1], wherein the plurality of annotations include second and third annotations, and when a first annotation is made by a first user, the second annotation is made by a second user or was made by the first user earlier than the first annotation, and the third annotation is made by a third user, or was made by the first user earlier than the first annotation, or was made by the second user earlier than the second annotation, and the presentation process causes a computer to generate the presentation screen so as to present to the first user classification content of the at least one label associated with the at least one region image by the second and third annotations while the first user is making the first annotation or after the first annotation is completed. [4] The program according to any one of [1] to [3], wherein the at least one region image is defined on the medical image by a grid or a designation frame in each annotation.[5] The program according to any one of [1] to [4], wherein the at least one region image includes a label-comparable region image, the label-comparable region image is a region image to which a plurality of labels are associated by the plurality of annotations, and the presentation process causes a computer to generate the presentation screen so as to present classification contents of the plurality of labels associated with the label-comparable region image by the plurality of annotations in a manner that allows a user to compare and visually recognize them. [6] The program according to any one of [1] to [5], wherein the at least one region image includes a label-comparable region image, the label-comparable region image is a region image to which a plurality of labels are associated by the plurality of annotations, and causes a computer to execute a process of calculating a comparison result of comparing classification contents of the plurality of labels, and the presentation screen presents the comparison result to a user to present classification contents of the plurality of labels associated with the label-comparable region image. [7] A program according to any one of [1] to [6], wherein the presentation screen presents to a user classification content of at least one label associated with multiple region images in the medical image by at least one of multiple annotations. [8] A program according to any one of [1] to [7], wherein the presentation screen has a region image and a label image, the region image is an image of the at least one region in the medical image, and the label image is an image corresponding to a label associated with the at least one region image, and the program causes a computer to execute processing to change the display mode of the region image and / or the label image on the presentation screen according to classification content and / or according to a comparison result between multiple labels when multiple labels are associated with a region image. [9] A program according to any one of [1] to [8], wherein the classification content of the label includes or is an index value, and the index value indicates how likely the classification content of the label is.
[10] A support method for supporting annotation of medical images, comprising: causing a computer to execute a presentation process for generating a presentation screen, wherein the presentation screen presents to a user the classification content of at least one label associated with at least one region image in the medical image by at least one of a plurality of annotations.
[11] A system for supporting annotation of medical images, comprising: a presentation unit for generating a presentation screen, wherein the presentation screen presents to a user the classification content of at least one label associated with at least one region image in the medical image by at least one of a plurality of annotations.
[0007] According to the present invention, it is possible to present information that is useful when making annotations on a presentation screen.
[0008] 1A and 1B are diagrams illustrating an example of a system 1 (server 1), each user terminal 3, and a diagnostic device 4 according to an embodiment. FIG. 2A is a block diagram illustrating an example of the hardware configuration of the server 1. FIG. 2B is a block diagram illustrating an example of the hardware configuration of the user terminal 3. FIG. 3A is a block diagram illustrating the functional configuration of the server 1. FIG. 3B is a functional block diagram of a presentation unit. A control flow illustrating a control operation according to an embodiment is shown. An example of a work screen 26 for performing grid-based annotation is shown. An example of a scene in which input on the presentation screen 27 is accepted to issue a presentation instruction SN1 on the work screen 26 in the case of grid-based annotation is shown. FIG. 6 illustrates an example of a presentation screen 27 in parallel presentation mode (number of calls: 1) and layer display. FIG. 6 illustrates an example of a presentation screen 27 in parallel presentation mode (number of calls: 2) and layer display. FIG. 6 illustrates an example of a presentation screen 27 in parallel presentation mode (number of calls: 2) and window display. FIG. 6 illustrates an example of a presentation screen 27 in simple presentation mode (number of calls: 1) and window display. FIG. 6 illustrates an example of a presentation screen 27 in simple presentation mode (number of calls: 2) and window display. FIG. 6 shows an example of a presentation screen 27 in comparative presentation mode (number of calls: 2) and layered display, comparing the annotation currently being worked on by the user with one other annotation. FIG. 6 shows an example of a presentation screen 27 in comparative presentation mode (number of calls: 2) and windowed display. FIG. 6 shows an example of a presentation screen 27 in comparative presentation mode (number of calls: 2) and windowed display, comparing annotations from two other users (U001, U002) other than the logged-in user (U010). FIG. 6 shows an example of a presentation screen 27 in comparative presentation mode (number of calls: 2) and layered display, comparing two other annotations other than the logged-in user. FIG. 6 shows an example of a presentation screen 27 in selective presentation mode (number of calls: 2, displaying only mismatched labels) and windowed display. FIG. 6 shows an example of a presentation screen 27 in selective presentation mode (number of calls: 2, displaying only tumor labels) and layered display. FIG. 6 shows an example of a presentation screen 27 in parallel presentation mode (number of calls: 2) and pop-up display. In the case of an annotation using the arbitrary designation frame method (bounding method), an example of a scene in which input is accepted on the presentation screen 27 to issue a presentation instruction SN1 on the work screen 26 is shown. Figure 19 shows an example of the presentation screen 27 in simple presentation mode (number of calls: 1) and layer display.FIG. 21A illustrates a portion of the presentation screen 27 in FIG. 19 in simple presentation mode (number of calls: 1) with grid setting ON and layer display. FIG. 21B illustrates a parallel presentation mode (number of calls: 1) for the case of FIG. 21A. FIG. 21C illustrates a comparative presentation mode (number of calls: 1) for the case of FIG. 21A. FIG. 22A illustrates a portion of the presentation screen 27 in parallel presentation mode (= grid setting OFF) and layer display for the case of FIG. 19 in overlapping portion display (= grid setting OFF) and layer display. FIG. 22B illustrates a comparative presentation mode (number of calls: 1) for the case of FIG. 22A. FIG. 22C shows an example of the presentation screen 27 in parallel presentation mode (number of calls: 2) for the case of FIG. 22A. FIG. 23A illustrates a portion of the presentation screen 27 in simple presentation mode (number of calls: 2) with grid setting ON and layer display for the case of FIG. 19. FIG. 23B illustrates a portion of the presentation screen 27 in parallel presentation mode (number of calls: 2) for the case of FIG. 23A. FIG. 23C illustrates an example of a comparison presentation mode (number of calls: 2) for the case of FIG. 23A. FIG. 24A is a diagram illustrating another example of grid size. FIG. 24B is a diagram illustrating another example of an arbitrarily specified frame (bounding frame). FIG. 25A is an image display variation for contrasting or comparing annotations between oneself and others. FIG. 25B is an image display variation for contrasting or comparing multiple annotations by others. FIG. 25C is an additional image display variation. FIG. 25D is an example of a label with an index value.
[0009] Hereinafter, several embodiments of the present invention will be described with reference to the accompanying drawings. The various features shown in the following embodiments can be combined with each other. Furthermore, each feature can be an invention independently.
[0010] <1. Terminology, etc.> (1-1. Annotation) In the field of machine learning, the term "annotation" refers to the act of labeling an image or a group of assigned labels, depending on the context. Annotation as an annotation act refers to "the act of associating annotation information as correct answer data with target information (digital information) for machine learning." Various types of data, such as images, videos, and audio, can be annotation targets. For example, image annotation includes object detection, region extraction, and image classification. Image annotation associates a label with an annotation target. An annotation target may be an individual image (the entire image) or an image of a region (area) within the image. Labels are also referred to as tags or metadata. Labels represent classification content (classification results) and serve as correct answer data during supervised learning. In the embodiments, "annotation" refers to "associating a label with a region image, which is an image of at least one region specified in an image," unless a different definition is specified or there is an obvious contradiction in the context.
[0011] (1-2. Images to be Annotated) In the embodiments, images to be annotated include at least "medical images." Medical images include "images that visualize the internal structure and function of the human body, primarily for the purpose of diagnosing and treating illness." Specifically, medical images may include plain X-rays, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), nuclear medicine tests, and angiography. In addition to the various images described above, medical images according to the present disclosure may also include "endoscopic images" and "pathological images." Endoscopic images may include, for example, images of the esophagus, stomach, intestines, and other digestive system images. Specifically, pathological images may be "microscopic images of tissue collected by endoscopy or surgery, (1) fixed to prevent deterioration, (2) stained for easy observation, and (3) sliced into thin slices of approximately several microns." The present invention is equally applicable to any medical image, and any label classification type and classification content can be set. In the embodiment, a pathological image is used as an example of a medical image, the classification type of the label is a classification of whether it is a tumor or not, and the classification content of the label is "tumor" or "non-tumor."
[0012] (1-3. Regions and Region Images) There are various methods for specifying regions (areas) in medical images. In the embodiments, a grid method is used in some examples, and a specification frame method is used in other examples.
[0013] The "grid method" uses a grid (i.e., a grid-like boundary line) to specify a desired position and range in an image. When the grid has multiple squares, each square can define a single area. The grid may be uniformly spaced, and each area may be specified as a rectangle of uniform size. Such areas may be called "tiles." The grid spacing may be fixed or arbitrarily specified by the user. Note that the term "grid" here has a broad meaning and is not limited to a rectangular grid. It may also be a grid in which any polygonal shape that can be tiled on a plane is periodically arranged. The grid may be, for example, a honeycomb grid. The term "honeycomb grid" here has a broad meaning and may be, for example, a regular hexagonal honeycomb, but it may also be, for example, a honeycomb structure of equilateral triangles or parallelograms, or a honeycomb structure of parallel hexagons (hexagons with three pairs of parallel sides but with unequal side lengths). For convenience, the area specified by the grid may be referred to as a "grid area." In the embodiment, grid-based annotation is adopted in the examples of FIGS.
[0014] In the "designated frame method," the user designates an area at a desired position and range in an image using an arbitrary designated frame. The designated frame is sometimes called a bounding box. The designated frame can be set to any size and any shape, and may be, for example, a rectangle, but is not limited to this, and may also be any polygon or closed curve. For convenience, the area designated by the designated frame may be called a "designated frame area." In the embodiment, the designated frame method annotation is adopted in the examples of Figures 19 to 24B.
[0015] When an area is specified in an image, the image corresponding to that area is also referred to as an "area image." When the area image is an image specified as a uniform rectangular size using a grid system, this area image may be referred to as a tile image. The size of the area and area image (number of pixels in the vertical and horizontal directions) may be the same as the size of the learning data that can be input to the learning model. When multiple designation frames are set in an image, if two designation frame areas match perfectly, the designation frame areas match each other, but if the designation frame areas differ, the designation frame areas also differ. When designation frame areas overlap, an overlapping area occurs where multiple designation frame areas overlap.
[0016] The program, method, and system of the embodiment can employ either or both of the grid method and the designated frame method. If both are employed, the method to be used for the current annotation may be set by a user setting operation and / or by a computer-generated automatic recommendation process.
[0017] (1-4. Training Data) Annotation is typically performed to construct a training dataset. When annotating medical images, the training dataset consists of a "training data image" cut out from a medical image and a label associated with the training data image. The size (number of vertical and horizontal pixels) of the training data image is set to a size that can be input to a training model. The "training data image" may be a medical image cut out along a grid, or a portion of a medical image cut out using a specified frame (bounding frame). The training data image may be a medical image cut out according to a region image at the time of annotation; that is, the region image at the time of annotation and the training data image may match. However, this is not limited to this, and the training data image may differ from the region image at the time of annotation. As an example, if a medical image is annotated using a specified frame, the training data image may be generated by cutting it out using a grid (e.g., a uniformly spaced rectangular grid). As another example, if annotation is performed using a honeycomb grid or the like, training data images may be generated by cutting out this using another grid (for example, a uniformly spaced rectangular grid).
[0018] It should be noted that cropping of medical images is not necessary during the annotation process. For example, while images of each region of a medical image are displayed on a work screen 26 and a presentation screen 27 (described later), cropping of the medical image is not necessary at this display stage. After annotation is complete, cropping of the medical image may be performed when an instruction to create a learning dataset is issued.
[0019] 2. Configuration of the embodiment (2-1. Overview of the system 1 and diagnostic device 4) As shown in FIG. 1, the system 1, user terminal 3, and diagnostic device 4 of the embodiment can communicate with each other via a wireless or wired communication network 2. The system 1 of FIG. 1 can support annotation of medical images. In the embodiment, the medical images are, for example, pathological images. The system 1 is a computer, specifically a server. The server executes a program to provide the system 1. For convenience, the system 1 may be referred to as the server 1. Users U001, U002, and U010 who use the system 1 are, for example, pathologists.
[0020] The system 1 includes a learning data generation function (learning data generation unit). In the embodiment, the learning data generation unit can generate learning data using input images (specifically, medical images). The learning data can be used as training data for training a learning model included in the diagnostic device 4. The diagnostic device 4 uses a trained model trained using the generated learning data to determine whether or not features such as pathological abnormalities are present in the medical image.
[0021] The system 1 displays on the screen a medical image stored in the system 1 and a boundary line of a region image specified on the medical image, superimposed on the medical image. The system 1 accepts annotation operations on the region images by users. That is, the system 1 accepts label inputs from users U001, U002, and U010 for each region image displayed on the screen.
[0022] The system 1 may generate training data from a single medical image, or may generate training data from multiple medical images. When generating training data from multiple medical images, the system 1 repeats the process of specifying one or more region images for the medical image and accepting label input for each medical image. Once the user has completed assigning labels to all medical images, the system 1 is ready to generate training data. To generate training data, the system 1 first cuts out the medical image to generate "training data images." The system 1 generates training data by associating the image data of each training data image with the labels assigned to each training data image (i.e., region image). The generated training data is sent to the diagnostic device 4.
[0023] The diagnostic device 4 trains a learning model using the learning data sent from the system 1. The learning model is, for example, a neural network that can be given a predetermined capability through learning. The diagnostic device 4 inputs a medical image to be diagnosed into the trained model generated by learning, and determines whether or not an abnormality exists in the medical image based on the output result from the trained model.
[0024] The system 1 can provide a presentation screen 27 as a support function for supporting annotation. The support function will be described in detail later.
[0025] (2-2. Hardware Configuration of System 1) Fig. 2A is a diagram showing an example of the hardware configuration of system 1. System 1 includes a processor 11 such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit), a storage device 12 such as a memory, an HDD (Hard Disk Drive) and / or an SSD (Solid State Drive), and a communication IF (Interface) 13 for wired or wireless communication. Via the communication IF 13, system 1 accepts user input operations from a user terminal 3 and displays various screens on a display device 34 (see Fig. 2B) of the user terminal 3.
[0026] (2-3. Hardware Configuration of User Terminal 3) As shown in FIG. 2B , the user terminal 3 includes, for example, a control unit 31 including a processor, a storage unit 32 including a non-volatile memory, a communication unit 33 for wireless and / or wired communication, a display device 34, a speaker 35, a microphone 36, a camera 37, and an operation device 38. The operation device 38 can be any input device such as a keyboard, a mouse, and / or a touch pen. The specific configuration of the user terminal 3 is not limited, and may be various types of mobile terminals, smartphones, tablet terminals, or notebook PCs. If the user terminal 3 has a touch panel display, the display device 34 and the operation device 38 may be integrated into the touch panel display. The operation device 38 and the microphone 36 serve as an input interface (or input means) for the user to perform various input operations on the user terminal 3.
[0027] (2-4. Functional Block Configuration) FIG. 3A is a diagram showing an example of the functional block configuration of the system 1. The system 1 includes a storage unit 20, a display control unit 21, an input unit 22, a generation unit 23, and a presentation unit 25. The storage unit 20 can be realized using the storage unit 12 included in the system 1. The display control unit 21, the input unit 22, the generation unit 23, and the presentation unit 25 are provided by the processor 11 of the system 1 executing a program stored in the storage unit 12. The program may be provided over a network or may be stored in a storage medium. The storage medium storing the program may be a non-transitory computer-readable medium. The non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a USB memory or a CD-ROM.
[0028] (2-4-1. Configuration of the storage unit 20) The storage unit 20 includes various databases (DBs). For example, the storage unit 20 stores an account DB, an image DB, an area DB, an annotation DB, a label DB, and a learning data DB. Each DB assigns an identification ID to the stored data, and related data can be cross-referenced.
[0029] The account DB stores each user's user ID, user details, login history, etc. The user details may include an email address, a name associated with the account (nickname, avatar icon, etc.), and / or more personal and specific information such as name and affiliation.
[0030] The image DB stores one or more medical images (e.g., pathology images) used to generate learning data. The region DB stores information identifying a region image in each medical image. As an example, the region DB stores a region image ID, a medical image ID to which the region image is set, and image coordinate data. The image coordinate data specifies the range of the medical image that each region image occupies.
[0031] The annotation DB stores annotation information for distinguishing annotations made using the system 1 together with annotation IDs. The annotation information is preferably information that can identify which medical image was annotated, when, by whom, and what type of annotation was made. The annotation information can preferably store "annotation subject information." The annotation subject information is identification information for identifying "who made the annotation (i.e., the annotation subject)." The annotation subject information may be, for example, a user ID (e.g., a login account), or may be any one or more pieces of information in the user details stored in the account DB. The annotation information may include whether the annotation can be disclosed (i.e., whether it can be disclosed on the presentation screen 27 of another person with a different user ID) and may also include disclosure conditions for the annotation (i.e., the conditions under which the annotation can be disclosed to another person).
[0032] The label DB stores information about labels associated with region images by each annotation, linked to the annotation ID. The label DB stores each label ID in association with label detail information. The label detail information includes the classification type and classification content of the label (e.g., the type is tumor classification, and the label classification content is tumor, etc.).
[0033] The training data DB stores the generated training data. A user can download any training data by referring to the training data DB.
[0034] It is preferable that at least the image DB, area image ID, annotation ID, and label ID are linked to each other. This allows the content of any annotation (i.e., medical image, area image, label, etc.) to be easily retrieved from the annotation DB by specifying the annotation ID. Note that the above data management and DB configuration are merely examples. Any data management and DB configuration can be adopted depending on the specifications of the actual hardware and / or software, etc.
[0035] (2-4-2. Configuration of Other Functional Units) The display control unit 21 executes display control to display various screens on the display device 34 of the user terminal 3. For example, the display control unit 21 displays a work screen 26 in which the boundary line of the area image is superimposed on a medical image. For example, the display control unit 21 controls to provide a presentation screen 27 (see FIGS. 4, 6 to 18) generated by the presentation unit 25 (see FIG. 3B) together with the work screen 26 or independently from the work screen 26.
[0036] The input unit 22 accepts various inputs from the operation device 38 of the user terminal 3. The input unit 22 accepts annotations (label inputs) from the operation device 38 of the user terminal 3. The label input is an input operation for associating a label with each of a plurality of area images. The input unit 22 stores the labels associated with each area image in the label DB. For example, the input unit 22 may associate an area image ID that uniquely identifies each area image with the label associated with each area image and store the associated label in the label DB. The label input may be performed, for example, by selecting an area image for which a label is to be input from among the area images displayed on the screen and accepting specification of a label to be associated with the area image.
[0037] As an example, the generator 23 generates learning data for training a learning model by associating each of a plurality of region images with a label associated with the region image. For example, the generator 23 acquires a region image ID and a label from the label DB, and extracts image data of image coordinates corresponding to the region image ID from a medical image stored in the image DB. This extracted image data is a region image. The generator 23 can generate learning data by combining the image data of the extracted region image with the label stored in association with the region image ID.
[0038] 3B shows a functional block configuration of the presentation unit 25. The presentation unit 25 includes a data acquisition unit 25a, a presentation screen generation unit 25b, a comparison unit 25c, and a selection unit 25d. The data acquisition unit 25a can acquire information necessary for generating a presentation screen 27 (see FIGS. 4 to 18) from the storage unit 20. The presentation screen generation unit 25b can generate the presentation screen 27 based on the information acquired by the data acquisition unit 25a. The comparison unit 25c and the selection unit 25d can calculate and / or select the information acquired by the data acquisition unit 25a in a specific presentation mode (comparison presentation mode, selection presentation mode), and supply the information subjected to the calculation, etc. to the presentation screen generation unit 25b.
[0039] (2-5. Details of the Work Screen 26) Each of Fig. 5 and Fig. 19 is an example of the work screen 26 provided by the system 1. Fig. 5 is an annotation work screen using a grid system, and Fig. 19 is an annotation work screen using a designated frame system. The work screen 26 is a GUI used when a user performs annotation work. As an example, the work screen 26 includes selection menus M10, M11, and M12, a display area W10, an overview display area W11, a display frame V11, a display frame T10, a button B10 for generating learning data, and a button B11 for referring to annotations.
[0040] The selection menu M10 displays the type of medical image (pathological image, for example) currently being displayed. The selection menu M11 is a menu for specifying the medical image (pathological image, for example) to be displayed on the work screen 26 when multiple medical images have been input. In the examples of Figures 5 and 19, 40 pathological images relating to tumor cells have been input, and the third pathological image is currently being displayed.
[0041] The selection menu M12 allows the user currently annotating to select the disclosure setting (access rights) of the annotation to other users. The selection menu M12 allows at least one of disclosure allowed and disclosure not allowed to be selected. If disclosure not allowed, only the user who made the annotation can use the annotation. Use includes, for example, one or more of calling, referencing, duplicating, editing, and creating learning data. If disclosure allowed, other users besides the user who made the annotation can also use the annotation. If disclosure allowed, the disclosure conditions can also be selected. The disclosure conditions can be selected from "user disclosure" and "user confidentiality." The selection result of the selection menu M12 is registered in the annotation DB. If user disclosure is selected, when another user references the annotation via the presentation screen 27, the annotation subject information of the user who made the annotation is displayed on the presentation screen 27.
[0042] The overview display area W11 displays the entire medical image and a display frame V11. The display frame V11 specifies a region of the medical image that is to be enlarged and displayed in the display area W10. The user can change the position and size of the display frame V11 as desired.
[0043] In the example of Fig. 5, a grid (grid-like boundaries) is displayed overlaid on an enlarged image of a part of a medical image in the display area W10. In the example of Fig. 5, one square of the grid corresponds to one "region image." In Fig. 5, as an example, one region image has a width W g and height H gAs described above, in the embodiment, a computer (i.e., the system 1) executes a process for generating a display area W10 in which a grid (boundaries of multiple area images) is superimposed on a medical image. Each square of the grid is stored in the area DB in association with the position coordinates of each area image in the medical image. In the example of FIG. 5 , a display frame T10 is displayed in the display area W10. The display frame T10 indicates the position of the area image that accepts label input from the user. The position of the display frame T10 can be arbitrarily changed using the operation device 38. The display frame T10 can be placed in an arbitrary position, and labels can be input by additional operations on the operation device 38.
[0044] In the example of FIG. 19 , several designation frames and labels are input on an enlarged image of a portion of a medical image in the display area W10. One designation frame can designate one "region." If the position and range of the designation frame differ between annotations, they are different regions. In the embodiment, the operation device 38 includes a pointing device (a mouse, for example). For example, the designation frame may be set by mouse operation, and for example, the click position within the display area W10 may be set as the vertex of the designation frame. The designation frame is stored in the region DB in association with the position coordinates of each region image in the medical image. For example, clicking on the designation frame activates (selected state), and a label can be input by additional operation of the operation device 38.
[0045] For region images for which label input has been completed, information indicating the classification of the input label is displayed superimposed on the region image. In the embodiment, as an example, the label classification is displayed using the letters "T" and "N." That is, if a region image is determined to be an image of a tumor cell and, as a result, a label indicating a tumor cell is assigned to the region image, the letter "T" is displayed in a predetermined position (for example, the upper left) for the region image. Also, if a region image is determined not to be an image of a tumor cell and, as a result, a label indicating a non-tumor cell is assigned to the region image, the letter "N" is displayed in a predetermined position (for example, the upper left) for the region image.
[0046] The display control unit 21 may display a performance value of the number of area images to which labels have been assigned in the display area image N11 of the work screen 26. This performance value may be associated with each account and stored in a user account DB of the storage unit 20. As an example, the storage unit 20 may total the performance values of all medical images for each user. As an example, the storage unit 20 may store the performance value in association with each medical image, thereby making it possible to visually recognize the difference in performance values between users for the same medical image.
[0047] 3. Processing Flow of the Embodiment (3-1. First Example) A first example of the processing flow of the embodiment will be described using FIGS. 4 to 7. The first example is grid-based annotation. When the server 1 executes a program constructed according to the control flow of FIG. 4, annotation of medical images can be supported, and an annotation support method is provided.
[0048] To summarize the flow of Fig. 4, after the login process (S200, S100), annotation work (S201, S101, S202, S102) is performed using the work screen 26 of Fig. 5. In accordance with the user's instructions, the system 1 presents a presentation screen 27 to the user (S203, S204, S103, S104, S205, Figs. 5 and 6). By generating the presentation screen 27 shown in Figs. 7 to 18, the user can refer to the classification content of at least one label associated with at least one region image in a medical image by multiple annotations.
[0049] Each step will be described in detail below. First, in Fig. 4, a user U010 logs in to the system 1 (S200, S100). In the following description, the user U010 may be simply referred to as the "user."
[0050] Assume that after logging in, the user requests an annotation work screen (S201). In this case, the input unit 22 accepts from the user a selection of medical images to be used to generate learning data. Note that the system 1 may set access rights to each medical image for each user account in the image DB 20. The user can select any medical image stored in the image DB, or can select only medical images for which access rights have been granted according to the login account.
[0051] In step S101, in response to a user's annotation work screen request, the system 1 outputs a work screen 26 (see FIG. 5 ) corresponding to the medical image and label classification type selected by the user. The display control unit 21 displays the work screen 26 in which a grid is superimposed on one medical image selected by the user. The grid is the boundary line of an area. As an example of the grid in this embodiment, a rectangular grid with orthogonal vertical and horizontal axes is used, which defines multiple rectangular cells, and one rectangular cell defines one area image. After outputting the work screen 26, the system 1 enters a label assignment reception mode.
[0052] The label classification type is, for example, information regarding the type of pathology. As an example, the pathology type may be either "tumor" or "hypermutation." For example, if each medical image is an image suspected of having a tumor, the pathology type may be set to "tumor," and in that case, the label classification content may be "tumor" or "non-tumor." If each of multiple medical images is an image suspected of having a hypermutation, the pathology type may be set to "hypermutation," and in that case, the label classification content may be "hypermutation" or "non-hypermutation." The label classification type to be applied to the selected medical image may be automatically set by the input unit 22 in accordance with the menu M10. Alternatively, the input unit 22 may receive a user's selection of the label classification type to be applied to the selected medical image.
[0053] The user operates the user display frame T10 to input labels to each area image. In Fig. 5, the user U010 has finished inputting labels to some of the area images. The input unit 22 can accept label input for each area image.
[0054] In step S202, the user terminal 3 transmits the user's labeling operation from the operation device 38 to the system 1 each time. If the medical image is an image of a tumor, the label assigned by the user to each region image is either a label indicating that the region image is an image in which the presence of a tumor is suspected, or a label indicating that the region image is not an image in which the presence of a tumor is suspected. Also, if the medical image is an image of hypermutation, the label assigned by the user to each region image is either a label indicating that the region image is an image in which the presence of hypermutation is suspected, or a label indicating that the region image is not an image in which the presence of hypermutation is suspected.
[0055] In step S102, the input unit 22 associates the label with the area image ID based on the label assignment operation information from the user terminal 3, and stores the association information as provisional registration information.
[0056] During the annotation work (S201 to S102), the user U010 may want to refer to other annotations. In this case, the user can click the annotation reference button B11 on the user terminal 3.
[0057] In response to clicking the button B11, the determination result in step S203 becomes affirmative (YES), and the presentation screen 27 illustrated in FIG. 6 is launched. At this launch stage, label classification information has not yet been presented. At the same time, a presentation target frame V12 is displayed on the work screen 26. In the embodiment, as an example, the display control unit 21 displays the presentation target frame V12 in the display area W10. The presentation target frame V12 determines the target range of the presentation screen 27 (i.e., the area image in which labels based on other annotations are presented). The user can select one or more areas within any range within the display area W10 by operating the operation device 38, and instruct the system 1 to set or update the presentation target frame V12.
[0058] By entering necessary information into the input fields of the presentation screen 27 in FIG. 6 , the user can send a presentation instruction SN1 (see step S204 in FIG. 4 ) to the system 1. In the embodiment, this presentation instruction SN1 includes, as an example, a presentation range, a presentation mode, and call information. The "presentation range" can be set by the user arbitrarily selecting an area image using the presentation target frame V12 in FIG. 6 , for example. The "presentation mode" can be selected from simple presentation, parallel presentation, comparative presentation, and selective presentation, for example. The "call information" specifies "which annotation to reference" and "how many annotations to reference," and specifically includes a "call number" and a "call annotation ID." The presentation screen 27 in FIG. 6 has three input boxes (selection menus) for inputting the presentation mode, the call number, and the call annotation ID.
[0059] An example of the presentation instruction SN1 may be as follows: The area images within the presentation target frame V12 are to be presented, the number of area images is 6 x 3 = 18 in total, the presentation mode is "parallel mode," the call count is "1," and the call annotation ID is "U001-1" (an example of these settings is shown in FIG. 7 ). Note that, as an example, the system 1 of the embodiment allows one user to annotate the same medical image multiple times. The annotation ID "U001-1" can be used to call up the label from user U001's first annotation. Setting the call annotation IDs to "U001-1," "U001-2," etc., makes it possible to call up user U001's first, second, etc. annotations. Note that the call count can be set to any number of two, three, or more. An example in which the call count is two will be explained later.
[0060] In response to the presentation instruction SN1, the system 1 starts the process of generating the presentation screen 27 (S103, S104).
[0061] First, the data acquisition unit 25a references the storage unit 20 according to the call annotation ID based on the call information of the presentation instruction SN1. The data acquisition unit 25a references at least the image DB, area DB, and label DB of the storage unit 20, thereby reading out the labels of each area image to be presented on the presentation screen 27 (S103). As an example, the data acquisition unit 25a sequentially references the area image IDs of each area image in the presentation target frame V12. The data acquisition unit 25a can extract all labels associated with one area image ID by searching the label DB using the area image ID as a search key. The data acquisition unit 25a compares the annotation IDs of each extracted label with the call annotation IDs of the presentation instruction SN1. The data acquisition unit 25a retains labels that match between both accounts as they are to be presented, and excludes mismatched labels. This makes it possible to extract only the labels to be presented from the label DB.
[0062] In step S104, the presentation screen generation unit 25b generates the presentation screen 27 for displaying the label classification content of each area image based on the information acquired by the data acquisition unit 25a. The presentation screen generation unit 25b transmits data for displaying the presentation screen 27 to the user terminal 3.
[0063] When receiving the data for the presentation screen 27 from the system 1, the user terminal 3 updates the currently displayed presentation screen 27 (S205). FIG. 7 shows an example of the updated presentation screen 27. As can be seen by comparing FIG. 6 with FIG. 7, the presentation screen 27 in the example of FIG. 7 displays each label in layers superimposed on each area image within the presentation target frame V12 of the display area W10. This allows the user U010 to compare the labels he or she is currently annotating with the labels annotated by another user U001. Note that only the labels of the user U001 are displayed on area images that have not yet been labeled by the user U010 currently annotating.
[0064] After that, the user U010 labels the medical image by a predetermined number (or a predetermined range), and the time comes to end the annotation work. In the embodiment, the work ends when the user presses button B10, for example. When the user clicks button B10, the work ends (S206), a label confirmation instruction is sent to system 1 (S207), and system 1 registers all labels that were provisionally registered in step S102 in the label DB (S105).
[0065] At step S105, the generation unit 23 generates training data and stores it in the training data DB. As an example, the medical image may be cropped according to the grid of the work screen 26, so that the size of the region image matches the size of the training data image. The training data is composed of each training data image and each label (see FIG. 5 ) that the logged-in user U010 has associated with each region image on the work screen 26.
[0066] (3-2. Second Example) A second example of the processing flow of the embodiment will be described using Figures 19 and 20. The second example is a designated frame annotation. Like the first example, the second example is also processed by a computer (system 1) according to the processing flow of Figure 4, but there are differences in steps S101, S202, and S104. In step S101, the work screen 26 of Figure 19 is provided to user U010. In step S202, user U010 sets a designated frame and inputs a label to the medical image via the work screen 26 of Figure 19. Note that in the example of Figure 19, annotation disclosure is permitted, and the disclosure conditions are set to user confidentiality.
[0067] In the second example, as in the first example (FIG. 6), the user can activate the presentation screen 27 and input necessary information into the input fields of the presentation screen 27, thereby transmitting a presentation instruction SN1 (see step S204 in FIG. 4) to the system 1. However, in the second example, the presentation screen 27 in FIG. 20 is presented to the user in step S104. A difference from the first example (FIGS. 6 and 7) is that annotations (annotation IDs = U010-3, U001-3) using the designated frame method by the logged-in user (U010) and another user U001 are called.
[0068] 19 and 20, the presentation screen 27 in the example of Fig. 20 displays several overlapping designation frames in layers within the display area W10. Each designation frame is labeled, and the user can visually recognize the classification content of the label associated with the region image designated by each designation frame in the medical image.
[0069] Thereafter, as in the first example, when the process reaches step S105, the generator 23 generates training data and stores it in the training data DB. At this time, in the second example, there are variations in the method of cutting out medical images. As an example, a medical image may be cut out according to each designation frame on the work screen 26 of FIG. 19 , and the cut-out image may be used as the training data image. As another example, as illustrated in FIG. 21A , a grid may be further superimposed on the designation frame, and the area image within the designation frame may be cut out along the grid. In this way, training data can be constructed in the second example as well.
[0070] 4. Variations of Presentation Screen 27 and Presentation Processing, etc. Figure 7 (grid system) and Figure 20 (designated frame system) are merely examples of the presentation screen 27, and at least Figures 8 to 18 and Figures 21 to 24B are other examples. There are various variations in the presentation mode and display method of the presentation screen 27, and there are also various variations in the appearance of the area image and label image. There are also various variations in the processing related to the presentation processing.
[0071] (4-1. Variations of Presentation Mode) The presentation mode of the embodiment can be at least any one of a simple presentation mode, a parallel presentation mode, a comparative presentation mode, and a selective presentation mode. Of these, only one mode may be adopted, or multiple modes may be adopted in a switchable manner.
[0072] Examples of the "parallel presentation mode" are shown in Figures 7 to 9, 18, 21B, 22A, 22C, and 23B. In this mode, multiple labels associated with region images are grouped together and arranged in parallel for each region. The arrangement direction may be vertical, horizontal, or diagonal, and the labels may partially overlap.
[0073] The parallel presentation mode of Fig. 21B will be described. First, refer to Fig. 21A. Fig. 21A is an enlarged view of the lower left region of Fig. 20. As an example, region image conversion may be performed by further superimposing a grid on the specified frame.
[0074] This "area image conversion" refers to conversion between a designated frame area and a grid area. For area conversion, for example, the label of the designated frame is associated with the area defined by the grid within the extent of the designated frame. For example, if the overlapping area between the designated frame area and the grid area is equal to or greater than a "predetermined overlap amount," the grid area image may be associated with the same label as the designated frame area image. The "predetermined overlap amount" may be, for example, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or 100% of the area of the grid area, or may be within a range between any two of the values exemplified here. When the predetermined overlap amount is 100%, a grid area and the designated frame area completely overlap. In FIG. 21B , which shows the result of area image conversion, a group of T-labeled areas (3 rows and 2 columns) and a group of N-labeled areas (2 rows and 3 columns) partially overlap. From this point on, as in FIG. 7 , labels can be arranged in parallel in each grid area image according to a predetermined rule. A display similar to that shown in FIG. 21B may be displayed over the entire display area W10 of FIG.
[0075] In Fig. 22A, as an example, only the overlapping area where multiple designation frames overlap each other is set as the area to be presented. Fig. 22C is similar, but as an example, three or more designation frames overlap. In Fig. 22C, as an example, the position of the mouse pointer T110 is set as the active area, and labels are displayed in parallel only in the active area, which may improve visibility. Fig. 23B shows the situation in Fig. 23A where three designation frames overlap, after area image conversion has been applied to Fig. 23A.
[0076] Examples of the "simple presentation mode" are shown in Figures 10, 11, 20, 21A, 23A, and 24A-24B. In this mode, a set of area groups and labels is generated for each annotation, and the sets are arranged separately for each annotation. For example, Figure 10 displays area images within the presentation target frame V12 in a separate window above the display area W10, and displays labels based on annotations by the second user U001. Figure 11 displays two area images within the presentation target frame V12 side by side, displaying labels based on annotations by the second user U001 and the third user U002.
[0077] 20, Figures 21A, 23A, and 24A to 24B are also in the simple presentation mode, but the grid display is ON in Figures 21A, 23A, and 24A. Note that there is no limitation on the grid size here, and it may be set smaller as in Figure 24A, for example, or may be adjustable by the user, for example.
[0078] Figure 24B shows a non-rectangular designation frame, for example, a polygonal designation frame. Grid overlay and area image conversion can also be applied to Figure 24B. If the parallel presentation mode is adopted in Figure 24B, only the overlapping areas of multiple designation frames may be presented, as in Figures 22A and 22C.
[0079] In both the parallel presentation mode and the simple presentation mode, multiple labels associated with label-comparable area images can be displayed in a comparable manner. A "label-comparable area image" is an area image to which multiple labels are associated by multiple annotations. A "label-comparable area image" is an image of an overlapping portion of multiple area images to which multiple labels are associated, or an identical area image to which multiple labels are associated, and it is possible to compare the multiple associated labels. As in these modes, a computer may execute a presentation process to generate a presentation screen 27 so that the classification contents of the multiple labels associated with the label-comparable area images are displayed in a manner that allows the user to visually compare them.
[0080] Examples of the "comparison presentation mode" are shown in FIGS. 12 to 15, 21C, 22B, and 23C. In the comparison presentation mode, the computer (server 1) may execute a process (processing of the comparison unit 25c) to calculate a comparison result obtained by comparing the classification contents of multiple labels associated with label-comparable area images. The presentation screen 27 may present the comparison result to the user, thereby presenting the classification contents of multiple labels associated with label-comparable area images. In this comparison presentation mode, instead of displaying the labels themselves, the user is presented with the comparison results obtained by comparing multiple labels. The "comparison" may be a match or mismatch for labels of the same type, or a correlation or non-correlation determined according to a predetermined rule for labels of different types. When there are three or more comparison targets, the comparison result may be displayed as unanimous or mismatch, or as a majority in favor, a majority against, or a balanced result. 12 to 15, 21C, 22B, and 23C, as an example, when the labels match, "COMT" is displayed if all the labels are "T", and "COMN" is displayed if all the labels are "N". When the labels do not match (are different), "DIFF" is displayed.
[0081] There is no upper limit to the number of annotations that can be presented on the presentation screen 27, and it can be set arbitrarily. Therefore, there is no upper limit to the number of labels that can be displayed for comparison in the parallel presentation mode or the simple presentation mode, and it can be set arbitrarily. Similarly, there is no upper limit to the number of labels that can be compared with each other in the comparison presentation mode, and it can be set arbitrarily.
[0082] Examples of the "selective presentation mode" are shown in FIGS. 16 and 17. In this mode, the selection unit 25d selects labels to be presented according to a selection item specified by the user, and the presentation screen generation unit 25b generates the presentation screen 27. For example, selection may be based on the label classification content, or the presentation screen 27 may be generated by targeting only area images with mismatched labels (FIG. 16). Selection may also be based on label type. For example, the presentation screen 27 may be generated by targeting only area images with a tumor label ("T" label) based on other annotations (FIG. 17). The selective presentation mode can also be employed for annotations using a designated frame system. For example, in FIGS. 22A, 22C, 23A, etc., only area images with mismatched labels may be presented, or only designated frames with a specific label classification content (e.g., only the T label) or a specific label classification type (e.g., only tumor / non-tumor labels) may be presented.
[0083] The presentation mode does not have to be selectable from multiple modes, and may be only one predetermined mode. In this case, the function for specifying the presentation mode can be omitted. Furthermore, the presentation instruction SN1 does not have to be able to selectively set call information (call annotation ID). The function for setting call information may be omitted, and all annotations for the medical image to be presented may be automatically called, or all labels associated with the area image to be presented may be automatically called. In this case, the specification of the annotation ID can be omitted. In these cases, since a large number of labels are usually associated with one area image, the number of labels assigned for each label content (e.g., the number of tumor labels assigned, the number of non-tumor labels assigned) may be expressed numerically (see FIG. 25C ).
[0084] (4-2. Variations in Display Method) The display method of the embodiment can be any one of at least layer display, window display, and pop-up display. Only one display method may be used, or multiple display methods may be switchably used. Any presentation mode and any display method can be combined.
[0085] Examples of "layer display" are shown in Figures 7, 8, 12, 15, 17, and 20 to 24B. As can be seen from a comparison of Figures 5 to 7, the presentation screen 27 is overlaid as a layer on the display area W10 of the work screen 26. In other words, the presentation screen 27 displays labels for each area image overlaid on the display area W10. Each label image may be transparent to make the area image easier to view, but may also be opaque.
[0086] Examples of "window display" are shown in Figures 9 to 11, 13, 14, and 16. A presentation screen 27 is displayed as a separate window on top of the work screen 26. Note that in these figures, the display area W10 is a photograph, while the area image of the presentation screen 27 is shown in an illustration style, but this is merely for convenience of illustration and does not limit the present invention. The area image of the presentation screen 27 may be the same as the photograph in the display area W10. This also applies to the pop-up display described below. A window display can also be used for annotations using the designation frame method, and the medical image, designation frame, and label of another annotation can be displayed in a separate window.
[0087] An example of a "pop-up display" is shown in FIG. 18. The label classification details for only the selected area image are displayed in a pop-up window near the selected area image. As another example, the pop-up display shown in FIG. 20 may be employed. For example, when the mouse pointer T110 is placed at the lower left side of the display area W10 as shown in FIG. 20 and a pop-up display command is issued near the specified frame, a display such as that shown in FIG. 21B, FIG. 22A, or FIG. 22B may be displayed in a pop-up window.
[0088] (4-3. Number of Annotations, Area Images, and Labels That Can Be Presented) In the embodiment, the presentation screen 27 presents to the user the classification content of at least one label associated with at least one area image by a plurality of annotations. The terms "at least one area image" and "at least one label" are intended to include the following aspects. In the embodiment, any combination of annotation methods (grid method or designated frame method), presentation modes, display methods, and various other modified examples can be adopted, and the number of area images that the presentation screen 27 presents can be changed in various ways depending on the combination. The number of labels also changes depending on whether or not there are missing labels. Missing labels are caused, for example, by omitting to enter a label in the annotation.
[0089] Specifically, for example, the pop-up display (see FIG. 18 ) of the embodiment can present the user with the classification information of at least one label for one area image. While two labels are illustrated in FIG. 18 , if a label is missing from one of the multiple annotations (U010-1, U001-1), the number of labels associated with one area image may be less than two. Thus, the presentation screen 27 employing the pop-up display (see FIG. 18 ) can present at least one label classification information for one area image. Furthermore, even if the presentation target frame V12 selects only one area image in layer display or window display, the same behavior is achieved as with the pop-up display. As another example, for example, if two designation frames are set in substantially the same position and range in FIG. 21A , one area image is labeled with multiple annotations. As yet another example, for example, each designation frame in FIG. 21A may be smaller than shown, substantially the same size as the grid size. In this case, a situation may occur in which two designation frames surround one grid-based area image, and a label is assigned only to that area image. In this case, a missing label may also occur. Even in this situation, the presentation screen 27 can present at least one piece of label classification information for one area image.
[0090] On the other hand, in some examples of the embodiment, the presentation screen 27 can present to the user the classification contents of multiple labels associated with multiple region images in a medical image by multiple annotations. For example, in the embodiment, the presentation screen 27 can be generated when multiple region images are selected in the presentation target frame V12, or for the entire display area W10 including the multiple region images, or by a pop-up display for the multiple region images. In this case, the classification contents of "multiple labels" associated with "multiple region images" can be presented to the user.
[0091] Alternatively, for example, if there are no missing labels in the multiple annotations, multiple labels are associated with all the area images without omission by the multiple annotations. In this case, too, the presentation screen 27 can present to the user the classification contents of the multiple labels associated with the multiple area images by the multiple annotations.
[0092] As a modified example, the computer (system 1) may notify the user that a label is missing, for example, at step S104 in the processing flow of FIG. 4 . For example, a label may be missing from a specific region image in some or all of the multiple annotations. In that case, for example, at step S104, the computer (system 1) may present a notification signal (such as an error message) to the user instead of or in addition to the presentation screen 27, or may display the specific region image in a prominent manner.
[0093] The presentation target frame V12 may be omitted. In this case, for example, the entire area image of the display area W10 may always be the target. In this case, the presentation screen 27 can also present to the user the classification contents of multiple labels associated with multiple area images by multiple annotations.
[0094] Note that, for convenience, the embodiment illustrates first to third annotations. However, there is no upper limit to the number of annotations that can be presented on the presentation screen 27, and there is no upper limit to the number of labels that can be presented in association with one area image on the presentation screen 27. The system 1 can store any number of annotations by any number of users in the storage unit 20, thereby associating any number of labels with one area image ID in the label DB. The system 1 may read any number of annotations (sets of area images and labels) of three or more from the image DB and the label DB, and thereby present the classification contents of any number of labels (sets of three or more) for each area image on the presentation screen 27.
[0095] Here, the maximum number of annotations that the system 1 can present on the presentation screen 27 is defined as "Z." The value of Z may be, for example, 1 to 100, for example, 2 to 20, or for example, 3 to 10. Specifically, the value of Z may be, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, 75, or 100, and may be within the range of any two of the values exemplified here. If there are no missing labels for each annotation, as the number of annotations increases, the number of labels that can be associated with one area image also increases proportionally. Therefore, for example, if Z = 5, a maximum of five labels can be associated with one area image on the presentation screen 27.
[0096] The system 1 of the embodiment may be used by a limited number of members, for example, only members of a certain company, laboratory, or research group. However, the system 1 is not limited to this. The system 1 may accept user account registrations from members of many different research institutions, universities, or companies, or may be accessible to users from all over the world, and therefore may be made widely available to the public via the communication network 2. This has the advantage of making it possible to collect annotation data from many different users. However, in this case, not all users are necessarily willing to disclose their user information. To address this issue, the system 1 of the embodiment is equipped with a function for restricting annotation disclosure and / or a "function for displaying / hiding annotator information," which will be described later.
[0097] (4-4. Adding performance values, displaying / hiding annotation subject information) The "performance values" may be reflected on the presentation screen 27. The performance values are the cumulative number of labels assigned for each user. The "performance values" may be incorporated in any presentation mode. For example, labels associated with users with high performance values may be highlighted more than labels associated with users with low performance values. In a selective presentation mode, only labels of users with performance values above a certain value may be selected and presented.
[0098] Annotation subject information may be displayed on the presentation screen 27. The annotation subject information is information that can identify the annotation subject who input each label. For example, in FIG. 8, "U001-1" and "U002-1" are displayed in the input box for the call annotation ID, and this functions as the annotation subject information. For example, in FIG. 11, there are two sets of area images and label display area images in the presentation screen 27. One area image is marked with "U001-1" and the other area image is marked with "U002-1," making the annotation subject information easily distinguishable. This is just one example, and other examples of annotation subject information may include the user's name, affiliation, nickname, or avatar icon, etc., that can be displayed on the presentation screen 27.
[0099] However, on the other hand, the annotation subject information may be hidden. As an example, the subject information of the called annotation may not be displayed on the presentation screen 27. For example, when the call mode is the full call mode, the annotation subject information may not be displayed at all. An annotation set to "Not Disclosed" in the selection menu M12 of FIG. 5 or the like may be made callable on the condition that the annotation subject information is hidden. Hiding the annotation subject information makes it possible to conceal the annotation subject. Even if a user is not willing to disclose their user information, there is an advantage in that the user's annotation data can be utilized while ensuring anonymity. Examples of utilizing annotation data while ensuring anonymity will be explained later in the call mode variations.
[0100] (4-5. Variations in Display Mode, Label Appearance, etc.) There are various variations in the means for distinctively displaying the classification content of the label associated with "at least one region image in a medical image." For example, as shown in Figures 25A to 25D, there are various variations in the appearance of the label.
[0101] Images L1a to L4b shown in FIG. 25A can be used to compare and contrast annotations between the user and others, or multiple annotations of the user. Image L1a is an example of a label display for the user (logged-in user), and image L1b is an example of a label display for others. The two can be displayed side by side, shifted vertically. The same display format can be used for the user's label and the other user's label, and the colors do not need to be different. Images L2a and L2b have the other user's labels in the same color, while images L3a and L3b have the other user's labels in different colors (T is black, N is white). Images L4a and L4b have comparison labels (match / mismatch) in different characters and colors.
[0102] FIG. 25B illustrates an example of comparing or contrasting multiple annotations of others. As in images L5a to L6b, multiple other labels may be displayed in parallel below the self-label. As in images L7a to L8b, comparison result labels may be displayed below the self-label, and COMT and COMN may be displayed in different colors. When presenting three or more other-labels in a comparative manner, the classification contents of the labels associated with the area images may be displayed in a visually distinguishable manner. For example, as in image L9, they may be expressed using numerical values, and in this example, there are 13 "T" labels and 7 "N" labels.
[0103] In Figure 25C, images L11a to L11c display only matching labels highlighted with a bold frame, while images L12a to L12c display only mismatching labels highlighted with a bold frame. As in images L13a and L13b, any number of labels may be displayed partially overlapping. Image L16 is a modified version of image L9.
[0104] To generalize or expand the above example, the "display mode difference" may be, for example, a static and / or dynamic difference in the appearance of a region image or a label image. For example, the "static difference" may be, for example, highlighting or color difference on a specific region image or a specific label, or, for example, overlaying any additional image other than a label on a specific region image. The "additional image" may be an image representing any symbol, color, and / or pattern. The "symbol" may be, for example, a character symbol, a mathematical symbol, or a graphic symbol. A label may be applied with a different shape, color, and / or pattern. The highlight is a technique for making a region image or a label stand out using any method, such as a bold frame or an emphasis display. The region image may be applied with a different color and / or transparency. The color difference may be, for example, monochrome / color color coding or color coding using any multiple colors. The transparency difference may be, for example, such that one area image or label image is displayed with transparency, while the remaining area image or label image is displayed with a non-transparent display, so that the user can distinguish and view these area images. As an example, the "dynamic difference" may be to add animation (display movement) to a specific area image or a specific label image, and the animation may include blinking, repeated changes in transparency, and / or repeated changes in size.
[0105] As described above, when the presentation screen 27 includes area images and label images, the display mode of the area images and / or label images may be changed on the presentation screen 27. Specifically, the computer (server 1) may be caused to execute a process (S104) of changing the display mode of the area images and / or label images according to the classification content of the labels and / or the comparison result of a plurality of labels associated with one area image.
[0106] (4-6. Addition of Index Values) As shown in FIG. 25D , the classification content of a label associated with a region image to be annotated may include an index value. For example, the index value may be displayed as a subscript to “T” or “N” in images L17a and L17b. The index value indicates the degree of accuracy of the classification content of the label. For example, the index value may be manually input by the user during annotation work on the work screen 26. The index value can be used for both grid-based and designated frame-based annotation. The classification content of the label may be an index value. For example, in the display example of images L17a and L17b, a value closer to 100 indicates a 100% probability of tumor, and a value closer to 0 indicates a 0% probability of tumor, i.e., non-tumor. The index value may be arbitrarily set by the user in any increment (e.g., a fixed increment such as 1% or 5%). Alternatively, the index value may be presented as, for example, a plurality of options (e.g., an arbitrary number of levels such as three or four levels) from which the user can select a desired index value. For example, three levels of index values may be used: "high" (e.g., 70% or more), "medium" (e.g., 31% to 69%), and "low" (e.g., 30% or less). This has the advantage of improving the convenience of user input of index values. In the comparison presentation mode, calculations can also be performed between index values. As an example, a calculated value (e.g., total, average, etc.) may be calculated for the index values of multiple "T" labels, and the calculated value may be displayed on the label. Alternatively, the similarity between multiple labels may be calculated in the comparison presentation mode. The similarity may be displayed, for example, by displaying the difference in similarity between the labels as a numerical value on the label, or by displaying the result of a determination on the label based on whether the difference in index value between the labels is equal to or less than a predetermined value.
[0107] (4-7. Call Mode Variations) The system 1 of the embodiment can be equipped with various call modes. As an example, the call mode of the embodiment can be any one of an annotation designation call mode, an all call mode, and a layer-specific call mode. The user may selectively use two or more of these modes. Any call mode can be adopted together with the annotation method of the embodiment (grid method or designation frame method), each presentation mode, each display method, and various other modified examples.
[0108] In the embodiment, the annotation designation call mode is mainly described, in which an annotation ID is input to the presentation screen 27 (see, for example, FIGS. 7 and 20). The all call mode is a mode in which all annotations associated with a medical image at the time of call are automatically called and presented on the presentation screen 27.
[0109] The stratified call mode is a mode in which only annotations that meet specified conditions are extracted and presented on the presentation screen 27. Various types of stratification can be set. For example, stratification may be based on whether or not the annotation subject information meets a predetermined condition. For example, only annotations whose performance values are equal to or greater than a predetermined value may be extracted, thereby making it possible to present only annotations with high performance and reliability. For example, a period may be specified as one of the call conditions, and only annotations registered (specifically, newly registered or last edited) within the specified period may be extracted. As another example, index values may be used. For example, only annotations whose summary statistics of all labels within the annotation (e.g., the average index value or the median index value of all labels) are equal to or greater than a predetermined threshold may be extracted, thereby making it possible to extract only annotations created by the annotator with a certain degree of confidence.
[0110] As an example, the system 1 of the embodiment is configured to be switchable between a "disclosure mode" in which annotation subject information is displayed on the presentation screen 27 and a "non-disclosure mode" in which annotation subject information is not displayed. In the disclosure mode, any call mode can be adopted. Note that in the disclosure mode, annotations set to "disclosure allowed / user confidential" in the selection menu M12 of FIG. 5 or the like may be prohibited from being called in the annotation DB.
[0111] In contrast, in the non-disclosure mode, there are limitations on the call modes that can be used. In the non-disclosure mode, at least, a mechanism that allows annotation subject information to be recognized, such as the annotation specification call mode that requires input of an annotation ID including a user ID, cannot be used. The all call mode and layer call mode can be used even in the non-disclosure mode, and in these call modes, it is sufficient to prohibit the display of annotation subject information on the presentation screen 27, etc.
[0112] 5. Features and Effects of the Embodiments The features and effects of the embodiments can be summarized as follows, for example.
[0113] (5-1) In the embodiment, the computer (system 1) executes the process of generating the presentation screen 27 (S103, S104). The presentation screen 27 presents to the user U010 the classification content (e.g., "T" or "N") of at least one label associated with at least one region image in the medical image by at least one of the multiple annotations. This makes it possible to present useful information for decision-making during annotation work, etc., by the presentation screen 27. Labels based on multiple annotations can be easily compared.
[0114] Here, for the sake of convenience, user U010 may also be referred to as the "first user," user U001 may be referred to as the "second user," and user U002 may be referred to as the "third user."
[0115] (5-2) In the embodiment, as an example, the multiple annotations may include first and second annotations. The first annotation may be made by a first user U010. The second annotation may be made by a second user U001 different from the first user U010, or may be made by the first user U010 earlier than the first annotation. The presentation process (S104) may cause the computer (system 1) to generate a presentation screen 27 to present to the first user U010 the classification content of at least one label associated with at least one region image in the medical image by the first and second annotations while the first user U010 is making the first annotation or after the first annotation is completed. The presentation screen 27 may disclose or not disclose information that can identify the second user.
[0116] (5-2-1) Specifically, as an example, when a first user U010 is making a first annotation (S201 to S202), a computer is caused to execute a process (S103, S104) of generating a presentation screen 27 to present to the first user U010 the classification content of a label associated with at least one region image in a medical image by the first annotation and the classification content of a label associated with at least one region image in a medical image by a second annotation (annotation by the second user U001), which is different from the first annotation. This allows the first user (user U010) and the second user (user U001 and / or U002) to compare the labels created by their respective annotations.
[0117] (5-2-2) As another example, when a first user U010 is making a "second or subsequent annotation (U010-2)" (S201 to S202), the computer (system 1) may execute a process (S103, S104) to generate a presentation screen 27 to present to the first user U010 the classification content of the label associated with at least one region image in the medical image by the first annotation (U010-2) and the classification content of at least one label associated with at least one region image in the medical image by a second annotation (U010-1) previously made by the user U010. This allows the first user U010 to compare one of his or her annotations (U010-1) with another one (U010-2).
[0118] The "second annotation" can be distinguished from any one annotation of the first user U010. Multiple annotations (e.g., U010-1, U010-2) saved by the same user (e.g., the first user U010) with different annotation IDs may be treated as separate annotations. As an example, consider a case where the user U010 makes a second annotation (U010-2) on the same medical image. In this case, the "second annotation" may include any one or more annotations of another user (the second user U001 in this embodiment) and / or may include the first user U010's past annotation (U010-1).
[0119] (5-3) In another example of the embodiment, the multiple annotations may include second and third annotations. When the first annotation is made by the first user U010, the second and third annotations may be as follows: The second annotation may be made by the second user U001, or may be made by the first user U010 earlier than the first annotation. The third annotation may be made by the third user U002, or may be made by the first user U010 earlier than the first annotation, or may be made by the second user U002 earlier than the second annotation. The second and third annotations may be two annotations made by the first user U010 earlier than the first annotation, but the two annotations are considered to have been made at different times. The presentation process (S104) may cause the computer (system 1) to generate a presentation screen 27 to present to the first user U010 the classification content of at least one label associated with at least one region image in the medical image by the second and third annotations while the first user U010 is making the first annotation or after the first annotation is completed. The presentation screen 27 may disclose or not disclose information that can identify at least one of the second user and the third user.
[0120] (5-3-1) Specifically, as an example, when a first user U010 is making a first annotation, the computer (system 1) may execute processes (S103, S104) to generate a presentation screen 27 so as to present to the first user U010 the classification content of a label associated with at least one region image in a medical image by "each of the annotations of multiple other users (the second user U001 and the third user U002)." The labels by the annotations of multiple other users (the second user U001 and the third user U002) can be compared.
[0121] (5-3-2) Specifically, as another example, when a first user U010 is making a first annotation, the computer (system 1) may execute processes (S103, S104) to generate a presentation screen 27 so as to present to the user U010 the classification content of the label associated with at least one region image in the medical image by "each of the first and second annotations (U001-1, U001-2) by the second user U001." The labels by the second and third annotations (U001-1, U001-2) made by the second user U001 can be compared.
[0122] The timing of presenting the presentation screen 27 is not limited to when the first user U010 is performing the first annotation work. The computer (system 1) may be caused to present the presentation screen 27 when the first user U010 is not performing the first annotation (for example, after finishing the work). For example, a "reference mode" may be set on the menu screen after logging in, and in this reference mode, the computer (system 1) may be caused to execute a process of generating the presentation screen 27 for the first user U010 to refer to his / her own and / or others' annotations without calling up the work screen 26.
[0123] <6. Other> (6-1. Criteria for Substantial Identity of Images, and DB Management) In the embodiment, the following criteria for "substantially identical" can be adopted for identity between multiple medical images. At least the following first or second situation can be cited as examples of when two medical images are substantially identical. In the first situation, the two medical images are completely identical and have the same image data. In the second situation, even if there are differences between the medical images (for example, differences in image data and / or image IDs), the differences can be considered identical for annotation purposes.
[0124] For example, suppose a medical image is acquired by photographing the same medical image object. Then, a new medical image is generated by duplicating the medical image data, changing the image format after duplication (e.g., saving in a compressed format), or applying image processing (resizing, changing brightness, contrast, saturation, color tone, etc.) after duplication. The original medical image and the new medical image are substantially identical, even if there are differences in the image ID, image data, etc. This is because the photographed object of the medical image is the same, and the differences can be ignored for annotation purposes. The same applies to region images; the substantially identical standard can be applied to region images generated from substantially identical medical images.
[0125] Alternatively, for example, a system known as a "virtual slide" may be used. In a virtual slide, the same imaging target (pathological specimen, i.e., tissue) is photographed at various magnifications, thereby obtaining, for example, several to several dozen pathological images at different magnifications, and these numerous pathological images are registered in an image database. The use of digitized pathological specimen data has the advantage of making it easier to improve the pathologist's observation accuracy. A group of pathological images in a virtual slide is essentially identical, since they merely represent the same pathological specimen at different magnifications. This concept is also applicable to any medical images in which the same imaging target is photographed at different magnifications. A standard of substantial identity can also be adopted for region images generated from these substantially identical medical images.
[0126] Furthermore, for example, when tissue is continuously cut to generate serial sections, a pathological section may have almost the same structure as a pathological section directly above or below it. In such cases, the pathological images of each of these pathological sections are substantially identical. Similar cases are also expected for all medical images other than pathological images. For example, when a three-dimensional structure is sliced, it is expected that the planar images of each adjacent layer are substantially identical. In these cases, multiple medical images are also substantially identical. A standard of substantially identical can also be adopted for regional images generated from these substantially identical medical images.
[0127] The embodiment may employ at least the "criteria for substantial identity of images" described in the above examples. For example, in the embodiment, the image IDs of medical images that are to be treated as substantially identical may be linked to each other by a manual user operation or automatic computer processing so that the images are treated as a group together in the image DB of the storage unit 20.
[0128] As an example, suppose a first user (e.g., U010) annotates a first medical image. A second user and / or a third user (e.g., U001 and / or U002) annotates a second medical image. The second medical image is a copy of the first medical image, or a copy of the first medical image that has undergone image processing. The first and second medical images are linked in the image database, and both images are treated as identical. Even when considering such a case and referring to Figures 7 to 18 and 20 to 24, it can be understood that multiple annotations associate multiple labels with each region image in the substantially identical first and second medical images.
[0129] In the embodiment, when multiple labels (e.g., first and second labels) are associated with each region image of a substantially identical medical image, several examples of ID management in each DB of the storage unit 20 are possible. In some examples, the substantially identical medical images may be annotated using a grid method, or a grid may be set using a designated frame method. In this case, the grids of each medical image may be set to the same size and with the same coordinates.
[0130] As a first example of ID management, the DB in the storage unit 20 may associate one region image ID with label IDs for each of multiple annotations. In this case, a group of region image IDs may be commonly used for multiple medical images. In other words, simply, a region image ID associated with a certain label (first label) may also be used as a region image ID associated with another label (second label).
[0131] However, this is not limiting, and a second example of ID management may be as follows. There may be cases where a first medical image (master medical image) is copied to generate one or more second medical images (copy medical images). Furthermore, image processing may be performed on the second medical image. In at least these cases, the first and second medical images are essentially the same image, but have different image data and may be assigned different image IDs. Therefore, as a second example of ID management, the image IDs of the first and second medical images may be linked within an image DB. In an image group consisting of a master medical image ID and one or more copy medical image IDs, if a label ID is associated with one region image ID, this association may be followed and the same label ID may be automatically associated with the medical images of the remaining image IDs.
[0132] At least, a medical image ID, a region image ID, and a label ID may be associated directly as in the first example, or indirectly as in the second example, thereby allowing multiple annotations to be performed on substantially identical medical images, and allowing multiple labels to be associated with one region image using multiple annotations.
[0133] (6-2. Computer-Based Annotation) The entity that performs the annotation (annotation entity) is not limited to a person and may also include a computer. When a computer classifies images using rule-based classification processing or machine inference using a learned discriminant model, the classification content (classification result) is associated with each image, and this may be considered an annotation. Multiple annotations that contrast or compare labels may be made by different or the same entity. A computer user ID may be set in the account DB, and an annotation entity ID unique to each computer may be assigned when registered in the label DB.
[0134] (6-3. No Restrictions on Hardware Configuration) In the example of FIG. 1, the system 1 is illustrated as a single information processing device, but this embodiment is not limited to this. For example, the system 1 may be composed of one or more physical servers, or may be composed using a virtual server running on a hypervisor, or may be composed using a cloud server. Furthermore, the program processing related to the generation and display of the presentation screen 27 can be shared arbitrarily between the system 1 (server 1) and the user terminal 3. Please note that the way in which processing is shared in the control flow of FIG. 4 is one example.
[0135] In the specification and claims of this application, the term "computer" should be understood to mean "at least one computer" unless expressly limited to a different meaning. A "computer" may include any one or more of a user terminal and / or a server. When a "computer" executes a program, the entire execution process may be completed on a single computer, or part of the execution process may be executed on a first computer (e.g., a server) and the rest of the process may be executed on one or more other computers (e.g., a user terminal or other servers). The one or more other computers may be any one or more of the second through nth computers (n is any integer equal to or greater than 3). The process of generating a presentation screen may be executed on a user terminal. Alternatively, the process of generating a presentation screen may be executed on a server, in which case the completed presentation screen data may be sent to the user terminal. Part of the process of generating a presentation screen may be executed on a user terminal, and the rest of the process may be executed on a server. However, if it is clearly stated that "a computer" should be interpreted differently from the above, this description is limited to that interpretation.
[0136] While various embodiments of the present invention have been described, they are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as set forth in the claims.
[0137] 1: System (Server), 2: Communication Network, 3: User Terminal, 4: Diagnostic Device, 11: Processor, 12: Storage Device, 13: Communication IF, 20: Storage Unit, 21: Display Control Unit, 22: Input Unit, 23: Generation Unit, 25: Presentation Unit, 25a: Data Acquisition Unit, 25b: Presentation Screen Generation Unit, 25c: Comparison Unit, 25d: Selection Unit, 25d: Presentation Screen Generation Unit, 26: Work Screen, 27: Presentation Screen, 31: Control Unit, 32: Storage Unit, 33: Communication Unit, 34: Operation Input Unit, 34: Operation Device, 34: Display Device , 35: Speaker, 36: Microphone, 37: Camera, 38: Operation device, M10: Selection menu (type of medical image), M11: Selection menu (specifying medical image), M12: Selection menu (annotation disclosure setting), N11: Display area, T10, V11: Display frame, T110: Mouse pointer, U010: User (first user), U001: User (second user), U002: User (third user), V12: Presentation target frame, W10: Display area, W11: Overview display area
Claims
1. A program for assisting annotation of medical images, the program causing a computer to execute a presentation process for generating a presentation screen, the presentation screen presenting to a user the classification content of at least one label associated with at least one region image in the medical image by at least one of a plurality of annotations.
2. A program as described in claim 1, wherein the multiple annotations include a first and a second annotation, the first annotation is made by a first user, the second annotation is made by a second user or was made by the first user prior to the first annotation, and the presentation process causes a computer to generate the presentation screen so as to present to the first user the classification content of the at least one label associated with the at least one area image by the first and second annotations while the first user is making the first annotation or after the first annotation is completed.
3. A program as described in claim 1, wherein the multiple annotations include second and third annotations; and when a first annotation is made by a first user, the second annotation is made by a second user or is made by the first user earlier than the first annotation; and the third annotation is made by a third user, is made by the first user earlier than the first annotation, or is made by the second user earlier than the second annotation; and the presentation process causes a computer to generate the presentation screen so as to present to the first user the classification content of the at least one label associated with the at least one area image by the second and third annotations while the first user is making the first annotation or after the first annotation is completed.
4. A program according to any one of claims 1 to 3, wherein the at least one region image is defined for the medical image by a grid or by a designated frame in each annotation.
5. A program according to any one of claims 1 to 3, wherein the at least one area image includes a label-comparable area image, the label-comparable area image being an area image to which a plurality of labels are associated by the plurality of annotations, and the presentation process causes a computer to generate the presentation screen so as to present the classification contents of the plurality of labels associated with the label-comparable area image by the plurality of annotations in a manner that allows a user to visually compare them.
6. A program according to any one of claims 1 to 3, wherein the at least one area image includes a label-comparable area image, the label-comparable area image being an area image to which a plurality of labels are associated by the plurality of annotations, the program causing a computer to execute a process of calculating a comparison result obtained by comparing the classification contents of the plurality of labels, and the presentation screen presents the comparison result to a user, thereby presenting the classification contents of the plurality of labels associated with the label-comparable area image.
7. A program as claimed in any one of claims 1 to 3, wherein the presentation screen presents to a user the classification content of at least one label associated with a plurality of area images in the medical image by at least one of a plurality of annotations.
8. A program as claimed in any one of claims 1 to 3, wherein the presentation screen has an area image and a label image, the area image being an image of the at least one area in the medical image, and the label image being an image corresponding to a label associated with the at least one area image, and the program causes a computer to execute a process of varying the display mode of the area image and / or the label image on the presentation screen according to classification content and / or according to a comparison result between a plurality of labels when the plurality of labels are associated with the area image.
9. A program according to any one of claims 1 to 3, wherein the classification content of the label includes an index value or is an index value, and the index value indicates how likely the classification content of the label is.
10. A method for supporting annotation of medical images, comprising: causing a computer to execute a presentation process for generating a presentation screen; and the presentation screen presents to a user the classification content of at least one label associated with at least one region image in the medical image by at least one of a plurality of annotations.
11. A system for supporting annotation of medical images, comprising: a presentation unit that generates a presentation screen, the presentation screen presenting to a user the classification content of at least one label that is associated with at least one region image in the medical image by at least one of a plurality of annotations.
Citation Information
Patent Citations
Data generation device, data generation method, and program
WO2021206053A1
System, method and program for supporting annotation
JP2022131937A
System and method for assisting verification of labeling and contouring of multiple regions of interest
US20220199231A1
Annotation assistance device, annotation assistance method, and annotation assistance program
WO2020153493A1
