Program, generation method, and system

The program and system streamline the generation of learning data for medical images by associating labels with regions using multiple information sources and index values, addressing the labor-intensive and expertise-dependent challenges of manual labeling, resulting in efficient and accurate data generation for machine learning algorithms.

WO2024247990A1PCT designated stage expired Publication Date: 2024-12-05DENKA CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/019506
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2024-05-28
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

The manual labeling of large numbers of medical images for machine learning algorithms is labor-intensive and requires high expertise, leading to a need for a more convenient learning data generation technology.

Method used

A program and system that generate learning data by associating labels with medical images through manual or automatic processes, using multiple pieces of information to determine definitive labels for regions within the images, and incorporating index values to represent the certainty of classification results.

Benefits of technology

This approach enables efficient and convenient generation of learning data, reducing the workload and improving accuracy by leveraging multiple annotations and index values to determine final labels, thus enhancing the efficiency of the learning data generation process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024019506_05122024_PF_FP_ABST
    Figure JP2024019506_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided are a program, a generation method, and a system for providing a novel learning data generation technique that is highly convenient. A program for generating learning data from a medical image is provided, the program causing a computer to execute generation processing wherein a plurality of pieces of information are associated with a region image that is an image of at least one region in the medical image. In the generation processing, the computer is caused, by means of a manual operation received via an operation input unit or automatic processing by the computer, to perform label association with the region image using the plurality of pieces of information, to generate the learning data from the medical image.
Need to check novelty before this filing date? Find Prior Art

Description

Program, generation method and system

[0001] The present invention relates to a program, a generation method, and a system for generating training data for machine learning algorithms.

[0002] The data generation device disclosed in Patent Document 1 generates training data for a machine learning algorithm. The training data is composed of a plurality of images and labels associated with each image.

[0003] International Publication No. 2021 / 206053

[0004] In order to generate training data for medical images, experts currently manually label a large number of images one by one. This situation arises from the high level of expertise and accuracy required to interpret medical images. However, the workload of manually labeling a large number of images is an issue. A more convenient technology for generating training data is desired.

[0005] The present invention provides a program, a generation method, and a system that provide a highly convenient new learning data generation technique.

[0006] According to the present disclosure, at least the following inventions are provided: [1] A program for generating learning data from medical images, the program causing a computer to execute a generation process in which a plurality of pieces of information are associated with a region image that is an image of at least one region in the medical image, the generation process causing the computer to associate labels with the region image using the plurality of pieces of information through manual operation received via an operation input unit or automatic processing by a computer, thereby generating the learning data from the medical image. [2] The program according to [1], the plurality of pieces of information include first and second pieces of information, the first information being a first label and the second information being a second label, and the program causing a computer to execute a process in which the first label and the second label are associated with the same region image through a first annotation and a second annotation. [3] The program according to [1] or [2], wherein each of the plurality of pieces of information is a label, and the generation process causes a computer to determine a definitive label for the region image from a plurality of labels associated with the same region image by a manual operation received via an operation input unit or an automatic process by a computer, and to generate the training data by associating the definitive label with a training data image generated from the medical image. [4] The program according to [1], wherein the plurality of pieces of information include first and second information, and the region images include a first region image and a second region image, and the generation process causes a computer to generate the training data by associating a label determined for the first region image using the first information with the second region image selected using the second information.[5] A program according to [4], which causes a computer to execute a process of acquiring input data including a plurality of the region images, wherein the plurality of first region images included in the plurality of the region images are generated from first and second medical images, respectively, the first and second medical images are associated with the first information, the first medical image is an image for which the first information satisfies a predetermined criterion, and the second medical image is an image for which the first information does not satisfy the criterion, and causes a learned discriminant model to perform machine inference to classify the plurality of second region images included in the plurality of the region images of the input data, The program generates the learning data by causing a computer to associate a label indicating that the first information satisfies the criteria with the second region image, which is included in the same region image as the first region image generated from the first medical image and classified into the first classification group as the classification result, and to associate a label indicating that the first information does not satisfy the criteria with the second region image, which is included in the same region image as the first region image generated from the second medical image and classified into the first classification group as the classification result. [6] The program described in [4] or [5], wherein the medical images include pathology images. [7] The program described in [6], wherein the first information includes at least one of a gene mutation type, a TMB-H (tumor mutation burden) type, a result of biochemical analysis, or a patient's age, sex, medical history, a result of immunohistochemical analysis, lifestyle, or prognosis. [8] A program according to any one of [1] to [7], wherein at least one of the plurality of pieces of information has or is an index value, and the index value represents the likelihood of the content of the information. [9] A program according to [8], citing [2] or [3], wherein a label associated with the region image has or is an index value, and the index value represents the likelihood of the classification result of the label.

[10] The program described in [8] citing [5], wherein the trained discriminant model outputs a probability value representing the likelihood of a classification result of the region image by the machine inference, and the probability value is used as the index value.

[11] A method for generating training data from medical images, comprising: a generating step in which a computer is caused to execute a generating process in which a region image that is an image of at least one region in the medical image is associated with a plurality of pieces of information, and the generating step causes the computer to associate labels with the region image using the plurality of pieces of information by manual operation received via an operation input unit or automatic processing by a computer, thereby generating the training data from the medical image.

[12] A system for generating training data from medical images, comprising: a generating unit in which a region image that is an image of at least one region in the medical image is associated with a plurality of pieces of information, and the generating unit generates the training data from the medical image by associating labels with the region image using the plurality of pieces of information by manual operation received via an operation input unit or automatic processing by a computer.

[0007] According to the present invention, a novel technique is provided for associating labels with region images using a plurality of pieces of information associated with the region images, thereby enabling highly convenient generation of learning data.

[0008] 1A and 1B are diagrams illustrating an example of a system 1 (server 1), each user terminal 3, and a diagnostic device 4 according to an embodiment. FIG. 2A is a block diagram illustrating an example of the hardware configuration of the server 1. FIG. 2B is a block diagram illustrating an example of the hardware configuration of the user terminal 3. FIG. 3A is a block diagram illustrating the functional configuration of the server 1 according to the first embodiment. FIG. 3B is a functional block diagram of a generation unit 23 according to the first embodiment. FIG. 3B is a functional block diagram of a generation unit 23 according to the first embodiment. FIG. 3B is a first example (grid method, manual confirmation processing) of a process flow illustrating a control operation according to the first embodiment. FIG. 3B is a second example (grid method, automatic confirmation processing) of a process flow illustrating a control operation according to the first embodiment. FIG. 7A illustrates an example of a state in which grid display is turned on in the selection screen 27 of FIG. 6 (designation frame method). FIG. 7B is an example of a result of performing area image conversion according to the grid of FIG. 7A. FIG. 9A is a fourth example (designation frame method, automatic confirmation processing) of a process flow illustrating a control operation according to the first embodiment. FIG. 9A illustrates an example in which there are three designation frames in FIG. 7A. Fig. 9B is an example of the result of performing region image conversion according to the grid of Fig. 9A. Fig. 10A is an example of an image in which labels have index values, and Fig. 10B is an example of an image in which labels are index values. Fig. 11A is a block diagram showing the functional configuration of the server 1 of the second embodiment. Fig. 11B is a functional block diagram of the generation unit 23 of the second embodiment. Fig. 11B is a first example of a processing flow representing a control operation of the second embodiment. Fig. 11B is a second example (processing using index values) of a processing flow representing a control operation of the second embodiment.

[0009] Hereinafter, several embodiments of the present invention will be described with reference to the accompanying drawings. The various features shown in the following embodiments can be combined with each other. Furthermore, each feature can be an invention independently.

[0010] <1. Terminology, etc.> (1-1. Annotation) In the field of machine learning, the term "annotation" can refer to the act of labeling images or to a group of assigned labels, depending on the context. Annotation as an annotation act means "the act of associating annotation information as correct answer data with target information (digital information) for machine learning." Various types of data, such as images, videos, and audio, can be annotation targets. For example, image annotation includes object detection, region image extraction, and image classification. Image annotation associates a label with the annotation target. The annotation target may be an individual image (the entire image) or a region image within the image (including an area, or the entire image). Labels are also called tags or metadata. Labels represent classification content (classification results) and serve as correct answer data during supervised learning. In the embodiments, "annotation" means "associating a label with an area image, which is an image of at least one area specified in an image," unless a different definition is specified or there is an obvious contradiction in the context.

[0011] (1-2. Images to be Annotated) In the embodiments, images to be annotated include at least "medical images." Medical images include "images that visualize the internal structure and function of the human body, primarily for the purpose of diagnosing and treating illness." Specifically, medical images may include plain X-rays, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), nuclear medicine tests, and angiography. In addition to the various images described above, medical images according to the present disclosure may also include "endoscopic images" and "pathological images." Endoscopic images may include, for example, images of the esophagus, stomach, intestines, and other digestive system images. Specifically, pathological images may be "microscopic images of tissue collected by endoscopy, surgery, etc., that have been (1) fixed to prevent deterioration, (2) stained for easy observation, and (3) sliced ​​into thin slices of approximately several microns."

[0012] The present invention is equally applicable to any medical image, and any label classification type and classification content can be set. The label classification type is, for example, information regarding the type of pathology. As an example, the pathology type may be either "tumor" or "hypermutation." For example, if each medical image is an image suspected of having a tumor, the pathology type may be set to "tumor," and in that case, the label classification content may be "tumor" or "non-tumor." If each of multiple medical images is an image suspected of having hypermutation, the pathology type may be set to "hypermutation," and in that case, the label classification content may be "hypermutation" or "non-hypermutation." In the following first and second embodiments, a pathology image is used as an example of a medical image, and the label classification type is, for example, a classification of whether or not the image is tumor, and the label classification content is "tumor" or "non-tumor."

[0013] In the embodiment, a pathological image is used as an example of a medical image, the classification type of the label is a classification of whether it is a tumor or not, and the classification content of the label is "tumor" or "non-tumor."

[0014] (1-3. Regions and Region Images) There are various methods for specifying regions (areas) in medical images. In the embodiments, a grid method is used in some examples, and a specification frame method is used in other examples.

[0015] The "grid method" uses a grid (i.e., a grid-like boundary line) to specify a desired position and range in an image. When a grid has multiple squares, each square can define a region. The grid may be uniformly spaced, and each region may be specified as a rectangle of uniform size. Such regions may be called "tiles." The grid spacing may be fixed or arbitrarily specified by the user. Note that the term "grid" here has a broad meaning and is not limited to a rectangular grid. It may also be a grid in which any polygonal shape that can be tiled on a plane is periodically arranged. The grid may be, for example, a honeycomb grid. The term "honeycomb grid" here has a broad meaning and may be, for example, a regular hexagonal honeycomb, but may also be, for example, a honeycomb structure of equilateral triangles or parallelograms, or a honeycomb structure of parallel hexagons (hexagons with three pairs of opposite sides parallel but with unequal side lengths). In this embodiment, grid-based annotation is used in the examples of Figures 4 and 5.

[0016] In the designation frame method, a region is designated at a desired position and range in an image using a designation frame arbitrarily specified by the user. The designation frame is sometimes called a bounding box. The designation frame can be set by the user to any size and any shape. The designation frame may be, for example, a rectangle, but is not limited to this, and may also be any polygon or closed curve. In the embodiment, the designation frame method annotation is adopted in the examples of FIGS. 6 and 8.

[0017] In the following description, for convenience, an area specified by a grid may be referred to as a "grid area" to facilitate understanding. Also, for convenience, an area specified by a designation frame may be referred to as a "designated frame area" to facilitate understanding. On the other hand, when simply referring to an "area," this may be intended to include both the grid area and the designated frame area, unless it is clear from the context that only one of the grid area and the designated frame area is being referred to.

[0018] When an area is specified in an image, the image corresponding to that area is also referred to as an "area image." When the area image is an image specified as a uniform rectangular size using a grid system, this area image may be referred to as a tile image. The size of the area and area image (number of pixels in the vertical and horizontal directions) may be the same as the size of the learning data that can be input to the learning model. When multiple designation frames are set in an image, if two designation frame areas match perfectly, the designation frame areas match each other, but if the designation frame areas differ, the designation frame areas also differ. When designation frame areas overlap, an overlapping area occurs where multiple designation frame areas overlap.

[0019] The program, method, and system of the embodiment can adopt either or both of the grid method and the designated frame method for annotation. If both are adopted, the method to be used for the current annotation may be set by a user setting operation and / or by automatic computer processing.

[0020] (1-4. Training Data) Annotation is typically performed to construct a training dataset. When annotating medical images, the training dataset consists of a "training data image" cut out from a medical image and a label associated with the training data image. The size (number of vertical and horizontal pixels) of the training data image is set to a size that can be input to a training model. The "training data image" may be a medical image cut out along a grid, or a portion of a medical image cut out using a specified frame (bounding frame). The training data image may be a medical image cut out according to a region image at the time of annotation; that is, the region image at the time of annotation and the training data image may match. However, this is not limited to this, and the training data image may differ from the region image at the time of annotation. As an example, if a medical image is annotated using a specified frame, the training data image may be generated by cutting it out using a grid (e.g., a uniformly spaced rectangular grid). As another example, if annotation is performed using a honeycomb grid or the like, training data images may be generated by cutting out this using another grid (for example, a uniformly spaced rectangular grid).

[0021] It should be noted that medical images do not need to be cropped during the annotation process or before the learning data generation process. For example, while images of each region of a medical image may be displayed on a work screen 26 and a selection screen 27 (described later), cropping of the medical image is not required at this display stage. After annotation is complete, the medical image may be cropped when an instruction to create a learning dataset is issued.

[0022] 2. First Embodiment (2-1. Overview of System 1 and Diagnostic Apparatus 4) As shown in FIG. 1 , a system 1, a user terminal 3, and a diagnostic apparatus 4 of the first embodiment can communicate with each other via a wireless or wired communication network 2. The system 1 in FIG. 1 can generate learning data from medical images. The system 1 is a computer, specifically a server. The server executes a program to provide the system 1. For convenience, the system 1 may be referred to as the server 1. Users U001 and U002 who use the system 1 are, for example, pathologists.

[0023] The system 1 includes a generation unit 23 (see FIG. 3A ) that has a function of generating training data. The generation unit 23 can generate training data using input images (specifically, medical images). The training data can be used as training data for training a training model included in the diagnostic device 4. The diagnostic device 4 uses a trained model trained using the generated training data to determine whether or not a pathological abnormality exists in the medical image.

[0024] The system 1 can display a medical image (in this embodiment, a pathological image) stored in the storage unit 20 and a grid (boundaries between multiple regions) on the screen, superimposed on the screen. An example of the screen is a screen on the display device 34 (see FIG. 2B ) of the user terminal 3. The system 1 accepts annotation operations on region images by users. That is, the system 1 accepts label input from users U001 and U002 for each region image displayed on the screen. As a GUI for label input operations, the system 1 generates a work screen 26 in FIG. 4 (described later) and displays it on the user terminal 3.

[0025] The system 1 may generate training data from a single pathology image, or may generate training data from multiple pathology images. When generating training data from multiple pathology images, the system 1 divides the pathology image into multiple region images and repeats the process of accepting label input for each pathology image. Once the user has completed assigning labels to all pathology images, the system 1 generates training data. For example, the system 1 generates training data images by cutting out medical images according to a grid, and generates training data by associating the image data of the training data images with the labels assigned to each region image. The generated training data is sent to the diagnostic device 4.

[0026] The diagnostic device 4 trains a learning model using the learning data sent from the system 1. There are no limitations on the specific configuration of the learning model, and it may be, for example, a neural network that can be given a predetermined capability through machine learning. The diagnostic device 4 inputs a pathological image of a diagnosis target into the trained model generated by machine learning, and determines whether or not a pathological abnormality is present in the pathological image based on the output result from the trained model.

[0027] The system 1 can provide a selection screen 27 as a support function for supporting annotation. This support function will be described in detail later.

[0028] (2-2. Hardware Configuration of System 1) Fig. 2A is a diagram showing an example of the hardware configuration of system 1. System 1 includes a processor 11 such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit), a storage device 12 such as a memory, an HDD (Hard Disk Drive) and / or an SSD (Solid State Drive), and a communication IF (Interface) 13 for wired or wireless communication. Via the communication IF 13, system 1 accepts user input operations from a user terminal 3 and displays various screens on a display device 34 (see Fig. 2B) of the user terminal 3.

[0029] (1-3. Hardware Configuration of User Terminal 3) As shown in FIG. 2B , the user terminal 3 includes, for example, a control unit 31 including a processor, a storage unit 32 including a non-volatile memory, a communication unit 33 for wireless communication and / or wired communication, a display device 34, a speaker 35, a microphone 36, a camera 37, and an operation device 38. The operation device 38 is an example of an operation input unit and is any input device such as a keyboard or a mouse. The specific configuration of the user terminal 3 is not limited, and may be various types of mobile terminals, smartphones, tablet terminals, or notebook PCs. If the user terminal 3 has a touch panel display, the display device 34 and the operation device 38 may be integrated into the touch panel display. The operation device 38 and the microphone 36 serve as an input interface (or input means) for the user to perform various input operations on the user terminal 3.

[0030] (2-4. Functional Block Configuration of System 1 and Generator 23) FIG. 3A is a diagram showing an example of the functional block configuration of the system 1. The system 1 includes a storage unit 20, a display controller 21, an input unit 22, and a generator 23. The storage unit 20 can be realized using the storage device 12 included in the system 1. The processor 11 of the system 1 executes a program stored in the storage device 12, thereby providing the display controller 21, the input unit 22, and the generator 23. The program may be provided over a network or may be stored in a storage medium. The storage medium storing the program may be a non-transitory computer-readable medium. The non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a USB memory or a CD-ROM.

[0031] (2-4-1. Configuration of the storage unit 20) The storage unit 20 includes various databases (DBs). For example, the storage unit 20 stores an account DB, an image DB, an area DB, an annotation DB, a label DB, and a learning data DB. Each DB assigns an identification ID to the stored data, and related data can be cross-referenced.

[0032] The account DB stores each user's user ID, user details, login history, etc. The user details may include an email address, a name associated with the account (nickname, avatar icon, etc.), and / or more personal and specific information such as name and affiliation.

[0033] The image DB stores one or more medical images (e.g., pathology images) used to generate learning data. The region DB stores information identifying a region image in each medical image. As an example, the region DB stores a region image ID, a medical image ID to which the region image is set, and image coordinate data. The image coordinate data specifies the range of the medical image that each region image occupies.

[0034] The annotation DB stores annotation information for distinguishing annotations made using the system 1 together with annotation IDs. The annotation information is preferably information that can identify which medical image was annotated, when, by whom, and what type of annotation was made. The annotation information can preferably store "annotation subject information." The annotation subject information is identification information for identifying "who made the annotation (i.e., the annotator)." The annotation subject information may be, for example, a user ID (e.g., a login account), or may be any one or more pieces of information in the user details stored in the account DB. The annotation information may include whether the annotation can be disclosed (i.e., whether it can be disclosed to others with different user IDs on their selection screen 27) and may also include disclosure conditions for the annotation (i.e., the conditions under which the annotation can be disclosed to others).

[0035] The label DB stores information about labels associated with region images, which are images of regions, by each annotation, in association with the annotation ID. The label DB stores each label ID in association with label detail information. The label detail information includes the classification type and classification content of the label (for example, the type is tumor classification, and the label classification content is tumor, etc.).

[0036] The training data DB stores the generated training data. A user can download any training data by referring to the training data DB.

[0037] It is preferable that at least the image DB, area image ID, annotation ID, and label ID are linked to each other. This allows the content of any annotation (i.e., medical image, area image, label, etc.) to be easily called up from the annotation DB by specifying the annotation ID. Note that the above data management and DB configuration are merely examples. Any data management and DB configuration can be adopted depending on the specifications of the actual hardware and / or software, etc. Note that if management is not performed on an annotation basis, the annotation DB can be omitted.

[0038] (2-4-2. Configuration of Other Functional Units) The display control unit 21 executes display control to display various screens on the display device 34 of the user terminal 3. For example, the display control unit 21 may be able to display a work screen 26 (see FIG. 4) in which a grid (boundary lines of regions) is superimposed on a medical image. A balloon extending from the work screen 26 in FIG. 4 illustrates a screen in which a medical image Q is divided into four by a grid. In reality, a larger medical image Q is divided by a grid, and a dozen by dozen region images are displayed on the work screen 26. A region image ID is associated with each region image. In the example of FIG. 4, the top left square of the medical image Q is 1 On the other hand, the display control unit 21 may be capable of displaying the medical image Q with a designation frame superimposed thereon.

[0039] The input unit 22 accepts various inputs from the operation device 38 of the user terminal 3. For example, the input unit 22 accepts a label input operation (i.e., an operation of the operation device 38 when the user performs annotation). The label input operation is an input operation for associating a label with each of a plurality of area images (area images). The input unit 22 stores the labels associated with each area image in the label DB. For example, the input unit 22 may associate an area image ID that uniquely identifies each area image with the label associated with each area image and store the associated label in the label DB. The label input may be performed, for example, by accepting a selection of an area image for which a label is to be input from among the area images displayed on the screen and accepting a designation of a label to be associated with the area image.

[0040] The generator 23 generates learning data for training the learning model. For example, the generator 23 acquires a region image ID and a label from the label DB, and generates a learning data image by cutting out a region image corresponding to the region image ID from a medical image stored in the image DB. The generator 23 generates the learning data by combining the learning data image data with the label stored in association with the region image ID.

[0041] 3B shows the functional block configuration of the generation unit 23. The generation unit 23 includes a data acquisition unit 23a, a label determination unit 23b, and a learning data configuration unit 23c. The operation of each functional block will be explained in the process flows of FIGS. 4 and 5.

[0042] (2-5. Processing Flow) Each of Figures 4 to 6 and 8 is a specific example of the processing flow of the first embodiment, and provides a program and a method for generating learning data. The processing flows of Figures 4 and 5 use grid-based annotation. The processing flows of Figures 6 and 8 use designated-frame-based annotation.

[0043] In the first embodiment, the generation process is executed by a computer (system 1) (see S103 to S106 in FIG. 4, or S103, S115, and S106 in FIG. 5). This generation process is performed by manually or automatically causing the computer (system 1) to associate area images with labels using a plurality of pieces of information associated with the area images, thereby generating learning data. As will be described later, in the first embodiment, first and second information are associated with each area image. Since label association of the area image can be performed using a plurality of pieces of information associated with the area image, learning data can be generated with high convenience.

[0044] The "plurality of pieces of information" used here differs between the first and second embodiments. In the first embodiment, the "plurality of pieces of information" includes first and second pieces of information, and first and second labels are used as the first and second pieces of information. Specifically, a computer (system 1) is caused to execute a process of acquiring a first label based on a first annotation and a second label based on a second annotation. This process is comprehensively described in step S100 in FIG. 4 and will be described in detail below.

[0045] In the first embodiment, multiple labels associated with region images in a medical image may have the same classification content. Such a case is simply referred to as "label match," and multiple labels associated with region images and with the same classification content are also referred to as "matching labels." On the other hand, there may be cases where labels with different classification content are associated with a region image, that is, where judgments based on multiple annotations differ. Such a case is simply referred to as "label mismatch," and multiple labels associated with region images and with different classification content are also referred to as "mismatching labels."

[0046] (2-5-1. Common Processing Step S100) In the processing flows of FIGS. 4 to 6 and 8, first, a user U001 who has logged in to the system 1 (S300a) receives a work screen 26 from the system 1 (display control unit 21) (S101a). The work screen 26 is a GUI for annotation work, that is, label input operation. When the user U001 performs a "first annotation" via this work screen 26, a label input operation is performed for each area image (S301a). In step S301a, for example, a first label La is input by the operation of the user U001. 1 and grid region image Q 1 When the grid region image Q of the medical image Q is associated with the grid region image Q of the medical image Q, the system 1 (the input unit 22) registers the association in the label DB (S102a). 2 , Q 3 Similar label correspondence is performed for ....

[0047] Similarly, the user U002 who has logged in to the system 1 (S300b) receives the work screen 26 from the system 1 (S101b). The user U002 performs a "second annotation" via the work screen 26, thereby inputting a label to each area image (S301b). In step S301b, for example, the second label Lb is input by the operation of the user U002. 1 and region image Q 1 The system 1 associates the grid region image Q with another grid region image Q in the medical image Q, and registers the association in the label DB (S102b). 2 , Q 3 Similar label correspondence is performed for ....

[0048] In the processing flows of Figures 4 and 5, a label is input to a grid area image in step S100. In the processing flows of Figures 6 and 8, a designated frame area image is set by a designation frame in step S100, and a label is input to this designated frame area image.

[0049] Here, for the area images for which label input has been completed in steps S301a and S301b, information indicating the classification content of the input label can be displayed superimposed on the area image. 1 or Lb1 As an example, the label classification content is displayed using the letters "T" and "N" (S301a, S30b). The letter "T" is displayed for an area that has been given a label indicating that the area image is an image suspected of containing tumor cells. The letter "N" is displayed for an area that has been given a label indicating that the area image is not an image suspected of containing tumor cells. 1 1 illustrates an example in which the label contents assigned in the first and second annotations are different, leading to different judgments by users U001 and U002.

[0050] For convenience of explanation, one grid region image Q 1 However, in reality, there are many grid region images Q k (i.e. Q 1 , Q 2 ...Q n ) is annotated. k is a region image identifier introduced for the sake of convenience, specifically k = 1, 2, ... n (n is an integer), and corresponds to the region image ID. By the first annotation, one or more region images Q 1 ~Q n A first label La 1 ~La n The second annotation allows one or more region images Q 1 ~Q n A second label Lb 1 ~Lb n Hereinafter, for the purpose of referring to any region image or label, we will refer to region image Q k , first label La k , or second label Lb k It may be written as follows.

[0051] (2-5-2. First Processing Flow Example) FIG. 4 shows a first example (manual confirmation processing) of a processing flow illustrating the control operation of the first embodiment. In the example of FIG. 4, label confirmation is performed by accepting a manual selection operation by the user. The first and second labels based on the first and second annotations have already been registered in the label DB in step S100 described above. At this stage, a user who desires learning data (user U002 in the example of FIG. 4) issues a learning data generation command to the system 1 via the operation device 38 (S302). This learning data generation command can specify input data required for generating the current learning data. As an example, a medical image ID, a range of area image IDs, or an annotation call mode can be specified as the target for adopting learning data.

[0052] In response to the learning data generation command (S302), the system 1 starts the learning data generation process (S103). In the example of FIG. 4, the generation process includes steps S103 to S106. First, the data acquisition unit 23a (see FIG. 3B) of the generation unit 23 references each DB of the storage unit 20. The data acquisition unit 23a acquires each region image Q to be included in the current learning data in accordance with the content of the learning data generation command. k On the other hand, by searching the label DB using the region image ID, each region image Q k The labels that are read here are the first and second labels La k , Lb k Includes:

[0053] The data acquisition unit 23a performs different data acquisition processes depending on the annotation call mode. The annotation call mode can be, for example, a designated call mode, a full call mode, or a layered call mode, and the user may be able to selectively use two or more of these modes.

[0054] In the specified call mode, a desired annotation is specified by an annotation ID, etc., and the area image ID, label ID, etc. of the specified annotation are referenced. In the all call mode, all annotations associated with the medical image at the time of call are referenced, and the area image ID, label ID, etc. of all the annotations are referenced.

[0055] In the stratified call mode, only annotations that meet specified conditions are extracted, and the region image ID, label ID, etc. of the extracted annotations are referenced. Various types of stratification can be set. For example, stratification may be based on whether or not the annotation subject information meets a specified condition. For example, only annotations whose performance values ​​are equal to or greater than a specified value may be extracted, thereby enabling the use of only highly proven and reliable annotations. For example, a period may be specified as one of the call conditions, and only annotations registered (specifically, newly registered or last edited) within the specified period may be extracted. As another example, index values ​​may be used. For example, only annotations whose summary statistics of all labels within the annotation (e.g., the average index value or the median index value of all labels) are equal to or greater than a specified threshold may be extracted, thereby enabling the use of only annotations created by the annotator with a certain degree of confidence.

[0056] Next, the label determination unit 23b executes label determination processing (S104, S105). The label determination unit 23b generates a selection screen 27 based on the data acquired in step S103 and transmits it to the user terminal 3 (S104). The selection screen 27 displays each area image Q k 4 is a GUI for allowing the user U002 to select which of the multiple labels associated with the region image Q should be set as the final label. 1 In the first and second labels La 1 , Lb 1 The system 1 accepts an operation to select either one of the above. For example, the user U002 can confirm the desired label by clicking with the mouse pointer P (S303). In practice, the selection screen 27 displays a plurality of area images Q k In the example of FIG. 4, the second label Lb1 is clicked. The label determination unit 23b determines the label selected by the user (in this example, the second label Lb 1 ) is determined as a definite label, and this definite label and the region image Q 1 These operations and processes are performed to obtain the region image Q required for learning data. k Repeat for each one.

[0057] Required area image Q k Once the labels and finalized labels are obtained, the process proceeds to step S106. In step S106, the training data construction unit 23c of the generation unit 23 cuts out the medical image Q according to the grid to generate training data images of the same size as the region image created by the grid. The training data construction unit 23c constructs training data in which the training data images and labels are associated with each other, and registers the data in the DB (S106). User U001 can then download the training data at will.

[0058] As described above, the process flow in FIG. 4 causes the computer (system 1) to execute the generation process. The generation process is performed by causing the computer (system 1) to generate a region image Q in accordance with the manual selection operation (S303) received via the operation device 38. k A plurality of labels La associated with k , Lb k The region image Q k The final label is determined for each region (S105), and the final label is associated with the region image, thereby generating learning data from the medical image (S106). 1 , Lb 1 ) based on each region image Q k It is possible to provide a mechanism for manually determining the definitive labels of a model in a highly convenient manner for users. It is possible to efficiently generate training data while taking multiple annotations into account, thereby improving convenience.

[0059] In step S104, each region image Q is processed without distinguishing between label matches and label mismatches. kAlternatively, a batch confirmation process may be provided as a variant. The batch confirmation process is a process in which, based on a user instruction (e.g., clicking a predetermined button), each area image with a matching label is collectively confirmed to the classification content of the matching label.

[0060] In FIG. 4, a second user U002 who has previously performed annotation work sends a learning data generation command (S302) to the system 1. However, this is just one example and is not limiting. As another example, a third user may perform only the confirmation operation (S303) without performing the label input operation (S301a, S301b) for the medical image Q. In this case, too, the label selection for each area image by the confirmation operation (S303) corresponds to the annotation work.

[0061] (2-5-3. Second Processing Flow Example) Fig. 5 shows a second example (automatic determination processing) of the processing flow showing the control operation of the first embodiment. In Fig. 5, the computer (system 1, specifically label determination unit 23b) automatically executes label determination processing (S115) in accordance with preset rules. This is the difference from the example in Fig. 4. The other steps S100 (S101a to S102b), S103, and S106 are the same as those in the flow in Fig. 4.

[0062] After step S103, the label determination unit 23b executes a label determination process (S115).

[0063] The label determination process when the labels match may be, for example, as follows: k , Lb k The region images Q show the same results as each other. k Let us consider the case where there are first and second labels La k , Lb k Specifically, if the region image Q 2 The first and second labels La 2 , Lb 2 (both not shown) are both "T" (i.e., tumors). In this case, the region image Q2 The definite label of is set to "T".

[0064] On the other hand, the label determination process when the labels do not match may be, for example, as follows: k , Lb k The region images Q show different results. k In this case, a preset process is executed as a label determination process. For example, the first and second labels La are determined according to a predetermined rule. k , Lb k As a specific example, as shown in the example of FIG. 5, when a region image Q 1 The first and second labels La 1 , Lb 1 In this case, the label determination unit 23b may determine the final label based on information of the annotation subject (users U001 and U002 in the first embodiment), for example. An example of the information of the annotation subject is "annotation performance value." For example, the label determination unit 23b may determine the final label based on information of the annotation subject (users U001 and U002 in the first embodiment), for example, based on information of the annotation subject (users U001 and U002 in the first embodiment). 1 The annotation performance value of user U001 who assigned the second label Lb 1 The annotation performance value of the user U001 who assigned the label to the region image Q1 may be read from the storage unit 20 and compared with the annotation performance value of the user U002 who assigned the label to the region image Q2. As a result of the comparison, the label assigned by the user with the relatively higher annotation performance value may be determined as the final label. k If there is a region image Q k As another example, the computer (system 1) may execute a process of excluding the region image Q from the learning data. k A manual operation selection process (for example, S104 and S105 in FIG. 4 can be used) may be adopted for this, and the computer (system 1) may be caused to execute the process of accepting label selection from the user.

[0065] As described above, the process flow in FIG. 5 causes a computer to execute the generation process. The generation process is performed by causing the computer (system 1) to generate a region image Q in accordance with the automatic process (label determination process in S115) of the computer (system 1). k A plurality of labels (first and second labels La k , Lb k ) to obtain the region image Q k Determine the final label for each region, and compare the final label with the region image Q k By associating the first and second annotations, training data is generated from the medical image. Based on the labels from the first and second annotations, a definitive label for each region image can be automatically determined. Training data that takes multiple annotations into consideration can be generated efficiently and with little workload, thereby improving convenience.

[0066] (2-5-4. Third Processing Flow Example) The processing flow in FIG. 6 is similar in many respects to the processing flow in FIG. 4. However, a major difference is that the example in FIG. 6 employs a designated frame annotation. In step S101a, the work screen 26 displays a medical image Q without a grid. In steps S301a and S301b, the users U001 and U002 use the work screen 26 to add a designated frame region image Q to the medical image Q using a designated frame. 1a , Q 1b Set the label La 1 , Lb 1 Enter.

[0067] In the selection screen presentation process (S104), the designated frame area image Q 1a , Q 1b and label La 1 , Lb 1 The selection screen 27 is generated so that the designated frame area image Q is displayed superimposed on the medical image Q. As an example of the confirmation operation (S303), the user U002 1a , Q 1b By this operation, the specified frame area image Q 1a and label La 1 and the designated frame area image Q 1b and label Lb 1The user U002 performs the selection operation in the same manner for all the designation frames associated with the medical image Q.

[0068] Thereafter, processing for constructing learning data is executed in step S106. The learning data constructing unit 23c can cut out the medical image Q according to the specified frame and generate a learning data image of the same size as the region image defined by the specified frame.

[0069] As a modified example, the computer (system 1) may execute the area image conversion process described below in step S104. Here, "area image conversion" refers to conversion between the image of the designated frame area and the image of the grid area. First, as illustrated in FIGS. 7A and 7B , a grid is superimposed on the designated frame. For area image conversion, for example, a label of the designated frame image is associated with each area image (width Wg × height Hg) defined by the grid in FIG. 7A within the range covered by the designated frame. For example, if the overlapping area between the designated frame area and the grid area is equal to or greater than a "predetermined overlap amount," the grid area image may be associated with the same label as the designated frame area image. The "predetermined overlap amount" may be, for example, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or 100% of the area of ​​the grid area, or may be within a range between any two of the values ​​exemplified here. When the predetermined overlap amount is 100%, a grid area and a designated frame area completely overlap. In FIG. 7B , which shows the area image conversion, a group of T-labeled areas with 3 rows and 2 columns and a group of N-labeled areas with 2 rows and 3 columns are generated. The two grid area images in FIG. 7B are logical product (AND) area images of the two designated frames, and both T and N labels are associated with each other. User U002 may perform a confirmation operation (S303) for each grid area image in FIG. 7B , similar to the flow in FIG. 4 . Then, in step S106, a training data image can be generated according to the grid in FIG. 7B . Please also refer to FIGS. 9A and 9B , which illustrate an example in which there are three designated frames.

[0070] (2-5-5. Fourth Processing Flow Example) The processing flow in FIG. 8 is similar in many respects to the processing flows in FIGS. 5 and 6. The difference from the processing flow in FIG. 5 is that the example in FIG. 8 uses a designated frame annotation method. The difference from the processing flow in FIG. 6 is that the example in FIG. 8 performs automatic label determination processing. Apart from these differences, the processing flow in FIG. 8 is similar to the processing flows in FIGS. 5 and 6.

[0071] In the flow of FIG. 8 , in a label determination process (S125), the label of the designated frame area image set in the medical image Q is determined. The process of step S125 can be the same as that of step S115 by compensating for the difference between the grid area and the designated frame area. In step S125, the designated frame may be automatically selected without performing area image conversion. In this case, as an example, the designated frame image and label with the higher annotation performance value may be automatically adopted. Thereafter, in step S106, a learning data image can be generated according to the designated frame image.

[0072] In the process of step S125, the generation unit 23 may perform region image conversion. After the region image conversion, a state in which labels are associated with the grid region image is created, as shown in FIG. 7B. After this, the process of step S115 can be similarly applied. Thereafter, in step S106, a training data image can be generated according to the grid of FIG. 7B.

[0073] When area image conversion is applied, for example, the computer (system 1) may execute the label determination process described below in step S125. In area image conversion, the overlap between the designated frame area image and the grid area image may differ for each designated frame. For example, in FIG. 7B , T and N labels are associated with two grid area images. However, referring to FIG. 7A , in the grid area image on the right side of the figure, the designated frame area image labeled N covers the entire grid area image, while the designated frame area image labeled T does not cover the entire grid area image. Such differences in overlap between the grid area image and the designated frame area image may be taken into account in the label determination process. For example, when there are differences in the degree of overlap between multiple designated frame area images and the grid area, the label of the designated frame area image with the largest overlap area may be used as the label of that grid area image. As another example, if the difference in overlap area between the designated frame area image with the largest overlap area and the other designated frame area images is equal to or greater than a threshold, the label of the designated frame area image with the largest area may be used as the label for that grid area image. If the difference in overlap area is less than the threshold, label determination based on the overlap area may not be performed, and determination processing may be performed based on other criteria. This allows the importance of designated frame area images with relatively small overlap areas to be lowered. The process described here can also be used when there are three designated frame area images, as in FIGS. 9A and 9B , or when there are four or more designated frame images.

[0074] (2-6. Modifications of the First Embodiment) The first embodiment can be modified in various ways, at least as described below.

[0075] (2-6-1. Index Value) As a modified example, an "index value" may be incorporated into the control of the first embodiment. An index value is a parameter that can be associated with each of a plurality of pieces of information. The index value represents the likelihood of the content of each of the plurality of pieces of information. This has the advantage that more accurate learning data can be generated based on the index value. The plurality of pieces of information (a plurality of labels in the first embodiment) may be index values ​​themselves, or may be provided with index values. This will be specifically explained below.

[0076] (2-6-1-1. Example of Adoption of Index Values) FIG. 10A is an example of a label image in which the label has an index value, and FIG. 10B is an example of a label image in which the label is an index value. As shown in FIGS. 10A and 10B, the first label La associated with the region image to be annotated is k and / or second label La k The index value in this example indicates how likely the label classification result is. The index value may be manually input by the user during annotation work on the work screen 26. The index value can be used in both grid-based and designated frame-based annotation. For example, the first label La in FIG. 10A k is written as "T90", which means that the label classification content is "T" (tumor) and its index value (likelihood) is 90(%). For example, the second label Lb k is written as "N70," which means that the label classification is "N" (non-tumor), and its index value (likelihood) is 70 (%). The index value may be manually input by users U001 and U002 in steps S301a and S302b of FIGS. 4 and 5. The index value may be arbitrarily set by the user in any increment (e.g., a fixed increment such as 1% or 5%), or the index value may be presented as, for example, a plurality of options (e.g., an arbitrary number of levels, such as three or four levels), from which the user can select a desired index value. For example, three levels of index values ​​may be "high" (e.g., 70% or more), "medium" (e.g., 31% to 69%), and "low" (e.g., 30% or less). This has the advantage of improving the convenience of the user's index value input.

[0077] (2-6-1-2. Example of application to the processing flow of FIG. 4 or FIG. 6) When index values ​​are used in the example of FIG. 4 or FIG. 6 (manual operation), labels with index values ​​may be associated with area images on the selection screen 27 as shown in FIG. 10A and FIG. 10B. The user can visually recognize and compare the index values ​​of each label. The user can select individual area images Q while taking into consideration the level of the index value. kIt is possible to determine a final label. The label with the highest index value among the multiple labels may be highlighted and displayed. For example, the label may be highlighted by attaching an arbitrary badge to the label, by highlighting (emphasizing), or by displaying the label itself relatively larger.

[0078] (1-6-1-3. Example of Application to the Processing Flow of FIG. 5 or FIG. 8) In the example of FIG. 5 or FIG. 8 (automatic processing), in step S115, a computer may execute a label determination process based on index values. As one example, the label with the largest index value among multiple labels may be adopted as the determined label. As another example, the difference in index values ​​between mismatched labels may be calculated, and if the difference is equal to or greater than a predetermined reference value, the label with the relatively larger reference value may be adopted as the determined label. On the other hand, if the difference in index values ​​between mismatched labels is less than the reference value, the area image associated with that label may not be included in the training data, or the user may decide which label to adopt for that area image through a manual selection process (e.g., S104 and S105 in FIG. 4 may be used). By integrating multiple annotation results while taking index values ​​into account, more accurate training data can be automatically generated.

[0079] (2-6-2. Number of "plural pieces of information") (2-6-2-1. Number of annotations and labels) In the first embodiment, as illustrated in Figs. 4 to 9, two pieces of information, namely, first and second information (first and second labels), are used as an example of "plural pieces of information". However, it should be noted that this only defines the minimum configuration, and any number of pieces of information (labels) greater than or equal to three can be added. For example, the first to third pieces of information may be used to associate the region image with the label. For example, if a region image Q is created using a third annotation by another third user, k A third label may be associated with each region image Q, and the third label may be acquired together with the first and second labels in step S103. kThe computer (system 1) may be caused to execute a process of determining a definitive label based on the first to third labels (steps S104 and S105 in FIG. 4 or S115 in FIG. 5), and a process of generating learning data in which the definitive labels are associated with region images (S106). In the first embodiment, the phrase "determining a definitive label based on the first to third labels" includes the meaning of "determining a definitive label from options ('T' or 'N') presented by the first to third labels."

[0080] In the above example, for convenience, first to third annotations are illustrated, and the first to third labels are assumed to be applicable to "multiple pieces of information." However, there is no upper limit to the number of "multiple pieces of information" that can be employed in the first embodiment. Any number of annotations by any number of users may be employed in the process flows of Figures 4 to 6 and 8. If the number of annotations that can be employed in the process flow of Figure 4 or Figure 5 in system 1 is "Z," the value of Z may be, for example, 2 to 100, for example, 3 to 20, or even 4 to 10. Specifically, the value of Z may be, for example, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, 75, or 100, and may be within the range of any two of the values ​​exemplified here.

[0081] The system 1 of the embodiment may be used by a limited number of members, for example, only members of a certain company, laboratory, or research group. However, the system 1 is not limited to this. The system 1 may accept user account registrations from members of many different research institutions, universities, or companies, or may be accessible to users from all over the world, and therefore may be made widely available to the public via the communication network 2. This has the advantage of making it possible to collect annotation data from many different users. However, in this case, not all users are necessarily willing to disclose their user information. To address this issue, the system 1 of the embodiment is equipped with a function for restricting annotation disclosure and / or a "function for displaying / hiding annotator information," which will be described later.

[0082] The system 1 generates a region image Qk The determination process (S104 and S105 in FIGS. 4 and 6 , S115 in FIG. 5 , and S125 in FIG. 8 ) may be performed by adopting all labels associated with the labels, but is not limited to this. For example, in the learning data generation command (S302), the user may be able to specify the label group to be adopted in the learning data generation process, that is, the user may be able to specify who made the annotations and when to adopt them. For example, in response to the learning data generation command, the system 1 may display on the user terminal 3 the user ID of each label associated with the area image ID for which learning data is to be generated. The user terminal 3 may accept a selection or specification of a user ID, and transmit the specified user ID to the system 1.

[0083] (2-6-2-2. Example of application to the process flow of FIG. 4 or FIG. 6) As an example, in the selection screen 27 of the process flow of FIG. 4, the first to third labels are assigned to one grid region image Q k The computer (the system 1, specifically the display control unit 21) may execute the process of displaying the information in the table. The user can select one of the first to third labels. The same applies when four or more pieces of information are used.

[0084] As another example, the selection screen 27 may be presented to the user in a form that allows the user to visually recognize the difference in the number of labels belonging to one of the label classification contents (e.g., "T") and the other (e.g., "N") among three or more labels. In this case, on the selection screen 27, each grid area image Q k One "T" label and one "N" label may be displayed superimposed on the grid area image, and the difference in the number of labels may be expressed in each label image. For example, on the selection screen 27, the number of "T" labels and the number of "N" labels may be displayed as numerical values ​​(the numerical values ​​may be displayed superimposed on the grid area image), or the label with the larger number may be displayed prominently. The method of highlighting may be the same as the example described above in "1-6-1-2. Example of application to the processing flow of Figure 4."

[0085] As another example, the three designated frame area images may have first to third labels, respectively, on the selection screen 27 of the processing flow of Fig. 6 (see Fig. 9A). As another example, when area image conversion is performed in the processing flow of Fig. 6, each of the application examples of Fig. 4 described above can be adopted for the grid area image after area image conversion (see Fig. 9B).

[0086] (2-6-2-3. Example of Application to the Processing Flow of FIG. 5 or FIG. 8) As an example, when the first to third information is adopted in the label determination process (S115) of FIG. 5, a majority vote process may be executed by a computer (the system 1, specifically, the label determination unit 23b). In the majority vote process, for example, the number of "T" labels and the number of "N" labels among the first to third labels may be counted in each area image, and the larger number of "T"s or "N"s may be adopted as the determined label in each area image. The majority vote process can also be applied in the case where area image conversion is performed in the processing flow of FIG. 8 (see FIG. 9B).

[0087] Majority voting can also be adopted when using four or more pieces of information (i.e., first to fourth labels, etc.). However, if the number of "plural pieces of information" is an even number, there may be a balance of half and half mismatched labels. When there is a balance, the final label may be determined using the annotation performance value, for example. Specifically, for example, the classification content of the label with the largest annotation performance value may be determined as the final label. As another example of a balance, the total value of the annotation performance values ​​of users who assigned the "T" label and the total value of the annotation performance values ​​of users who assigned the "N" label may be compared, and the label with the larger total value may be determined as the final label. As yet another example of a balance, the region image with mismatched labels may not be included in the learning data. As yet another example of a balance, the region image Q with mismatched labels may be determined as the final label. k The computer (system 1) may be caused to execute a process of accepting a label selection from the user in a manual operation selection process (for example, S104 and S105 in FIG. 4 can be used).

[0088] As an example, when three or more pieces of information (labels) are used in the label determination process (S115) of FIG. 5 , a computer (the system 1, specifically, the label determination unit 23b) may be caused to execute a calculation process based on index values. The calculation process based on index values ​​may be, for example, an arithmetic operation (e.g., addition, etc.) or a summary statistic calculation (e.g., calculating an average or maximum value). For example, among the first to third labels, the index values ​​of the “T” label may be added to obtain a first sum, and the index values ​​of the “N” label may be added to obtain a second sum, and the larger of the first and second sums may be adopted as the final label. Alternatively, instead of the first and second sums, first and second average values ​​may be calculated, and the larger of the first and second average values ​​may be adopted as the final label. Alternatively, for example, the label with the largest index value among the first to third labels may be adopted as the final label. The same applies to four or more pieces of information (labels). The calculation process based on index values ​​is also applicable when region image conversion is performed in the process flow of FIG. 8 (see FIG. 9B ).

[0089] (2-6-2-3. How to deal with missing labels) Multiple region images Q k The region image may include a region image with a missing label (i.e., an annotation omission). Specifically, for example, while many (e.g., the majority) of other region images have N labels associated with them, some region images may have fewer than N labels associated with them, or the number of labels may be zero. In this case, as an example, if at least one label is associated with an region image with a missing label, a process may be performed to determine a final label using that at least one label. As another example, for a region image with a missing label and / or with zero labels, that region image may be excluded from the target of learning data generation, or the user may be prompted to input additional labels for that region image through an annotation reception process (e.g., S101a, S301a, and S102a in FIG. 4 ).

[0090] (2-7. Disclosure / Non-Disclosure of Annotation Subject Information) The annotation subject information may be displayed on the selection screen 27. The annotation subject information is information that can identify the annotator who inputted each label. The annotation subject information may be a user ID or the like, or the user's name, affiliation, nickname, avatar icon, or the like. However, on the other hand, the annotation subject information may be hidden. As an example, the annotation subject information of the annotation called up on the selection screen 27 may not be displayed. The user may be able to set the annotation as "discloseable" or "non-discloseable." If the annotation is "discloseable," it may be displayed on the selection screen 27 under certain conditions (i.e., on the condition that the annotation subject information is not displayed). Hiding the annotation subject information allows the annotator to be kept confidential. Even if a user is not willing to disclose their user information, there is an advantage in that annotation data can be utilized while ensuring anonymity.

[0091] The entity that performs the first and second annotations (annotation entity) may be not only a person but also a computer. When a computer performs image detection processing using rule-based classification processing or machine inference classification using a trained discriminant model, the classification results are associated with each region image, which may be considered an annotation. Multiple annotations may be made by different or the same entity. A computer user ID may be set in the account DB, and an annotation entity ID unique to each computer may be assigned when registered in the label DB.

[0092] 3. Second Embodiment In the second embodiment, the description of the configuration that is the same as or similar to the first embodiment will be omitted or simplified, and the following description will focus on the differences.

[0093] (3-1. Functional Block Configuration of System 1 and Generator 23) The system 1 of the second embodiment shown in FIG. 11A includes a storage unit 20, a display controller 21, an input unit 22, a generator 23, and a discriminator 25. The system 1 differs from the system 1 of the first embodiment (see FIG. 3A) in that it includes the discriminator 25. The generator 23 of the second embodiment shown in FIG. 11B includes a data acquirer 23a, a label determination unit 223b, and a learning data constructor 23c. The second embodiment differs from the system 1 of the first embodiment (see FIG. 3A) in that it includes a label determination unit 223b that executes an automatic labeling process instead of the label determination unit 23b that determines a label selected by a user as a confirmed label.

[0094] (3-2. Description of the discrimination unit 25) The discrimination unit 25 is a trained discrimination model constructed using an arbitrary machine learning algorithm. There is no limitation on the machine learning algorithm that constructs the discrimination unit 25, and various known techniques can be adopted, such as a neural network, a decision tree, a support vector machine, or logistic regression analysis, or an arbitrary deep learning algorithm (DNN), or an arbitrary CNN model. The discrimination unit 25 has already completed learning using other learning data. The discrimination unit 25 is capable of performing image recognition processing on input image data, and is constructed so as to be able to perform machine inference to classify the input image data.

[0095] (3-3. Processing Flow) FIGS. 12 and 13 each show a specific example of a processing flow of the second embodiment, and provide a program and a method for generating training data. In the second embodiment, a generation process different from that of the first embodiment is executed by a computer (system 1) (see S152 to S157 of FIG. 12 or S152, S153, S164, S156, and S157 of FIG. 13). The generation process of the second embodiment is an automatic process in which the computer (system 1) associates a region image with a label using a plurality of pieces of information (first and second information) associated with the region image, thereby generating training data. As will be described later, in the second embodiment, the first and second information are also associated with each region image. Since label association with the region image can be performed automatically using a plurality of pieces of information associated with the region image, training data can be generated with high convenience.

[0096] In the second embodiment, the above-mentioned "plurality of information" includes first and second information, and "image-attached information G" is adopted as the first information (see FIG. 12), and "inference classification result by a trained discriminant model" is adopted as the second information (see FIG. 12). k , Ld k (Example shown in ). The image-attached information G is information that is attached to a medical image (the entire medical image). Typical examples include various types of medical information that explain the details of the medical image, or information about the patient from whom the medical image was obtained, and specific variations will be described later.

[0097] 12 and 13, a common processing step is that the computer (system 1) executes a process of receiving input data including a plurality of region images (S151). In the second embodiment, the storage unit 20 stores a first medical image Q and a second medical image R. The plurality of region images QM k , R.M. k can be generated from each of the first medical image Q and the second medical image R. As an example, a plurality of region images QM k , R.M. k Alternatively, the first and second medical images Q and R may be cut out into uniform sizes according to a grid.k , R.M. k The first and second medical images Q and R may be cut out from the first and second medical images Q and R according to one or more designated frames set for each of the first and second medical images Q and R. The first and second medical images Q and R are associated with first information (image-associated information G). Here, gene mutation types are used as an example of the image-associated information G. The gene mutation types of each medical image registered in the image DB are known in advance at the time of registration in the storage unit 20 (image DB) through various tests such as gene mutation testing and cancer panel testing. The storage unit 20 stores gene mutation types (type A applicable / not applicable, type B applicable / not applicable, etc.) in association with each medical image.

[0098] In the second embodiment, as an example, the first information (image-attached information G) is a "gene mutation type," and the predetermined criterion (labeling criterion) is "whether or not the image corresponds to gene mutation type A." The first medical image Q is an image for which the first information satisfies the predetermined criterion. The second medical image R is an image for which the first information does not satisfy the criterion. While FIG. 12 shows the image-attached information G in a schematic table, in reality the image-attached information G is stored in the image DB in association with the image ID.

[0099] Although the term "gene mutation type A" is used here for convenience, it is assumed that the specific type of gene mutation that constitutes type A has been set in advance. Depending on the location of the cancer, the types of gene mutations include, for example, EGFR gene mutation, HER2 gene amplification (ERBB2 copy number abnormality), RAS gene mutation (KRAS and NRAS), BRAF gene mutation (V600E or V600K), ALK fusion gene, ROS1 fusion gene, NTRK fusion gene, high-frequency microsatellite instability (MSI-High), BRCA gene mutation, or gene mutation burden (TMB). It is sufficient to predefine what specific genetic mutations, qualitatively and / or quantitatively, qualify as "gene mutation type A," thereby making it possible to distinguish between those that qualify as type A and those that do not. Each type (A, B, C, etc.) may be registered in the storage unit 20, and the presence or absence of each type may be selectively set when registering a medical image in the storage unit 20.

[0100] (3-3-2. First Processing Flow Example) In the first processing flow example shown in FIG. 12, when a learning data generation command is received from user U001 (S351), for example, system 1 starts the learning data generation process. The processing flow in FIG. 12 first causes the computer (system 1) to execute a process of acquiring input data including a plurality of region images (S151). Specifically, each DB in the storage unit 20 is referenced to acquire the input data necessary for generating the current learning data (S152). In the example in FIG. 12, user U001 specifies first and second medical images Q and R, specifies "corresponding / non-corresponding to gene mutation type A" as the first information, and specifies "tumor / non-tumor discrimination" as the second information, and a region image QM generated from these is k , R.M. k shall be processed.

[0101] In addition, any medical image or any region image may be specified, and / or the first and second information may be specified arbitrarily. This further improves convenience. For example, in step S151, the user U001 may be requested to enter several setting items. For example, the user U001 may be allowed to arbitrarily make at least one of a request to specify a medical image or region image, a request to specify the first information, and a request to specify the second information, and the content of the request may be transmitted from the user terminal 3 to the system 1.

[0102] In step S153, the region image QM k , R.M. k The computer (system 1) executes a process of making the trained discriminant model (discrimination unit 25) perform inference to classify the region images. The inference classification result for each region image is "T" or "N", and this inference classification result is the "second information" in the second embodiment. Here, in FIG. 12, for the purpose of making the explanation easier to understand, the label Lc of "T" (tumor) is used. k and "N" (non-tumor) labeled Ld k and the region image QM k , R.M. kHowever, in actual processing, such labeling is not essential (i.e., label DB registration is not required), and such illustration does not limit the scope of the present invention. In step S153, each region image may simply be classified as "T" or "N," and the classification results of each region image may be provided to subsequent processing (S154).

[0103] In step S154, the label determination unit 223b determines the region image QM k , R.M. k In the example flow of FIG. 12, only the region images labeled with "T" are extracted. Here, as an example, the region images QM labeled with "T" are extracted. k and region image RM k For convenience of explanation, in FIG. 12, the region image to which the "N" label is assigned is referred to as "QM j , R.M. m " and the labels associated with these are represented as Lc j , Ld m (j and m are integers). These region images to which the "N" label has been assigned are discarded in step S154. As a modified example, it is also possible to extract only region images to which the "N" label has been assigned.

[0104] In step S156, the label determination unit 223b executes an automatic labeling process. That is, in the process of step S156, one or more region images (region images QM after selection in S154) generated from the first medical image Q and classified into the first classification group (“T”) as a classification result are transmitted to the computer (system 1, specifically, the label determination unit 223b). k ) and label Le k and one or more region images (region images RM after selection in S154) generated from the second medical image R and classified into the first classification group ("T") as a classification result. k ) and label Lf k Each region image QM k , R.M. k Information such as whether the first or second medical image Q or R was generated from has already been stored in the image DB (see S151).

[0105] Label Le k is the region image QM k The first information satisfies the above-mentioned predetermined criteria (corresponding to gene mutation type A). k is the region image RM k In the generated learning data, the label Le k , Lf k is the correct answer data. In this way, the region images of the input data are classified using the trained discriminant model, and the region images classified into the first group as the classification result (second information) can be automatically associated with the labels determined using the first information (image-attached information G). Note that medical images or region images whose gene mutation type applicability is unknown may be excluded at the stage of step S152.

[0106] In the second embodiment, the first information (image-related information G) can be set in various ways. When the medical image used in the second embodiment is a pathology image, the first information may be a gene mutation type, a TMB-H (high tumor mutation burden) type, the results of biochemical analysis, or a patient prognosis prediction. The classification type and classification content of the label determined based on the first information in step S156 can be specified in various ways. The classification content of the label may be, for example, whether the gene mutation type corresponds to a predetermined type (e.g., Type A), whether it corresponds to a TMB-H (high tumor mutation burden) type, whether the results of biochemical analysis correspond to predetermined criteria, or whether the patient prognosis prediction is above a predetermined standard value. The patient prognosis prediction may be, for example, whether the life prognosis is above a predetermined period. The first information may be the patient's age, gender, medical history, results of immunohistochemical analysis, or lifestyle. These various classification types and classification content may be pre-registered in the storage unit 20 and selectively set when registering medical images in the storage unit 20.

[0107] In step S157, the computer (system 1) executes a process of generating training data using the region images and labels associated in step S156. The generated training data is registered in the training data DB, and the user U001 can download it at will.

[0108] As described above, the process flow in FIG. 12 causes a computer (system 1) to execute a generation process. In this generation process, the computer (system 1) determines a label (the label Le in S156 in FIG. 12) determined based on the first information (whether or not the gene mutation type A is present: S151). k , Lf k ) is sorted based on the second information (the inferred classification result of tumor / non-tumor: S153) into a region image (region image QM k , R.M. k The learning data can be automatically generated by performing label determination based on the first information and selection based on the second information on the region image.

[0109] (3-3-3. Second Processing Flow Example) Figure 13 is a second example of a processing flow (processing using index values) showing the control operation of the second embodiment. Specifically, the difference between the flows in Figures 12 and 13 is that step S154 and step S164 are interchanged. The other steps S151 to S153, S156, and S157 are the same as those in the flow in Figure 12.

[0110] In the second processing flow example shown in FIG. 13, it should be noted that the discrimination unit 25 outputs a probability value in step S153. Image recognition models using several known machine learning algorithms are capable of outputting the likelihood (i.e., probability or degree of coincidence) of classification results by image recognition. In the example of FIG. 13, the discrimination unit 25 is assumed to be a neural network model constructed using a deep learning algorithm. The output layer of the neural network outputs, as an inference result, a probability value indicating the probability with which the input image data belongs to each classification result. In the example of FIG. 13, this probability value is used as an index value. Through the inference in step S153, the trained discrimination model (discrimination unit 25) determines the likelihood of the region image QM k , R.M.k It outputs a probability value that indicates the likelihood of the classification result.

[0111] In step S164, the computer (the system 1, specifically the label determination unit 223b) determines the region image QM k , R.M. k Specifically, in the example flow of Fig. 13, the computer (system 1) extracts only area images for which the classification result of the inference classification process (S153) is "T" (tumor) and the index value (probability value) is inferred to be equal to or greater than a predetermined value. In the example of Fig. 13, the predetermined value is 0.80.

[0112] This predetermined value is, for example, 0.70 to 1.00, and may be, for example, 0.80 to 1.00. Specifically, this predetermined value may be, for example, 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, or 0.95, or may be within a range between any two of the values ​​exemplified here. The numerical values ​​exemplified here may be expressed as a percentage by converting 1.00 to 100%.

[0113] Thereafter, automatic labeling processing (S156) and learning data configuration processing (S157) are executed, similar to the processing flow of Fig. 12. By performing automatic selection using the probability values ​​obtained as the inference results, it is possible to automatically generate learning data with higher accuracy.

[0114] (3-4. Modifications of the Second Embodiment, etc.) In the second embodiment, the discrimination unit 25 is built into the system 1, but this is merely an example and the present invention is not limited to this configuration. The discrimination unit 25 does not have to be built into the system 1. The system 1 may cooperate with any external system having a trained discrimination model, and use this external system as the discrimination unit 25 to execute the processing of step S153.

[0115] <4. Other> (4-1. Criteria for Substantial Identity of Images, and DB Management) In the embodiment, the following criteria for "substantially identical" can be adopted for identity between multiple medical images. At least the following first or second situation can be cited as examples of when two medical images are substantially identical. In the first situation, the two medical images are completely identical and have the same image data. In the second situation, even if there are differences between the medical images (for example, differences in image data and / or image IDs), the differences can be considered identical for annotation purposes.

[0116] For example, suppose a medical image is acquired by photographing the same medical image object. Then, a new medical image is generated by duplicating the medical image data, changing the image format after duplication (e.g., saving in a compressed format), or applying image processing (resizing, changing brightness, contrast, saturation, color tone, etc.) after duplication. The original medical image and the new medical image are substantially identical, even if there are differences in the image ID, image data, etc. This is because the photographed object of the medical image is the same, and the differences can be ignored for annotation purposes. The same applies to region images; the substantially identical standard can be applied to region images generated from substantially identical medical images.

[0117] Alternatively, for example, a system known as a "virtual slide" may be used. In a virtual slide, the same imaging target (pathological specimen, i.e., tissue) is photographed at various magnifications, thereby obtaining, for example, several to several dozen pathological images at different magnifications, and these numerous pathological images are registered in an image database. The use of digitized pathological specimen data has the advantage of making it easier to improve the pathologist's observation accuracy. A group of pathological images in a virtual slide is essentially identical, since they merely represent the same pathological specimen at different magnifications. This concept is also applicable to any medical images in which the same imaging target is photographed at different magnifications. A standard of substantial identity can also be adopted for region images generated from these substantially identical medical images.

[0118] Furthermore, for example, when tissue is continuously cut to generate serial sections, a pathological section may have almost the same structure as a pathological section directly above or below it. In such cases, the pathological images of each of these pathological sections are substantially identical. Similar cases are also expected for all medical images other than pathological images. For example, when a three-dimensional structure is sliced, it is expected that the planar images of each adjacent layer are substantially identical. In these cases, multiple medical images are also substantially identical. A standard of substantially identical can also be adopted for regional images generated from these substantially identical medical images.

[0119] The embodiment may employ at least the "criteria for substantial identity of images" described in the above examples. For example, in the embodiment, the image IDs of medical images that are to be treated as substantially identical may be linked to each other by a manual user operation or automatic computer processing so that the images are treated as a group together in the image DB of the storage unit 20.

[0120] As an example, suppose a first user (e.g., U001) annotates a third medical image. A second user (e.g., U002) annotates a fourth medical image. The fourth medical image is a copy of the third medical image, or a copy of the third medical image that has been subjected to image processing. The third and fourth medical images are linked in the image DB, and the two images may be treated as the same.

[0121] In the embodiment, when multiple labels (e.g., first and second labels) are associated with each region image of a substantially identical medical image, several examples of ID management in each DB of the storage unit 20 are possible. In some examples, the substantially identical medical images may be annotated using a grid method, or a grid may be set using a designated frame method. In this case, the grids of each medical image may be set to the same size and with the same coordinates.

[0122] As a first example of ID management, the DB in the storage unit 20 may associate one region image ID with label IDs for each of multiple annotations. In this case, a group of region image IDs may be commonly used for multiple medical images. In other words, simply, a region image ID associated with a certain label (first label) may also be used as a region image ID associated with another label (second label).

[0123] However, this is not limiting, and a second example of ID management may be as follows. There may be cases where a third medical image (master medical image) is copied to generate one or more fourth medical images (copy medical images). Furthermore, image processing may be performed on the fourth medical image. In at least these cases, the third and fourth medical images are essentially the same image, but have different image data and may be assigned different image IDs. Therefore, as a second example of ID management, the image IDs of the third and fourth medical images may be linked within the image DB. In an image group consisting of a master medical image ID and one or more copy medical image IDs, if a label ID is associated with one region image ID, this association may be followed and the same label ID may be automatically associated with the medical images of the remaining image IDs.

[0124] At least, a medical image ID, a region image ID, and a label ID may be associated directly as in the first example, or indirectly as in the second example, thereby allowing multiple annotations to be performed on substantially identical medical images, and allowing multiple labels to be associated with one region image using multiple annotations.

[0125] It should be noted that, for example, the first and second medical images Q and R in the second embodiment are different images and are not substantially identical.

[0126] (4-2. Variations in Server Configuration, etc.) In the example of FIG. 1, the system 1 is illustrated as a single information processing device, but the embodiment is not limited to this. For example, the system 1 may be configured with one or more physical servers, or may be configured using a virtual server running on a hypervisor, or may be configured using a cloud server. Furthermore, the program processing related to the generation and display of the selection screen 27 can be shared arbitrarily between the system 1 (server 1) and the user terminal 3. It should be noted that the way in which processing is shared in each of the processing flows of FIGS. 4 to 6, 8, 12, and 13 is just an example.

[0127] In the specification and claims of this application, the term "computer" should be understood to mean "at least one computer" unless expressly limited to a different meaning. A "computer" may include any one or more of a user terminal and / or a server. When a "computer" executes a program, the entire execution process may be completed on a single computer, or part of the execution process may be executed on a first computer (e.g., a server) and the rest of the process may be executed on one or more other computers (e.g., a user terminal or other servers). The one or more other computers may be any one or more of the second through nth computers (n is any integer equal to or greater than 3). The selection screen generation process may be executed on a user terminal. Alternatively, the selection screen generation process may be executed on a server, in which case the completed selection screen data may be sent to the user terminal. Part of the selection screen generation process may be executed on a user terminal, and the remaining part may be executed on a server. However, if it is clearly stated that "a computer" should be interpreted differently from the above, this description is limited to that interpretation.

[0128] While various embodiments of the present invention have been described, they are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as set forth in the claims.

[0129] 1: System (Server), 2: Communication Network, 3: User Terminal, 4: Diagnostic Device, 11: Processor, 12: Storage Device, 13: Communication IF (Interface), 20: Storage Unit, 21: Display Control Unit, 22: Input Unit, 23: Generation Unit, 23a: Data Acquisition Unit, 23b, 223b: Label Determination Unit, 23c: Learning Data Composition Unit, 25, 225: Discrimination Unit, 26: Work Screen, 27: Selection Screen, 31: Control Unit, 32: Storage Unit, 33: Communication Unit, 34: Display Device, 35: Speaker, 36: Microphone, 37: Camera, 38: Operation Device, La 1 , La k : First label (first information in the first embodiment), Lb 1 , Lb k : second label (second information in the first embodiment), Q: first medical image, R: second medical image, G: image-attached information (first information in the second embodiment), Lc k , Ld k : Inference classification result (second information in the second embodiment), Le k , Lf k : Label, P: Mouse pointer, Q 1 , Q 2 , Q 3 , Q 4 , Q k , Q.M. k , R.M. k , Q.M. j , R.M. m : Area image, U001, U002: User

Claims

1. A program for generating learning data from medical images, the program causing a computer to execute a generation process in which a plurality of pieces of information are associated with a region image which is an image of at least one region in the medical image, and the generation process causes a computer to associate labels with the region image using the plurality of pieces of information through manual operations received via an operation input unit or automatic processing by a computer, thereby generating the learning data from the medical image.

2. A program as described in claim 1, wherein the plurality of pieces of information include first and second pieces of information, the first piece of information being a first label, and the second piece of information being a second label, and the program causes a computer to execute a process of obtaining the first label and the second label associated with one of the area images by the first annotation and the second annotation.

3. A program as described in claim 1 or claim 2, wherein each of the plurality of pieces of information is a label, and the generation process, by manual operation received via an operation input unit or automatic computer processing, causes a computer to determine a definitive label for one of the area images from a plurality of labels associated with the one of the area images, and generates the learning data by associating a learning data image generated from the medical image with the definitive label.

4. A program as described in claim 1, wherein the plurality of pieces of information include first and second information, the region images include a first region image and a second region image, and the generation process causes a computer to generate the learning data by associating a label determined for the first region image using the first information with the second region image selected using the second information.

5. A program as claimed in claim 4, comprising: causing a computer to execute a process of acquiring input data including a plurality of said region images; a plurality of said first region images included in said plurality of said region images being generated from first and second medical images, respectively; said first and second medical images being associated with said first information; said first medical image being an image where said first information satisfies a predetermined criterion, and said second medical image being an image where said first information does not satisfy said criterion; causing a computer to execute a discrimination process of causing a learned discrimination model to perform machine inference to classify a plurality of said second region images included in said plurality of said region images of said input data; and a classification result of said region images by said machine inference being said second information; The generation process causes a computer to associate a label indicating that the first information satisfies the criterion with the second region image included in the first region image generated from the first medical image and classified into the first classification group as the classification result, and to associate a label indicating that the first information does not satisfy the criterion with the second region image included in the first region image generated from the second medical image and classified into the first classification group as the classification result, thereby generating the learning data.

6. The program according to claim 4 or 5, wherein the medical image includes a pathological image.

7. The program according to claim 6, wherein the first information includes at least one of a gene mutation type, a tumor mutation-high (TMB-H) type, a result of a biochemical analysis, or a patient's age, sex, medical history, a result of an immunohistochemical analysis, a lifestyle, or a prognosis prediction.

8. A program according to any one of claims 1, 2, 4 and 5, wherein at least one of the plurality of pieces of information has an index value or is an index value, and the index value indicates the likelihood of the content of the information.

9. A program as claimed in claim 8, which cites claim 2, wherein the label associated with the region image has an index value or is an index value, and the index value indicates how likely the classification result of the label is.

10. A program as described in claim 8 which cites claim 5, wherein the trained discriminant model outputs a probability value representing the likelihood of the classification result of the region image by the machine inference, and the probability value is used as the index value.

11. A method for generating learning data from medical images, comprising the steps of: causing a computer to execute a generation step in which a plurality of pieces of information are associated with a region image, which is an image of at least one region in the medical image; and causing the computer to associate labels with the region image using the plurality of pieces of information through manual operation received via an operation input unit or automatic processing by a computer, thereby generating the learning data from the medical image.

12. A system for generating learning data from medical images, comprising a generation unit, wherein a plurality of pieces of information are associated with a region image, which is an image of at least one region in the medical image, and the generation unit generates the learning data from the medical image by associating labels with the region image using the plurality of pieces of information through manual operations received via an operation input unit or automatic computer processing.

Citation Information

Patent Citations

  • Generation device and generation method

    WO2021095698A1

  • Data generation device, data generation method, and program

    WO2021206053A1

  • Information processing device, method for operating information processing device, and program for operating information processing device

    WO2022209501A1