Annotation support device, annotation support method, and recording medium
Through hierarchical structure management of medical images and generating annotation information, the problem that the existing traditional Chinese medicine image diagnostic model is difficult to reflect the hierarchy and correlation of information is solved, and efficient annotation automation and quality improvement of diagnostic models is achieved.
Patent Information
- Application Number
- CN202080010815.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-01-24
- Filing Date
- 2020-01-24
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-01-24
AI Technical Summary
The prior art is difficult to generate high-quality medical image diagnostic models, and the manual annotation process consumes a lot of labor and cannot effectively reflect the hierarchy and correlation of information by doctors.
Manage medical images through hierarchical structures, acquire, display, input and store annotation information, generate training data that reflects the doctor's information reference process, and generate annotation guidance model using machine learning to automatically provide candidate annotation information.
It realizes automated processing of hierarchical and correlation annotation information of medical images, reduces manual annotation labor, and improves the quality and efficiency of the diagnostic model.
Smart Images

Figure CN113329684B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an annotation supporting device, an annotation supporting method and a recording medium. Background Art
[0002] In recent years, research has been conducted on automatic diagnosis support for medical images using machine learning. In order to achieve a high-performance model, a large amount of training data for supervised training is generally required, but for this purpose, annotations defined in a manner paired with medical images need to be prepared. Here, annotations refer to information on the location and area of lesions in medical images, and information on classification related to their characteristics and diagnosis. For example, the following patent document 1 discloses a technology for acquiring and storing annotations in accordance with an instruction operation from a user on a displayed medical image.
[0003] Previous technical literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Publication No. 2013-132514 Summary of the invention
[0006] Technical issues to be solved by the invention
[0007] When doctors interpret medical images, if a specific lesion exists in a medical image, in addition to the location and area of the lesion and the characteristics of the lesion, other examination results related to the lesion and information such as the patient's background are also referred to. In order to generate a high-quality model for the purpose of diagnosis support, it is desirable to provide training data that is appropriately constructed in a manner that reflects the hierarchy and correlation of annotation information as much as possible to reproduce the doctor's reference process to such information. In addition, in most cases, annotations are obtained through user instructions. However, since the manual process consumes a lot of labor, it is necessary to save labor by appropriately combining models.
[0008] Therefore, an object of the present invention is to provide an annotation supporting apparatus, an annotation supporting method, and an annotation supporting program that can provide hierarchical and related annotation information to a medical image.
[0009] Means for solving technical problems
[0010] An annotation support device according to one embodiment of the present invention supports the generation of annotation information related to a medical image. The annotation support device supports the generation of annotation information related to a medical image. The annotation support device manages the medical images according to the attributes of each medical image and by a hierarchical structure. The hierarchical structure takes one medical image as a node and includes at least one of a medical image hierarchy to which the node of the medical image belongs, a hierarchy above the medical image hierarchy, i.e., an upper hierarchy, and a hierarchy below the medical image hierarchy, i.e., a lower hierarchy. The annotation support device comprises: an image acquisition unit for acquiring a medical image; a display unit for displaying the medical image; an annotation input receiving unit for receiving a designation of any one of a node of the medical image and a node correspondingly associated with the node of the medical image in the hierarchical structure, and receiving an input of information related to the designated node, i.e., annotation information; and an annotation storage unit for storing the annotation information in a corresponding relationship with the designated node.
[0011] An annotation support method according to one embodiment of the present invention is an annotation support method in an annotation support device that supports the generation of annotation information related to medical images. The method manages medical images according to the attributes of each medical image and according to a hierarchical structure. The hierarchical structure takes a medical image as a node and includes at least one of a medical image hierarchy to which the node of the medical image belongs, an upper hierarchy of the medical image hierarchy, i.e., an upper hierarchy, and a lower hierarchy of the medical image hierarchy, i.e., a lower hierarchy. The annotation support method comprises: an image acquisition step for acquiring a medical image; a display step for displaying the medical image; an annotation input acceptance step for accepting the designation of a node of the medical image and any node that establishes a corresponding association with the node of the medical image in the hierarchical structure, and accepting the input of information related to the designated node, i.e., annotation information; and an annotation storage step for storing the annotation information in a corresponding association with the designated node.
[0012] An annotation support program involved in one embodiment of the present invention is used to enable a computer to function as an annotation support device that supports the generation of annotation information related to medical images. The annotation support program manages medical images according to the attributes of each medical image and by a hierarchical structure. The hierarchical structure takes a medical image as a node and includes at least one of the medical image hierarchy to which the node of the medical image belongs, the upper hierarchy of the medical image hierarchy, i.e., the upper hierarchy, and the lower hierarchy of the medical image hierarchy, i.e., the lower hierarchy. The annotation support program enables the computer to implement the following functions: an image acquisition function for acquiring medical images; a display function for displaying medical images; an annotation input acceptance function for accepting the designation of any node of the medical image node and the node that establishes a corresponding association with the node of the medical image in the hierarchical structure, and accepting the input of information related to the designated node, i.e., annotation information; and an annotation storage function for storing annotation information in a corresponding association with the designated node.
[0013] According to the above method, medical images are managed by a hierarchical structure, and the hierarchical structure includes at least one of the medical image hierarchy to which the nodes of the medical image belong, the upper hierarchy of the medical image hierarchy, and the lower hierarchy of the lower hierarchy. Then, the designation and annotation information of the node of the displayed medical image and any node correspondingly associated with the node are accepted, and the accepted annotation information is stored in a corresponding association with the designated node. Thus, a data group of a hierarchical structure in which the annotation information is appropriately associated with the nodes of the medical image and the nodes correspondingly associated with the node is obtained. By extracting training data from the data group, a model that can realize automatic diagnosis can be generated that reflects the doctor's reference process for various information.
[0014] In the annotation support device according to another aspect, the hierarchical structure may include a medical image hierarchy and at least one of an upper hierarchy and a lower hierarchy.
[0015] According to the above aspect, it is possible to appropriately associate upper-level information such as patient information and lower-level information such as observation information with the information on the medical image.
[0016] The annotation support device involved in another embodiment may also include: an area designation accepting unit, which accepts a part of the medical image displayed by the display unit and adds a designation of a seen area, i.e., a seen area, wherein the hierarchical structure includes a medical image hierarchy and the lower hierarchy, the annotation input accepting unit accepts input of annotation information on the seen area, the annotation storage unit stores the seen information representing the seen area as a node correspondingly associated with the medical image in the lower hierarchy, and stores the input annotation information in a corresponding association with the node.
[0017] According to the above method, input of annotation information for a sighted area designated as a part of a medical image is accepted, and the sighted information and annotation information are stored as nodes of a lower level of the medical image, thereby managing the medical image, sighted information, and annotation information for the sighted area in an appropriate hierarchical relationship.
[0018] In the annotation support device according to another aspect, the region specification accepting unit may accept a specification of a region of an arbitrary shape in the medical image as the specification of the observed region.
[0019] According to the above aspect, the specification of the observed area is accepted based on an arbitrary shape. The specification of the observed area is not limited to the specification based on a specific mark or the like, and therefore, the observed information that appropriately specifies the lesion area can be obtained.
[0020] In the annotation support device according to another aspect, the annotation input accepting unit may accept, in response to acceptance of a drag-and-drop operation of an object representing the annotation information with respect to the found region in the medical image displayed on the display unit, the input of the annotation information on the found region.
[0021] According to the above aspect, it is possible to easily associate annotation information with the viewed area.
[0022] In an annotation support device involved in another embodiment, the hierarchical structure may include patient information for identifying a patient as a node of an upper hierarchy, an annotation input receiving unit receives input of annotation information representing the patient information of a patient correspondingly associated with a medical image, and an annotation storage unit stores the input annotation information in a corresponding association with the node of the patient information.
[0023] According to the above method, the patient information for identifying the target patient of the medical image is managed as a node of the upper level of the medical image by the hierarchical structure, and the annotation information related to the patient is stored in a corresponding relationship with the patient information. Thus, the medical image, the patient information and the annotation information related to the patient are managed in an appropriate hierarchical relationship.
[0024] In the annotation support device according to another aspect, the annotation information may include relevance information indicating a relationship with other nodes other than the node associated with the annotation information.
[0025] According to the above method, each node can include, for example, the establishment of a correspondence between medical images and information found as annotation information. Thus, when the information established in correspondence with a certain node and the information established in correspondence with another node are related, the information group including the related information is provided to the training of the model for automatic diagnosis, thereby enabling automatic diagnosis reflecting the relatedness.
[0026] The annotation support device involved in another method further comprises: a training data acquisition unit, which acquires a data set including at least one medical image and annotation information correspondingly associated with a node of the medical image from information managed by a hierarchical structure as a training data set; a model generation unit, which uses the training data set acquired by the training data acquisition unit and generates an annotation guidance model through machine learning that takes the medical image as input and takes annotation information related to the medical image as output; and an annotation guidance unit, which displays the output obtained by inputting the medical image acquired by the image acquisition unit and the medical image to which the annotation is assigned into the annotation guidance model in a corresponding relationship with the medical image to which the annotation is assigned.
[0027] According to the above method, an annotation guidance model is generated by machine learning using a training data set extracted from an information group managed by a hierarchical structure, and a medical image is input into the generated annotation guidance model, thereby outputting candidate annotation information that should be associated with the medical image. Then, the candidate annotation information is displayed and presented to the user in association with the medical image, thereby facilitating the user's annotation work.
[0028] In an annotation support device involved in another embodiment, a training data acquisition unit acquires a data set as a training data set from information managed by a hierarchical structure, wherein the data set includes at least one medical image and annotation information, and the annotation information is correspondingly associated with a node of a medical image and at least one of other nodes correspondingly associated with a node of a medical image.
[0029] According to the above aspect, when the hierarchical structure for managing medical images is composed of a plurality of hierarchies, a training data set that appropriately includes necessary information can be acquired by setting a training task in model generation.
[0030] An annotation support device according to one embodiment of the present invention supports the generation of annotation information related to a medical image, and the annotation support device comprises: a training data acquisition unit that can pre-acquire a medical image to which annotation information is associated, and acquire a data set including the medical image and the annotation information associated with the medical image as a training data set; a model generation unit that uses the training data set acquired by the training data acquisition unit and generates an annotation guidance model that takes the medical image as input and takes the annotation information related to the medical image as output through machine learning; an image acquisition unit that acquires a medical image to which annotations are to be assigned; a display unit that displays the medical image acquired by the image acquisition unit; an annotation guidance unit that displays the output obtained by inputting the medical image acquired by the image acquisition unit into the annotation guidance model in association with the medical image to which annotations are to be assigned displayed by the display unit; an annotation input acceptance unit that accepts input of annotation information related to the medical image to which annotations are to be assigned; and an annotation storage unit that stores the annotation information accepted by the annotation input acceptance unit in association with the medical image to which annotations are to be assigned.
[0031] Effects of the Invention
[0032] According to an aspect of the present invention, hierarchical and related annotation information can be added to a medical image. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is a block diagram showing a functional structure of an annotation support device according to an embodiment.
[0034] Figure 2 This is a diagram showing the hardware configuration of the annotation support device.
[0035] Figure 3 The diagram shows a hierarchical structure for managing medical images, and shows a hierarchical structure in which the upper hierarchy of the medical images includes patient information and the lower hierarchy of the medical images includes observation information.
[0036] Figure 4 The diagram shows an example of a medical image displayed on a display device.
[0037] Figure 5 The diagram shows an example in which a plurality of medical images are displayed on a display device.
[0038] Figure 6 FIG. 1 is a diagram showing an example of inputting annotation information for patient information.
[0039] Figure 7 FIG. 1 is a diagram showing an example of inputting annotation information to a medical image.
[0040] Figure 8The diagram shows an example of designating a region related to a finding in a medical image and inputting annotation information for the finding information.
[0041] Fig. 9 This is a diagram showing an example of annotation information associated with each information of each hierarchy managed by a hierarchical structure.
[0042] Fig.10 This is a diagram showing a display example of candidate annotation information to be associated with a medical image.
[0043] Fig.11 This is a flowchart showing the processing contents of the annotation supporting method executed in the annotation supporting device.
[0044] Fig.12 This is a diagram showing the structure of the annotation support program. DETAILED DESCRIPTION
[0045] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and repeated descriptions are omitted.
[0046] Figure 1 1 is a diagram showing a functional configuration of an annotation support device according to the present embodiment. The annotation support device 1 is a device that supports generation of annotation information related to a medical image.
[0047] like Figure 1 As shown in the figure, the annotation support device 1 includes an image acquisition unit 11, a display unit 12, a region designation receiving unit 13, an annotation input receiving unit 14, an annotation storage unit 15, a training data acquisition unit 16, a model generation unit 17, and an annotation guidance unit 18. The functional units included in the annotation support device 1 of this embodiment can be dispersed in a plurality of devices. In addition, the functional units included in the annotation support device 1 are implemented by any one or any combination of hardware and software.
[0048] like Figure 1 As shown, the annotation support apparatus 1 can access storage means such as a medical image storage unit 20 and a model storage unit 30. These storage means 20 and 30 are configured to be accessible from the annotation support apparatus 1 via a network or a predetermined communication means. Furthermore, the storage means 20 and 30 may be provided in the annotation support apparatus 1.
[0049] Before describing each functional unit of the annotation support device 1, the storage units 20 and 30 will be described. The medical image storage unit 20 is a storage unit that stores medical images acquired in various examinations performed on a patient.
[0050] Medical images are, for example, DICOM images, which are one of the standard formats of medical images. Medical images may include images showing the results of various examinations such as X-ray examinations, CT examinations, MRI examinations, PET examinations, ultrasonic examinations, and endoscopy examinations. In addition to these images, medical images may also include information such as JPEG images, BMP images, PNG images, and MP4 animations as photographs and animations recorded during the diagnosis and treatment process. In addition, as described in detail later, the medical image storage unit 20 manages medical images in a hierarchical structure according to the attributes of each medical image.
[0051] The model storage unit 30 is a storage means for storing an annotation guidance model described in detail later. The annotation guidance model is a model generated by machine learning that takes a medical image as an input and outputs annotation information related to the medical image.
[0052] Figure 2 1 is a hardware structure diagram of the annotation support device 1. Figure 2 As shown, the annotation support device 1 is physically configured as a computer system, which includes a CPU 101, a main storage device 102 composed of memories such as RAM and ROM, an auxiliary storage device 103 composed of a hard disk and memory, a communication control device 104, etc. The annotation support device 1 may also include an input device 105 such as a keyboard, a touch panel, a mouse, etc. as an input device, and an output device 106 such as a display. In addition, the annotation support device 1 may be configured to include a GPU.
[0053] Figure 1 The functions shown in are implemented as follows, that is, in Figure 2 The CPU 101, main storage device 102 and other hardware shown read the prescribed computer software, thereby making the communication control device 104 and other devices operate under the control of the CPU 101, and read and write the data in the main storage device 102 and the auxiliary storage device 103. The data and database required for the processing are stored in the main storage device 102 and the auxiliary storage device 103.
[0054] Again, reference Figure 1 The functional units of the annotation support device 1 will be described. The image acquisition unit 11 acquires a medical image. Specifically, the image acquisition unit 11 acquires a medical image stored in the medical image storage unit 20 .
[0055] As described above, medical images are images obtained during various examinations performed on patients. Medical images can have patient information for identifying the patient, examination type, and other information as attribute information. For example, when multiple images are obtained during one examination, information for grouping these multiple images can be included as attribute information.
[0056] Medical images are managed by a hierarchical structure in the medical image storage unit 20 according to the attributes of each medical image. Specifically, the hierarchical structure related to the medical image takes the medical image as a node and includes at least one of the medical image hierarchy to which the node of the medical image belongs, the hierarchy above the medical image hierarchy, i.e., the upper hierarchy, and the hierarchy below the medical image hierarchy, i.e., the lower hierarchy. The upper hierarchy, for example, can include patient information as a node. The lower hierarchy, for example, can include observation information given to the medical image as a node.
[0057] Furthermore, the hierarchical structure for managing medical images may include a medical image hierarchy and at least one of an upper hierarchy and a lower hierarchy. By managing with such a hierarchical structure, upper hierarchy information such as patient information and lower hierarchy information such as observation information can be appropriately associated with medical image information.
[0058] Figure 3 Schematically shows an example of a hierarchical structure HS for managing medical images. Figure 3 In the example shown, the hierarchical structure HS includes a medical image hierarchy, a hierarchy above the medical image hierarchy, namely, an upper hierarchy, and a hierarchy below the medical image hierarchy, namely, a lower hierarchy. Figure 3 As shown, the hierarchical structure HS includes a root level hr, a first level h1, a second level h2, and a third level h3.
[0059] Figure 3 The hierarchical structure HS shown is rooted at the project nr marked as a project, and various information is managed in the lower hierarchical level. Figure 3 In the example, the case where the item nr is used as the root node is described, but the hierarchical structure may be configured without having an item as the root node.
[0060] The medical image n2 labeled as the image series (ImageSeries) constitutes a node belonging to the second level h2. The medical image has an aspect of examination information as an examination unit acquired for each of various examinations that can be performed on a certain patient.
[0061] The first level h1, which is the upper level of the node of the medical image n2, has patient information n1 labeled as patient information (Patient) as a node. The patient information n1 groups a plurality of medical images related to the same patient according to the patient information as attribute information of each medical image n2.
[0062] The lower layer of the node of the medical image n2, that is, the third level h3, has the observed information n3 marked as the disease category (Disease Class) as a node. The observed information n3 is information defined to maintain the regionality of a part of the medical image. Figure 3 As shown, multiple observation information can be defined for one medical image.
[0063] Reference again Figure 1 The display unit 12 displays the medical image. Specifically, the display unit 12 displays the medical image acquired by the image acquisition unit 11 on a monitor. Figure 4 1 is a diagram showing an example of a medical image displayed on the display by the display unit 12. Figure 4 As shown, a medical image mil, which is one of the examination results, is displayed on the display screen D1.
[0064] Furthermore, the display screen D1 includes a browse list IB and a reading report display section IR. The browse list IB is a section that displays a list of medical images for selecting medical images to be displayed on the display screen D1. When a user's selection input for a plurality of medical images displayed in the browse list IB is obtained, the display section 12 displays the selected medical image on the display screen D1. In addition, the list displayed in the browse list IB reflects the reference image. Figure 3 The structure of the hierarchical structure HS is described. The diagnostic reading report display unit IR is a part that presents the diagnostic reading report data stored in a corresponding relationship with the medical image displayed on the display screen D1 to the user as reference information when the user enters annotations. In addition, in the diagnostic reading report display unit IR, any clinical information stored in text form such as medical records can be displayed in a corresponding relationship with the displayed medical image.
[0065] Figure 5 FIG. 1 is a diagram showing an example in which a plurality of medical images are displayed on a display by a display unit. Figure 5 As shown, a plurality of medical images mi21, mi22, and mi23 are displayed on the display screen D2. Medical image mi21 is a medical image obtained by CT examination. Medical image mi22 is a medical image obtained by PET examination. Medical image mi23 is a medical image obtained by X-ray examination. Figure 5 As shown, in the annotation support device 1 of this embodiment, medical images of a plurality of related examination results can be displayed side by side.
[0066] Reference again Figure 1 The region designation accepting unit 13 accepts designation of a visible region of the medical image displayed by the display unit 12. The visible region is a region that is a part of the medical image and to which the visible region is attached. The designation of the visible region can be accepted in various ways.
[0067] The region designation accepting unit 13 can accept the designation of the observed region by, for example, inputting a designation operation of a portion of the displayed medical image by the user. Specifically, for example, by overlapping a pointer at an arbitrary position in the medical image and clicking, the region designation accepting unit 13 accepts the position as the observed region.
[0068] Furthermore, the region designation accepting unit 13 can accept the designation of any region in the medical image as the designation of the observed region. Specifically, for example, if a graphic of a shape surrounding any region in the medical image is input by operating various indicating devices, the region designation accepting unit 13 can accept the region of the arbitrary shape as the designation of the observed region. Thus, the designation of the observed region is not limited to the designation based on a specific mark, etc., and thus the observed information that appropriately designates the lesion region can be obtained.
[0069] The annotation input receiving unit 14 receives the designation of any node of the medical image and the nodes associated with the nodes of the medical image in the hierarchical structure, and receives the input of information related to the designated node, that is, annotation information. Figure 3 The specification of any node in the hierarchical structure HS of the description. Then, the comment input accepting unit 14 accepts input of information to be associated with the specified node as input of comment information.
[0070] The annotation storage unit 15 stores the annotation information received by the annotation input receiving unit 14 in association with the designated node. Figure 3 The input comment information is stored in association with the designated node among the nodes managed in the hierarchical structure HS of the description.
[0071] Below, reference Figure 6 to Figure 8 , a specific example of accepting annotation information is explained. Figure 6 FIG. 4 is a diagram showing an example of inputting annotation information of patient information. Figure 6 As shown, the display unit 12 displays a plurality of medical images mi21 , mi22 , and mi23 on the display screen D3 .
[0072] Medical images mi21, mi22, and mi23 are medical images showing the examination results of the patient identified by the label "patient information 1". Medical images mi21, mi22, and mi23 respectively constitute nodes belonging to the second level h2 of the hierarchical structure HS, and these nodes are associated with the node of the patient information "patient information 1" in the first level h1.
[0073] For example, when an input is made to designate the tag at1 in the display screen D3 , the comment input accepting unit 14 accepts this as a designation of a node of the patient information “patient information 1 ”.
[0074] Furthermore, if annotation information is input into the annotation information input field ail, the annotation input receiving unit 14 receives the input as annotation information to be associated with the node of the patient information "patient information 1". Examples of annotation information to be managed in patient units include the type of the patient's main disease and gender. Figure 6 In the example shown, information “Diseases: Lung cancer” is input as annotation information to be associated with the patient “patient information 1”.
[0075] The annotation storage unit 15 stores the input annotation information “Diseases: Lung cancer” in association with the node of the patient information “patient information 1”.
[0076] According to reference Figure 6 In the example described, the patient information for identifying the target patient of the medical image is managed by the hierarchical structure as a node of the upper level of the medical image, and the annotation information related to the patient is stored in a corresponding relationship with the patient information. Thus, the medical image, the patient information, and the annotation information related to the patient are managed in an appropriate hierarchical relationship.
[0077] Figure 7 FIG. 1 is a diagram showing an example of inputting annotation information for a medical image. Figure 7 As shown, the display unit 12 displays a plurality of medical images mi21 , mi22 , and mi23 on the display screen D4 .
[0078] Medical images mi21, mi22, and mi23 are medical images showing the examination results of the patient identified by the label "patient information 1". Medical images mi21, mi22, and mi23 respectively constitute nodes belonging to the second level h2 of the hierarchical structure HS, and these nodes are associated with the node of the patient information "patient information 1" in the first level h1.
[0079] For example, when an input for designating the label at2 is performed on the display screen D4 , the annotation input accepting unit 14 accepts the designation input as a designation of a node of the medical image “image series 1 - 2 ”.
[0080] Furthermore, when annotation information is input into the annotation information input field ai2 , the annotation input accepting unit 14 accepts the input as annotation information to be associated with the node of the medical image “image series 1 - 2 ”.
[0081] The annotation information that should be managed with a medical image is the information that should be managed with the inspection unit involved in the acquisition of the image, for example, clinical conditions such as "preoperative" and "postoperative", and the observed information that should be defined based on the information of the overall image (for example, the stage of the tumor, etc.). In addition, the correlation between medical images can be defined as the annotation information of the medical image. Regarding the correlation between a medical image and other medical images, as the annotation information of a medical image, other medical images can be explicitly specified to define the content of their correlation. In addition, the correlation can also be defined by default by establishing a corresponding association with the node of the same patient information.
[0082] The annotation storage unit 15 stores the annotation information input into the annotation information input field ai2 in association with the node of the medical image “image series 1 - 2 ”.
[0083] Figure 8 FIG. 1 is a diagram showing an example of specifying a region related to a finding in a medical image and inputting annotation information for the finding information. Figure 8 As shown, the display unit 12 displays the medical image mi21 on the display screen D5.
[0084] Medical image mi21 is a medical image showing the examination result of the patient identified by the label "patient information 1". Medical image mi21 constitutes a node belonging to the second level h2 of the hierarchical structure HS, and is associated with the node of the patient information "patient information 1" in the first level h1.
[0085] For example, by specifying a label in the display screen D5, when the medical image "image series 1-2" is selected, the region specification accepting unit 13 accepts the specification of the observed region or3 according to the user's input of the region specification. The observed region is a part of the medical image, for example, used to define a lesion or the like.
[0086] like Figure 8 As shown, the region specification accepting unit 13 can accept the specification of a region of any shape in the medical image as the specification of the observed region. In this way, the specification of the observed region is not limited to the specification based on a specific mark, etc., so that the observed information that appropriately specifies the lesion region can be obtained.
[0087] In addition, in a state where the label at3 corresponding to the observed area or3 is specified, if annotation information is input into the annotation information input field ai3, the annotation input receiving unit 14 receives the input as annotation information that should be associated with the observed area or3. The observed unit annotation information is information that can be defined to maintain regionality for a certain area in the image, for example, information that represents image features as expressed by a clinician through language. In this embodiment, the qualitative information shown in the observed area is labeled for the observed area, thereby realizing the input of annotation information.
[0088] The annotation storage unit 15 stores the observed information representing the observed area as a node correspondingly associated with the medical image in the lower level of the hierarchy of the medical image, and stores the inputted annotation information in a corresponding relationship with the node. Specifically, the annotation storage unit 15 stores the observed information representing the observed area or3 in the hierarchical structure HS as a node correspondingly associated with the medical image "image series 1-2" in the third level h3, which is the lower level of the second level h2. In addition, the annotation storage unit 15 stores the annotation information inputted to the annotation information input field ai3 in a corresponding relationship with the node representing the observed information of the observed area or3.
[0089] For example, in the case where there are multiple lesions in a medical image, the region designation accepting unit 13 can accept the designation of multiple visible regions in a medical image. For example, by accepting labels based on multiple different colors and line types on the display screen, the designation of multiple visible regions can be accepted. In this case, the annotation input accepting unit 14 accepts the input of annotation information for each visible region. Then, the annotation storage unit 15 generates a node corresponding to each visible region as a node of the lower level of a node of a medical image, and stores the input annotation information in a corresponding association with each node.
[0090] In addition, the input of annotation information can be realized in various ways. For example, by displaying an object representing the annotation information input to the annotation information input field ai3 in the annotation information input field ai3, for example, and accepting a drag-and-drop operation of the object relative to the visible area, the annotation storage unit 15 can establish a corresponding association between the input annotation information and the nodes of the visible area. In this way, the annotation information can be easily associated with the visible area.
[0091] Furthermore, the correlation between the observations in the image can be defined as annotation information of the observation information. The correlation between one observation information and other observation information can be explicitly designated as other observation information as annotation information of the observation information, thereby defining the content of its correlation. Furthermore, a corresponding association can be established with the subordinate nodes of the same medical image, thereby defining the correlation by default.
[0092] Fig. 9 1 is a diagram showing an example of annotation information associated with each information of each hierarchy managed by the hierarchical structure. As described above, the annotation storage unit 15 stores input annotation information in association with a node specified when the annotation information is input.
[0093] exist Fig. 9 In the figure, annotation information A1 is annotation information of the patient unit, which is associated with the node of the patient information belonging to the node of the first level h1 and marked as "patient information". Annotation information A1 can include information such as the presence or absence of disease, age, gender, initial diagnosis date, initial diagnosis time stage, content of the first treatment, recurrence date, and final metastasis.
[0094] Annotation information A2 is annotation information of a medical image unit (examination unit), which establishes a corresponding association with a node of a medical image that belongs to the node of the second level h2 and is marked as "image series". Annotation information A2 can include the clinical condition of the examination (preoperative, postoperative, recurrence time, etc.) and information obtained from the overall image (tumor staging, etc.). In addition, the annotation information correspondingly associated with the medical image can include setting information (Add to Dataset = True / False) indicating whether the medical image (examination) is included in the training data set for machine learning described later. The setting information can be arbitrarily set according to the purpose of machine learning and the generated model and according to the user's input. When the input is "True", the medical image is included in the training data set. In addition, it is not limited to such explicitly determining the specified setting information, and whether the medical image is included in the training data set can be determined based on whether there is specific annotation information.
[0095] Furthermore, the node of the medical image may have information that should be associated with the medical image unit as the annotation information, but in addition, the found information associated with the node in the lower level of the node of the medical image may be regarded as the annotation information of the medical image. That is, the found information may have both aspects of being a node of the lower level of the medical image and aspects of being the annotation information of the medical image.
[0096] The annotation information A3 is annotation information of the observed unit, and is associated with the node of the observed information belonging to the node of the third level h3, namely, the "disease category". The annotation information A3 can include information indicating the observed area and qualitative information of the observed area. The information indicating the observed area can define the lesion area by including coordinate information corresponding to the format of the medical image (for example, DICOM format). The qualitative information of the observed area can include qualitative label information (image characteristics, location, classification, etc.) for each observed area. In addition, the node of the observed information corresponds to Figure 8 And the following references Fig.10 Tags such as "structure" in .
[0097] Reference again Figure 1 The training data acquisition unit 16 acquires a data set including at least one medical image and annotation information correspondingly associated with a node of the medical image from the information managed by the hierarchical structure as a training data set. Furthermore, in the case where the hierarchical structure for managing medical images is composed of multiple hierarchies, the training data acquisition unit 16 can acquire a data set as a training data set, wherein the data set includes at least one medical image and annotation information in the information managed by the hierarchical structure, and the annotation information is correspondingly associated with the node of the medical image and at least one of the other nodes correspondingly associated with the node of the medical image. The content of the data set is determined according to the training topic in the model generation as described later.
[0098] The model generator 17 generates an annotation guidance model that takes a medical image as input and outputs annotation information related to the medical image through machine learning using the training data set acquired by the training data acquirer 16. The model generator 17 stores the generated annotation guidance model in the model storage 30.
[0099] In this embodiment, an annotation guidance model is generated by implementing so-called supervised training, wherein the supervised training uses a training data set including medical images as training objects and annotation information as correct answer data corresponding to the training results. The medical images included in the training data set can be specified in the annotation information of the medical images, for example, as described above. In addition, as the content of the training topic, classification of medical images, detection of specific findings such as lesions included in the medical images, and segmentation of specific findings (extraction of regions) can be performed, but other training topics can also be set according to the design.
[0100] The data included in the training data set is not limited to the annotation information of a medical image. Depending on the purpose and subject setting of the model, it can include information about other nodes associated with a medical image, annotation information that establishes a corresponding association with the other nodes, information indicating the association between nodes at the same level or different levels (association information), etc.
[0101] The annotation guidance model is configured to include a neural network. The neural network can be composed of, for example, a CNN (Convolutional Neural Network). The annotation guidance model, which is a model including a trained neural network, is read or referenced by a computer and can be regarded as a program that causes the computer to perform a predetermined process and realize a predetermined function.
[0102] That is, the annotation guidance model of this embodiment is used in a computer having a CPU (and / or GPU) and a memory. Specifically, the CPU (and / or GPU) of the computer operates in a manner that outputs the result from the output layer, based on the trained weighting coefficients and response functions corresponding to each layer, on the input data input into the input layer of the neural network according to the instructions from the annotation guidance model stored in the memory.
[0103] The annotation guidance unit 18 displays an output obtained by inputting the medical image acquired by the image acquisition unit 11 and the medical image to be annotated into the annotation guidance model in association with the medical image to be annotated.
[0104] In this embodiment, for example, the model generation unit 17 performs machine learning using a medical image and the observation information input as annotation information on the lesion part displayed in the medical image as a training data set, thereby generating an annotation guidance model that uses the medical image as an input and outputs the observation information of the lesion part that should be associated with the medical image. Here, the observation information provided for training and the observation information output from the model include information indicating the observation area, features related to the observation area, and information such as the type of lesion.
[0105] Fig.10 FIG. 4 is a diagram showing an example of displaying candidate annotation information that should be associated with a medical image. Fig.10 As shown, the display unit 12 displays the medical image mi3 to which the annotation is to be assigned on the display screen D6 .
[0106] The annotation guidance unit 18 inputs the medical image mi3 into the annotation guidance model, thereby obtaining candidate observation information output from the model as annotation information that should be associated with the medical image mi3. Fig.10As shown, the annotation guide 18 obtains the candidate found region ar1 and the candidate found region ar2 as annotation information to be associated with the medical image mi3, that is, candidate found information, and displays them superimposed on the medical image mi3. In addition, the annotation guide 18 obtains the candidate found information ac1 to be associated with the candidate found region ar1 and the candidate found information ac2 to be associated with the candidate found region ar2, and displays them on the annotation candidate display part gw.
[0107] If the annotation information corresponding to the medical image is accumulated to a quantity greater than that which can be used as a training data set for machine learning, the model generation unit 17 can generate an annotation guidance model based on the accumulated medical images and annotation information. Then, the annotation guidance unit 18 can obtain and output candidate annotation information that should be correspondingly associated with the medical image to which the annotation is assigned, based on the generated annotation guidance model. By displaying and presenting the candidate annotation information output in this way to the user in a corresponding manner with the medical image, the user's annotation work can be assisted and the annotation work can be promoted. In addition, based on the accumulation of medical images and annotation information, the generation of annotation guidance models can be implemented sequentially, thereby improving the effectiveness of the candidate annotation information presented to the user.
[0108] Next, refer to Fig.11 , the operation of the annotation supporting device 1 according to the present embodiment will be described. Fig.11 It is a flowchart showing the processing contents of the annotation supporting method executed in the annotation supporting device 1 .
[0109] In step S1, the image acquisition unit 11 acquires a medical image stored in the medical image storage unit 20. In step S2, the display unit 12 displays the medical image acquired by the image acquisition unit 11 on a display device. In addition, when a medical image is displayed in step S2, the annotation guide 18 can display candidate annotation information that should be associated with the displayed medical image in association with the displayed medical image.
[0110] In step S3, the annotation input accepting unit 14 accepts designation of a node of the medical image and a node associated with the node of the medical image in the hierarchical structure. In step S4, the annotation input accepting unit 14 accepts input of annotation information to be associated with the designated node.
[0111] In step S5, the comment storage unit 15 stores the comment information received by the comment input receiving unit 14 in association with the designated node. Specifically, the comment storage unit 15 stores the input comment information in association with the designated node among the nodes managed in the hierarchical structure HS.
[0112] Next, refer to Fig.12 , an annotation support program for causing a computer to function as an annotation support device is described. Fig.12 The annotation support program P1 shown includes a main module mm10, an image acquisition module m11, a display module m12, an area designation acceptance module m13, an annotation input acceptance module m14, an annotation storage module m15, a training data extraction module m16, a model generation module m17 and an annotation guidance module m18.
[0113] The main module mm10 is the part that centrally controls the annotation support processing. It implements the image acquisition module m11, the display module m12, the area designation acceptance module m13, the annotation input acceptance module m14, the annotation storage module m15, the training data extraction module m16, the model generation module m17 and the annotation guidance module m18. Figure 1 The illustrated annotation support device 1 includes functions of an image acquisition unit 11 , a display unit 12 , a region specification accepting unit 13 , an annotation input accepting unit 14 , an annotation storage unit 15 , a training data acquisition unit 16 , a model generation unit 17 , and an annotation guidance unit 18 .
[0114] The annotation support program P1 can be provided via a storage medium M1 such as a magnetic disk, an optical disk, or a semiconductor memory, for example. Alternatively, the annotation support program P1 can be provided as a computer data signal superimposed on a transmission wave via a communication network.
[0115] In the annotation support device 1, the annotation support method and the annotation support program P1 of the present embodiment described above, the medical image is managed by a hierarchical structure, and the hierarchical structure includes at least one of the medical image hierarchy to which the nodes of the medical image belong, the upper hierarchy of the upper hierarchy of the medical image hierarchy and the lower hierarchy of the lower hierarchy. Then, the designation and annotation information of the node of the displayed medical image and any node correspondingly associated with the node are accepted, and the accepted annotation information is stored in a corresponding association with the designated node. Thus, a data group of a hierarchical structure in which the annotation information is appropriately associated with the nodes of the medical image and the nodes correspondingly associated with the nodes is obtained. By extracting training data from the data group, a model that can realize automatic diagnosis can be generated that reflects the doctor's reference process for various information.
[0116] The present invention has been described in detail above based on the embodiments. However, the present invention is not limited to the above embodiments. The present invention can be modified in various ways without departing from the gist of the present invention.
[0117] Explanation of symbols
[0118] 1-annotation support device, 11-image acquisition unit, 12-display unit, 13-region designation acceptance unit, 14-annotation input acceptance unit, 15-annotation storage unit, 16-training data acquisition unit, 17-model generation unit, 18-annotation guidance unit, 20-medical image storage unit, 30-model storage unit, M1-storage medium, m10-main module, m11-image acquisition module, m12-display module, m13-region designation acceptance module, m14-annotation input acceptance module, m15-annotation storage module, m16-training data extraction module, m17-model generation module, m18-annotation guidance module, P1-annotation support program.
Claims
1. An annotation support device that supports generating annotation information related to a medical image, The medical images are managed according to the attributes of each medical image and in a hierarchical structure, wherein the hierarchical structure takes a medical image as a node and includes at least one of a medical image hierarchy to which the node of the medical image belongs, a hierarchy above the hierarchy of the medical image hierarchy, i.e., an upper hierarchy, and a hierarchy below the hierarchy of the medical image hierarchy, i.e., a lower hierarchy, The annotation support device comprises: An image acquisition unit, which acquires medical images; A display unit, which displays the medical image; an area designation accepting unit that accepts designation of a visible area, the visible area being a part of the medical image displayed by the display unit and to which the visible area is attached; an annotation input receiving unit that receives a designation of any one of the nodes of the medical image and the nodes associated with the nodes of the medical image in the hierarchical structure, and receives input of annotation information that is information related to the designated node; and The annotation storage unit stores the annotation information in a corresponding relationship with the designated node. The hierarchical structure includes the medical image hierarchy, the upper hierarchy, and the lower hierarchy, The annotation input receiving unit receives input of annotation information on the observed area, the annotation storage unit stores the observed area information as a node associated with the medical image in the lower layer, and stores the input annotation information in association with the node. The hierarchical structure includes patient information identifying a patient as a node of the upper hierarchy, The annotation input accepting unit accepts input of annotation information indicating patient information of a patient associated with the medical image, and the annotation storage unit stores the input annotation information in association with a node of the patient information.
2. The annotation support device according to claim 1, wherein: The region designation accepting unit accepts designation of a region of an arbitrary shape in the medical image as designation of the observed region.
3. The annotation support device according to claim 1, wherein: The annotation input accepting unit accepts, based on acceptance of a drag-and-drop operation of an object representing annotation information with respect to the found region in the medical image displayed on the display unit, an input of the annotation information on the found region.
4. The annotation support device according to claim 1, wherein: The annotation information includes association information indicating a relationship with other nodes except the node to which the annotation information is associated.
5. The annotation support device according to claim 1, wherein: The annotation support device further comprises: A training data acquisition unit, which acquires a data set including at least one medical image and annotation information correspondingly associated with a node of the medical image from the information managed by the hierarchical structure as a training data set; a model generating unit that uses the training data set acquired by the training data acquiring unit and generates, through machine learning, an annotation guidance model that takes a medical image as an input and outputs annotation information related to the medical image; and The annotation guide displays an output obtained by inputting the medical image, which is a target of annotation, acquired by the image acquisition unit into the annotation guidance model in association with the medical image, which is a target of annotation.
6. The annotation support device according to claim 5, wherein: The training data acquisition unit acquires a data set as a training data set, and the data set includes at least one medical image and annotation information in the information managed by the hierarchical structure, and the annotation information is correspondingly associated with a node of the medical image and at least one of other nodes that are correspondingly associated with the node of the medical image.
7. An annotation support device, supporting generation of annotation information related to a medical image, The medical images are managed according to the attributes of each medical image and in a hierarchical structure, wherein the hierarchical structure takes a medical image as a node and includes at least one of a medical image hierarchy to which the node of the medical image belongs, a hierarchy above the hierarchy of the medical image hierarchy, i.e., an upper hierarchy, and a hierarchy below the hierarchy of the medical image hierarchy, i.e., a lower hierarchy, The annotation support device comprises: A training data acquisition unit is capable of pre-acquiring a medical image corresponding to the annotation information, and acquiring a data set including the medical image and the annotation information corresponding to the medical image as a training data set; a model generating unit that uses the training data set acquired by the training data acquiring unit and generates, through machine learning, an annotation guidance model that takes a medical image as an input and outputs annotation information related to the medical image; and An image acquisition unit that acquires a medical image to which an annotation is to be assigned; a display unit, which displays the medical image acquired by the image acquisition unit; an area designation accepting unit that accepts designation of a visible area, the visible area being a part of the medical image displayed by the display unit and to which the visible area is attached; an annotation guide unit that displays an output obtained by inputting the medical image acquired by the image acquisition unit into the annotation guidance model in association with the medical image to which the annotation is to be assigned and displayed by the display unit; an annotation input receiving unit for receiving input of annotation information related to the medical image to which the annotation is to be given; and an annotation storage unit that stores the annotation information received by the annotation input receiving unit in association with the medical image to which the annotation is to be given, The hierarchical structure includes the medical image hierarchy, the upper hierarchy, and the lower hierarchy, The annotation input receiving unit receives input of annotation information on the observed area, the annotation storage unit stores the observed area information as a node associated with the medical image in the lower layer, and stores the input annotation information in association with the node. The hierarchical structure includes patient information identifying a patient as a node of the upper hierarchy, The annotation input accepting unit accepts input of annotation information indicating patient information of a patient associated with the medical image, and the annotation storage unit stores the input annotation information in association with a node of the patient information.
8. An annotation support method, comprising: an annotation support method in an annotation support device for supporting generation of annotation information related to a medical image; The medical images are managed according to the attributes of each medical image and in a hierarchical structure, wherein the hierarchical structure takes a medical image as a node and includes at least one of a medical image hierarchy to which the node of the medical image belongs, a hierarchy above the hierarchy of the medical image hierarchy, i.e., an upper hierarchy, and a hierarchy below the hierarchy of the medical image hierarchy, i.e., a lower hierarchy, The annotation support method has: An image acquisition step, acquiring a medical image; A display step of displaying the medical image; an area designation accepting step of accepting a designation of a seen area, the seen area being a part of the medical image displayed in the displaying step and an additional seen area; An annotation input accepting step of accepting designation of any one of the nodes of the medical image and the nodes correspondingly associated with the nodes of the medical image in the hierarchical structure, and accepting input of information related to the designated node, i.e., annotation information; and The annotation storage step stores the annotation information in a corresponding relationship with the specified node. The hierarchical structure includes the medical image hierarchy, the upper hierarchy, and the lower hierarchy, In the annotation input accepting step, input of annotation information on the observed area is accepted, and in the annotation storing step, the observed information representing the observed area is stored as a node correspondingly associated with the medical image in the lower layer, and the input annotation information is stored in a correspondingly associated manner with the node. The hierarchical structure includes patient information identifying a patient as a node of the upper hierarchy, In the annotation input accepting step, input of annotation information indicating patient information of a patient associated with the medical image is accepted, and in the annotation storing step, the input annotation information is stored in association with a node of the patient information.
9. A recording medium having an annotation support program recorded thereon, the annotation support program being used to cause a computer to function as an annotation support device for supporting generation of annotation information related to a medical image, The medical images are managed according to the attributes of each medical image and in a hierarchical structure, wherein the hierarchical structure takes a medical image as a node and includes at least one of a medical image hierarchy to which the node of the medical image belongs, a hierarchy above the hierarchy of the medical image hierarchy, i.e., an upper hierarchy, and a hierarchy below the hierarchy of the medical image hierarchy, i.e., a lower hierarchy, The annotation support program enables the computer to implement the following functions: Image acquisition function, to acquire medical images; A display function for displaying the medical image; an area designation accepting function for accepting designation of a seen area, the seen area being a part of the medical image displayed by the display function and to which the seen area is attached; An annotation input accepting function accepts the designation of any one of the nodes of the medical image and the nodes correspondingly associated with the nodes of the medical image in the hierarchical structure, and accepts input of information related to the designated node, i.e., annotation information; and The annotation storage function stores the annotation information in a corresponding relationship with the specified node. The hierarchical structure includes the medical image hierarchy, the upper hierarchy, and the lower hierarchy, In the annotation input accepting function, input of annotation information on the observed area is accepted, and in the annotation storing function, the observed information representing the observed area is stored as a node correspondingly associated with the medical image in the lower layer, and the input annotation information is stored in a corresponding association with the node. The hierarchical structure includes patient information identifying a patient as a node of the upper hierarchy, The annotation input accepting function accepts input of annotation information indicating patient information of a patient associated with the medical image, and the annotation storing function stores the input annotation information in association with a node of the patient information.
