Information storage device, method and program, and analysis record generation device, method and program

Abstract

To provide an information saving device, method, and program and an analysis record generation device, method, and program that, when an image analysis record generated from an image is corrected, make it possible to recognize which of the property information derived by analyzing the image has been corrected.SOLUTION: In a medical information system, an information saving device 20 comprises an analysis unit 22 that analyzes an image and thereby derives a plurality of pieces of property information that represent the properties of a structure of interest included in the image, an analysis record generation unit 23 that generates an image analysis record including at least some of the plurality of pieces of property information, a correction unit 25 that accepts correction of property information by a user, and a save control unit 26 that saves the derived property information and the corrected property information separately.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an information storage device, method, and program, and an analysis record generation device, method, and program.

Background Art

[0002] In recent years, with the progress of medical devices such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices, it has become possible to perform image diagnosis using higher-quality and higher-resolution medical images. In particular, by performing image diagnosis using CT images, MRI images, etc., the lesion area can be accurately analyzed, and appropriate treatment has come to be performed based on the analysis results.

[0003] Also, by using CAD (Computer-Aided Diagnosis) with a learning model trained by machine learning such as deep learning, medical images are analyzed to determine the characteristics of structures of interest such as abnormal shadow candidates included in the medical images, such as shape, density, position, and size, and these are obtained as analysis results. The analysis results obtained by CAD are associated with examination information such as patient name, gender, age, and the modality from which the medical image was obtained, and stored in a database. The medical image and the analysis result are transmitted to the terminal of the radiologist who reads the medical image. The radiologist reads the medical image with reference to the transmitted medical image and analysis result on their own terminal, and creates a reading report.

[0004] On the one hand, with the improvement of the performance of the above-mentioned CT apparatus and MRI apparatus, the number of medical images for radiography is also increasing. However, since the number of radiologists does not keep up with the number of medical images, it is desired to reduce the burden on the radiologists' radiography work. For this reason, various methods for assisting in the creation of medical documents such as radiology reports have been proposed. For example, Japanese Patent Application Laid-Open No. 2019-153250 proposes a method for automatically generating a sentence for inclusion in a radiology report based on information representing the properties of a structure of interest (hereinafter referred to as property information) included in keywords input by a radiologist and the analysis results of medical images. In the method described in Japanese Patent Application Laid-Open No. 2019-153250, a medical sentence (hereinafter referred to as a medical sentence) is created using a learning model in which machine learning such as a recurrent neural network is performed so as to generate a sentence from characters representing the input property information. By automatically generating a medical sentence as in the method described in Japanese Patent Application Laid-Open No. 2019-153250, the burden on the radiologist when creating a medical document such as a radiology report can be reduced.

[0005] By the way, the automatically generated radiology report may be corrected by a radiologist. Also, when performing a comparison over time, when describing the radiology report of the latest medical image, it is often necessary to refer to the radiology report of the past medical image. For this reason, a method for extracting the corrected parts of the corrected radiology report (see Japanese Patent Application Laid-Open No. 2011-125402), and a method for extracting the difference between the radiology report of the past medical image and the radiology report of the latest medical image (see Japanese Patent Application Laid-Open No. 2007-122679) have been proposed.

Summary of the Invention

Problems to be Solved by the Invention

[0006] Incidentally, in a learning model that generates an image analysis record such as a radiology report from a medical image, there are user preferences regarding the content and expression of the analysis record, and it is desirable to construct a learning model that reflects such preferences. Examples of user preferences include, for example, which trait information to reflect in the final analysis record regarding the trait information analyzed from the image.

[0007] However, depending on the content of the teacher data used for learning the learning model or depending on the limitations of learning of the learning model, the generated image analysis record may not match the user's preferences. In such a case, the user needs to correct the generated image analysis record. Here, when the image analysis record is a text, by using the methods described in Japanese Patent Application Laid-Open No. 2011-125402 and Japanese Patent Application Laid-Open No. 2007-122679 to compare the text before and after correction, it is possible to recognize which parts of the text have been corrected.

[0008] However, in the case of the methods described in Japanese Patent Application Laid-Open No. 2011-125402 and Japanese Patent Application Laid-Open No. 2007-122679, although the corrected portions of the image analysis record can be known, it is not possible to recognize which of the trait information obtained by analyzing the image has been corrected. Since the learning model generates an image analysis record from the trait information, it is difficult to construct a learning model according to the user's preferences if it is not known which trait information has been corrected.

[0009] The present disclosure has been made in view of the above circumstances, and an object thereof is to enable recognition of which trait information derived by analyzing an image has been corrected when an image analysis record generated from the image is corrected.

Means for Solving the Problems

[0010] The information storage device according to the present disclosure includes at least one processor, The processor derives a plurality of pieces of trait information representing the traits of the structure of interest included in the image by analyzing the image, Generate an image analysis record including at least a part of a plurality of trait information, Accept user modification to the trait information, and is configured to distinguish and store the derived trait information and the modified trait information.

[0011] Note that in the information storage device according to the present disclosure, the processor may be configured to display the image analysis record on a display.

[0012] Also, in the information storage device according to the present disclosure, the processor may be configured to accept, as a modification, at least one of deletion of trait information included in the displayed image analysis record and addition of trait information not included in the image analysis record.

[0013] Also, in the information storage device according to the present disclosure, the processor may display all or part of the derived trait information on a display, and may be configured to accept a modification based on a user's selection of the displayed trait information.

[0014] Also, the information storage device according to the present disclosure may further include a learning model that has been learned to output an image analysis record when trait information is input.

[0015] Also, in the information storage device according to the present disclosure, the processor may generate a sentence including at least a part of the trait information as an image analysis record.

[0016] Also, in the information storage device according to the present disclosure, the image may be a medical image, and the sentence may be a medical sentence regarding a structure of interest included in the medical image.

[0017] The analysis record generation device according to the present disclosure includes at least one processor, and the processor derives a plurality of trait information representing the traits of the structure of interest included in the target image to be analyzed, configured to generate an object image analysis record including at least a part of the property information by referring to the information stored by the information storage device according to the present disclosure.

[0018] Note that in the analysis record generation device according to the present disclosure, the processor may identify the stored information including the property information that matches the property information derived from the object image, and generate the image analysis record associated with the identified stored information as the object image analysis record.

[0019] Also, in the analysis record generation device according to the present disclosure, the processor may be configured to further generate another object image analysis record including at least a part of the derived property information without referring to the stored information.

[0020] "Generating another object image analysis record without referring to the stored information" means generating another object image analysis record without referring to the stored information.

[0021] Also, in the analysis record generation device according to the present disclosure, the processor may be configured to display the object image analysis record and another object image analysis record on a display.

[0022] Also, in the analysis record generation device according to the present disclosure, the processor may be configured to receive a selection of either the displayed object image analysis record or another object image analysis record.

[0023] Also, in the analysis record generation device according to the present disclosure, the processor may be configured to generate a sentence including at least a part of the property information as the object image analysis record.

[0024] Also, in the analysis record generation device according to the present disclosure, the image may be a medical image, and the sentence may be a medical sentence regarding the structure of interest included in the medical image.

[0025] The information storage method according to the present disclosure analyzes an image to derive a plurality of property information representing the properties of the structure of interest included in the image, generates an image analysis record including at least a part of the plurality of property information, accepts a user's modification to the property information, and stores the derived property information and the modified property information separately.

[0026] The analysis record generation method according to the present disclosure derives a plurality of property information representing the properties of the structure of interest included in the target image to be analyzed, and generates a target image analysis record including at least a part of the property information with reference to the information stored by the information storage device according to the present disclosure.

[0027] Note that the information storage method and the analysis record generation method according to the present disclosure may be provided as a program for causing a computer to execute them.

Advantages of the Invention

[0028] According to the present disclosure, when an image analysis record generated from an image is modified, it is possible to recognize which property information derived by analyzing the image has been modified.

Brief Description of the Drawings

[0029]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. First, the configuration of a medical information system 1 to which an information preservation device and an analysis record generation device according to this embodiment are applied will be described. FIG. 1 is a diagram showing a schematic configuration of the medical information system 1. The medical information system 1 shown in FIG. 1 is based on an examination order from a doctor in a medical department using a known ordering system, and performs imaging of the imaging target site of the subject, storage of medical images acquired by imaging, reading of medical images by a radiologist and creation of a reading report, and viewing of the reading report by the doctor in the medical department of the requester and detailed observation of the medical images to be read.

[0031] As shown in FIG. 1, the medical information system 1 includes a plurality of imaging devices 2, a plurality of reading WS (WorkStation) 3 which are reading terminals, a medical WS 4, an image server 5, an image database (hereinafter referred to as image DB (DataBase)) 6, a report server 7, and a report database (hereinafter referred to as report DB) 8, which are connected to be communicable with each other via a wired or wireless network 10.

[0032] Each device is a computer installed with an application program for functioning as a component of the medical information system 1. The application program is recorded and distributed on a recording medium such as a DVD (Digital Versatile Disc) and a CD-ROM (Compact Disc Read Only Memory), and is installed from the recording medium into the computer. Alternatively, it is stored in a storage device of a server computer connected to the network 10 or in network storage in a state accessible from the outside, and is downloaded and installed into the computer in response to a request.

[0033] The imaging device 2 is a device (modality) that generates a medical image representing a diagnostic target site by imaging a site of a subject to be diagnosed. Specifically, it is a simple X-ray imaging device, a CT device, an MRI device, a PET (Positron Emission Tomography) device, etc. The medical image generated by the imaging device 2 is transmitted to the image server 5 and stored in the image DB 6.

[0034] The reading WS 3 is a computer used, for example, by a radiologist in the radiology department for reading medical images and creating reading reports, and includes the information storage device and the analysis record generation device (hereinafter represented by the information storage device) 20 according to the present embodiment. In the reading WS 3, a request for viewing a medical image to the image server 5, various image processes on the medical image received from the image server 5, display of the medical image, and reception of input of findings text regarding the medical image are performed. Also, in the reading WS 3, analysis processing on the medical image and the input findings text, support for creating a reading report based on the analysis result, registration requests and viewing requests for the reading report to the report server 7, and display of the reading report received from the report server 7 are performed. These processes are performed by the reading WS 3 executing a software program for each process.

[0035] The medical WS4 is a computer used by physicians in the medical department for detailed observation of images, viewing of radiology reports, creation of electronic medical records, etc., and is composed of a processing device, a display device such as a display, and an input device such as a keyboard and a mouse. In the medical WS4, a request to view an image from the image server 5, display of the image received from the image server 5, a request to view a radiology report from the report server 7, and display of the radiology report received from the report server 7 are performed. These processes are carried out by the medical WS4 executing software programs for each process.

[0036] The image server 5 is a general-purpose computer installed with a software program that provides the functions of a database management system (DBMS). The image server 5 also has a storage in which the image DB6 is configured. This storage may be a hard disk device connected by the image server 5 and a data bus, or may be a disk device connected to a NAS (Network Attached Storage) and a SAN (Storage Area Network) connected to the network 10. When the image server 5 receives a registration request for a medical image from the imaging device 2, it formats the medical image into a database format and registers it in the image DB6.

[0037] In the image DB6, the image data of the medical images acquired by the imaging device 2 and the attached information are registered. The attached information includes, for example, an image ID (identification) for identifying individual medical images, a patient ID for identifying the subject, an examination ID for identifying the examination, a unique ID (UID: unique identification) assigned to each medical image, the examination date when the medical image was generated, the examination time, the type of imaging device used in the examination for acquiring the medical image, patient information such as the patient's name, age, gender, etc., the examination site (imaging site), imaging information (imaging protocol, imaging sequence, imaging method, imaging conditions, use of contrast agent, etc.), information such as a series number or a collection number in the case where a plurality of medical images are acquired in one examination.

[0038] Also, when the image server 5 receives a viewing request from the reading WS3 and the medical treatment WS4 via the network 10, it searches for the medical images registered in the image DB6 and transmits the searched medical images to the reading WS3 and the medical treatment WS4 that are the request sources.

[0039] In the report server 7, a software program that provides the functions of a database management system is incorporated into a general-purpose computer. When the report server 7 receives a registration request for a reading report from the reading WS3, it formats the reading report into a format for the database and registers it in the report DB8.

[0040] In the report DB8, a reading report including at least the findings text created by the reading doctor using the reading WS3 is registered. The reading report may include, for example, the medical image to be read, the image ID for identifying the medical image, the reading doctor ID for identifying the reading doctor who performed the reading, the lesion name, the location information of the lesion, information for accessing the medical image including a specific area, and information such as the property information.

[0041] In addition, when the report server 7 receives a request to view a reading report from the reading WS3 and the medical treatment WS4 via the network 10, it searches for the reading report registered in the report DB8 and transmits the retrieved reading report to the original requesting reading WS3 and medical treatment WS4.

[0042] In this embodiment, the medical image is a three-dimensional CT image composed of a plurality of tomographic images with the lung as the diagnosis target. By reading the CT image in the reading WS3, a reading report on the abnormal shadow contained in the lung is created as a medical document. Note that the medical image is not limited to a CT image, and any medical image such as an MRI image and a two-dimensional image obtained by a simple X-ray imaging device can be used.

[0043] The network 10 is a wired or wireless local area network that connects various devices within the hospital. When the reading WS3 is installed in another hospital or clinic, the network 10 may be configured to connect the local area networks of each hospital via the Internet or a dedicated line.

[0044] Next, the information storage device and the analysis record generation device according to this embodiment will be described. FIG. 2 illustrates the hardware configuration of the information storage device and the analysis record generation device according to this embodiment. In FIG. 2, the information storage device and the analysis record generation device are represented by the information storage device 20. As shown in FIG. 2, the information storage device 20 includes a CPU (Central Processing Unit) 11, a non-volatile storage 13, and a memory 16 as a temporary storage area. In addition, the information storage device 20 includes a display 14 such as a liquid crystal display, an input device 15 such as a keyboard and a mouse, and a network I / F (InterFace) 17 connected to the network 10. The CPU 11, the storage 13, the display 14, the input device 15, the memory 16, and the network I / F 17 are connected to a bus 18. Note that the CPU 11 is an example of the processor in the present disclosure.

[0045] Storage 13 is implemented by a hard disk drive (HDD), a solid state drive (SSD), flash memory, or the like. The information storage program 12A and the analysis record generation program 12B are stored in the storage 13 as a storage medium. The CPU 11 reads the information storage program 12A and the analysis record generation program 12B from the storage 13 and expands them in the memory 16, and executes the expanded information storage program 12A and analysis record generation program 12B.

[0046] Next, the functional configuration of the information storage device and the analysis record generation device according to the present embodiment will be described. FIG. 3 is a diagram showing the functional configuration of the information storage device and the analysis record generation device according to the present embodiment. As shown in FIG. 3, the information storage device (and analysis record generation device) 20 includes an image acquisition unit 21, an analysis unit 22, an analysis record generation unit 23, a display control unit 24, a correction unit 25, a storage control unit 26, and a communication unit 27. Then, when the CPU 11 executes the information storage program 12A and the analysis record generation program 12B, the CPU 11 functions as the image acquisition unit 21, the analysis unit 22, the analysis record generation unit 23, the display control unit 24, the correction unit 25, the storage control unit 26, and the communication unit 27.

[0047] In the present embodiment, the image acquisition unit 21, the analysis unit 22, the analysis record generation unit 23, the display control unit 24, and the communication unit 27 have a common configuration in the information storage program 12A and the analysis record generation program 12B.

[0048] The image acquisition unit 21 acquires a medical image for creating a reading report from the image server 5 according to an instruction from the input device 15 by the reading doctor who is the operator. The medical image includes a target medical image to be analyzed, which will be described later.

[0049] The analysis unit 22 derives property information representing the properties of a structure of interest, such as an abnormal shadow candidate, included in a medical image by analyzing the medical image. For this purpose, the analysis unit 22 has a first learning model 22A that has been machine-learned to discriminate an abnormal shadow candidate in a medical image and discriminate the properties of the discriminated abnormal shadow candidate. In the present embodiment, the first learning model 22A is a convolutional neural network (CNN) that has been deep-learned using teacher data to discriminate whether each pixel (voxel) in a medical image represents an abnormal shadow candidate, and if it is an abnormal shadow candidate, to discriminate its properties.

[0050] FIG. 4 is a diagram showing an example of teacher data for learning the first learning model. As shown in FIG. 4, the teacher data 30 includes a medical image 32 including an abnormal shadow 31 and property information 33 about the abnormal shadow. In the present embodiment, the abnormal shadow 31 is a pulmonary nodule, and the property information 33 represents a plurality of properties of the pulmonary nodule. For example, as the property information 33, the location of the abnormal shadow, the size of the abnormal shadow, the shape of the boundary (clear and irregular), the type of absorption value (solid type and ground glass type), the presence or absence of spicules, whether it is a tumor or a nodule, the presence or absence of pleural contact, the presence or absence of pleural indentation, the presence or absence of pleural invasion, the presence or absence of a cavity, and the presence or absence of calcification are used. Regarding the abnormal shadow 31 included in the teacher data 30 shown in FIG. 4, as shown in FIG. 4, the property information 33 is such that the location of the abnormal shadow is subpleural in the left lung, the size of the abnormal shadow is 4.2 cm in diameter, the shape of the boundary is irregular, the absorption value is solid type, there are spicules, it is a tumor, there is pleural contact, there is pleural indentation, there is no pleural invasion, there is no cavity, and there is no calcification. In FIG. 4, a '+' is given when there is a presence, and a '-' is given when there is no presence. The first learning model 22A is constructed by learning a neural network using a large number of teacher data as shown in FIG. 4. For example, by using the teacher data 30 shown in FIG. 4, when the medical image 32 shown in FIG. 4 is input, the first learning model 22A is learned to discriminate the abnormal shadow 31 included in the medical image 32 and output the property information 33 shown in FIG. 4 regarding the abnormal shadow 31.

[0051] In addition, as the first learning model 22A, any learning model such as a convolutional neural network or, for example, a support vector machine (SVM) can be used.

[0052] Note that the learning model for detecting abnormal shadow candidates from medical images and the learning model for detecting the property information of abnormal shadow candidates may be separately constructed. Also, the property information derived by the analysis unit 22 is stored in the storage 13. FIG. 5 is a diagram for explaining the property information derived by the analysis unit 22. As shown in FIG. 5, the property information 35 derived by the analysis unit 22 is assumed to be "subpleural of the left lung", "4.2 cm", "irregular", "solid type", "no spicules", "tumor", "with pleural contact", "with pleural indentation", "no pleural invasion", "no cavity", and "no calcification".

[0053] The analysis record generation unit 23 generates an image analysis record using the property information derived by the analysis unit 22. In this embodiment, a medical text is generated as the image analysis record. The analysis record generation unit 23 consists of a second learning model 23A that has been trained to generate a text from the input information. As the second learning model 23A, for example, a recurrent neural network can be used. FIG. 6 is a diagram showing a schematic configuration of a recurrent neural network. As shown in FIG. 6, the recurrent neural network 40 consists of an encoder 41 and a decoder 42. The property information derived by the analysis unit 22 is input to the encoder 41. For example, the property information of "subpleural of the left lung", "4.2 cm", "spicules +", and "tumor" is input to the encoder 41. The decoder 42 has been trained to form text information into a text and generates a text from the input property information. Specifically, from the above-mentioned property information of "subpleural of the left lung", "4.2 cm", "spicules +", and "tumor", a medical text of "A 4.2 cm diameter tumor with spicules is observed in the subpleural region of the left lung." is generated. Note that in FIG. 6, "EOS" indicates the end of the text (End Of Sentence).

[0054] Thus, in order to output a medical text by inputting trait information, the recurrent neural network 40 is constructed by learning an encoder 41 and a decoder 42 using a large number of teacher data consisting of combinations of trait information and medical texts.

[0055] The display control unit 24 displays the medical text generated by the analysis record generation unit 23 on the display 14. FIG. 7 is a diagram showing an example of a display screen of a medical text in the present embodiment. As shown in FIG. 7, the display screen 50 includes an image display area 51 and a text display area 52. In the image display area 51, the slice image SL1 that most easily identifies the abnormal shadow candidate detected by the analysis unit 22 is displayed. The slice image SL1 includes an abnormal shadow candidate 53, and the abnormal shadow candidate 53 is surrounded by a rectangular area 54.

[0056] In the text display area 52, the medical text 55 generated by the analysis record generation unit 23 is displayed. The medical text 55 is "A tumor with an irregular maximum diameter of 4.2 cm is observed under the pleura of the left lung. It is in contact with the chest wall, and pleural indentation is observed, but infiltration is not observed." Note that the trait information used in the medical text 55 is "under the pleura of the left lung", "irregular", "4.2 cm", "tumor", "presence of chest wall contact", "presence of pleural indentation", and "absence of pleural infiltration" among the trait information derived by the analysis unit 22.

[0057] Below the image display area 51, a correction button 58A and a confirmation button 58B are displayed.

[0058] The radiologist reads the abnormal shadow candidate 53 in the slice image SL1 displayed in the image display area 51 and determines the appropriateness of the medical text 55 displayed in the text display area 52.

[0059] When the radiologist desires to correct the medical text 55, the correction button 58A is selected using the input device 15. Thereby, the correction unit 25 accepts the correction by the radiologist for the property information. That is, the medical text 55 displayed in the text display area 52 becomes in a state where it can be manually corrected by the input from the input device 15. Also, by selecting the confirmation button 58B, the medical text 55 displayed in the text display area 52 can be confirmed with its content. In this case, the medical text 55 is transcribed into the radiology report, and the radiology report in which the medical text 55 is transcribed is transmitted to and stored in the report server 7 together with the slice image SL1.

[0060] When the radiologist selects the correction button 58A to correct the medical text 55, if there are properties that are included in the abnormal shadow 31 but are lacking in the medical text 55, the radiologist corrects the medical text 55 to add the lacking properties. In this case, the radiologist inputs the lacking properties using the input device 15. For example, in the present embodiment, it is assumed that spiculation is recognized in the abnormal shadow 31, but the description regarding spiculation is lacking in the medical text 55. In this case, the radiologist inputs the property information of "spiculation" using the input device 15. Thereby, the correction unit 25 corrects the medical text 55 so as to add the property information of "spiculation".

[0061] Also, if there are unnecessary properties in the medical text 55 that are not seen in the medical image, or if there are properties that are seen in the medical image but the radiologist considers the description in the medical text 55 to be unnecessary, the radiologist corrects the medical text 55 to delete the unnecessary properties. For example, in the present embodiment, if the property of being in contact with the chest wall is unnecessary, the radiologist deletes the property information of "being in contact with the chest wall" using the input device 15. Thereby, the correction unit 25 corrects the medical text 55 so as to delete the property information of "being in contact with the chest wall".

[0062] FIG. 8 is a diagram showing an example of a display screen of a corrected medical text. As shown in FIG. 8, in the text display area 52, a corrected medical text 59 obtained by correcting the medical text 55 is displayed. The corrected medical text 59 is "A tumor with a maximum transverse diameter of 4.2 cm, which is irregular and has spicules, is observed under the pleura of the left lung. Pleural indentation is observed, but infiltration is not observed."

[0063] Here, when the medical text 55 is corrected as shown in FIG. 8, the character information is corrected such that "without spicules" is changed to "with spicules" and "with chest wall contact" is changed to "without chest wall contact".

[0064] If the radiologist selects the confirmation button 58B without making any corrections, the medical text 55 displayed in the text display area 52 can be confirmed as it is. In this case, the medical text 55 is transcribed into the reading report, and the reading report in which the medical text 55 is transcribed is transmitted to and stored in the report server 7 by the communication unit 27 together with the slice image SL1. Also, when the radiologist selects the confirmation button 58B after the correction, the corrected medical text 59 can be confirmed as it is. In this case, the corrected medical text 59 is transcribed into the reading report, and the reading report in which the corrected medical text 59 is transcribed is transmitted to and stored in the report server 7 by the communication unit 27 together with the slice image SL1 and the storage information 45 described later. In the report server 7, the reading report and the storage information 45 are stored in association with each other.

[0065] The storage control unit 26 stores, in a distinguishable manner in the storage 13, the property information derived by the analysis unit 22 and the corrected property information received by the correction unit 25. FIG. 9 is a diagram for explaining storage information representing the storage result of the property information. As shown in FIG. 9, the storage information 45 is obtained by correcting "without spicules" to "with spicules" and "with chest wall contact" to "without chest wall contact" in the property information derived by the analysis unit 22 shown in FIG. 5. In the storage information 45, a flag of +1 is assigned to the property information corrected from "without" to "with", a flag of -1 is assigned to the property information corrected from "with" to "without", and a flag of 0 is assigned to the property information without correction. As a result, in the storage information 45, the property information derived by the analysis unit 22 and the corrected property information can be distinguished by the flags. The storage information 45 stored in the storage 13 is transmitted to and stored in the report server 7 together with the above-described reading report, as described above.

[0066] On the other hand, when the image acquisition unit 21 of the information storage device 20 acquires a medical image to be analyzed (hereinafter referred to as a target medical image), the analysis unit 22 derives the property information of the target medical image by analyzing the target medical image. Further, the analysis record generation unit 23 generates a target medical text from the property information of the target medical image without referring to the storage information 45 stored in the report server 7, that is, without referring to the storage information 45. Specifically, the analysis record generation unit 23 generates the target medical text using only the property information derived by the analysis unit 22. The target medical text without referring to the storage information 45 corresponds to other target image analysis records of the present disclosure. Further, the analysis record generation unit 23 generates a target medical text as an alternative from the property information of the target medical image with reference to the storage information 45 stored in the report server 7.

[0067] Next, the generation of alternative cases will be described. First, the analysis record generation unit 23 identifies the storage information 45 including the property information that matches the property information of the target medical image derived by the analysis unit 22 by searching in the report server 7. Note that the property information and the storage information 45 include the location and size of the abnormal shadow, but the storage information 45 is identified such that the property items excluding the location and size of the abnormal shadow match. For example, when the property information of the target medical image is "tumor", "with pleural indentation", and "without invasion", the analysis record generation unit 23 searches for the storage information 45 including the property information of "tumor", "with pleural indentation", and "without invasion". Then, the analysis record generation unit 23 acquires the reading report associated with the storage information 45 that matches the property information of the target medical image from the report server 7.

[0068] Here, the matching of the property information means not only the case where all the property information except the location and size of the abnormal shadow among the plurality of property information matches, but also the case where the majority of the plurality of property information, for example, 80% or more, and further 90% or more of the plurality of property information matches.

[0069] Note that when there are multiple pieces of storage information 45 that match the property information of the target medical image, they can be selected according to criteria such as the one with the newer creation date or the one created by the reading doctor operating the reading WS3. Alternatively, the number of reading reports to be acquired can be limited to search for the storage information that matches the property information. Then, the analysis record generation unit 23 rewrites the location and size of the abnormal shadow in the acquired reading report to the location and size of the abnormal shadow included in the property information of the target medical image to generate an alternative case. Note that there may be a case where the storage information 45 that matches the property information of the target medical image is not stored in the report server 7. In such a case, no alternative case is generated in this embodiment.

[0070] The display control unit 24 displays the target medical text and an alternative plan for the target medical text on the display 14. FIG. 10 is a diagram showing the display screen of the target medical text and the alternative plan. Note that only one alternative plan is shown in FIG. 10. As shown in FIG. 10, the display screen 70 includes an image display area 71 and a text display area 72. In the image display area 71, the slice image SL2 that is most likely to identify the abnormal shadow candidate detected by the analysis unit 22 from the target medical image is displayed. The slice image SL2 includes the abnormal shadow candidate 73, and the abnormal shadow candidate 73 is surrounded by a rectangular area 74.

[0071] In the text display area 72, the target medical text 75 and the alternative plan 76 generated by the analysis record generation unit 23 are displayed. The target medical text 75 is "A tumor with a maximum transverse diameter of 4.2 cm, which is irregular and has spicules, is observed under the left lung pleura. Pleural indentation is observed, but infiltration is not observed." The alternative plan 76 is "A tumor with an irregular maximum transverse diameter of 4.2 cm is observed under the left lung pleura. It is in contact with the chest wall, and pleural indentation is observed, but infiltration is not observed."

[0072] Below the image display area 71, a correction button 78A and a confirmation button 78B are displayed.

[0073] The radiologist reads the abnormal shadow candidate 73 in the slice image SL2 displayed in the image display area 71 and determines the appropriateness of the target medical text 75 and the alternative plan 76 displayed in the text display area 72.

[0074] When the radiologist desires to make a correction without agreeing to either the target medical text 75 or the alternative plan 76, the correction button 78A is selected using the input device 15. As a result, the medical text 75 displayed in the text display area 72 can be corrected in the same manner as described above by input from the input device 15.

[0075] On the other hand, when the radiologist adopts either the target medical text 75 or the alternative plan 76, the radiologist can use the input device 15 to select either the target medical text 75 or the alternative plan 76 and select the confirmation button 78B to confirm either the target medical text 75 or the alternative plan 76 with its content. In this case, either the selected target medical text 75 or the alternative plan 76 is transcribed into the radiology report, and the radiology report with the text transcribed is transmitted to and stored in the report server 7 together with the slice image SL2.

[0076] The communication unit 27 exchanges information between the information storage device 20 and an external device via the network I / F 17.

[0077] Next, the processing performed in this embodiment will be described. FIG. 11 is a flowchart of the information storage process performed in this embodiment. It is assumed that the medical image to be radiographed is acquired from the image server 5 by the image acquisition unit 21 and stored in the storage 13. The process is started when an instruction to create a radiology report is given by the radiologist, and the analysis unit 22 analyzes the medical image to derive trait information representing the traits of structures of interest such as abnormal shadow candidates included in the medical image (step ST1). Next, the analysis record generation unit 23 generates a medical text regarding the medical image as an image analysis record based on the trait information (step ST2). Subsequently, the display control unit 24 displays the medical text generated by the analysis record generation unit 23 in the text display area 52 of the display screen 50 displayed on the display 14 (step ST3).

[0078] Next, the display control unit 24 determines whether the correction button 58A displayed on the display screen 50 has been selected (step ST4). If step ST4 is affirmed, the correction unit 25 accepts correction of the property information included in the medical text displayed in the text display area 52 using the input device 15 (step ST5). Subsequently, the correction unit 25 determines whether the confirmation button 58B has been selected (step ST6). If step ST6 is negated, the process returns to step ST5 and correction continues to be accepted. If step ST6 is affirmed, the storage control unit 26 stores the derived property information and the corrected property information separately in the storage 13 (step ST7). Then, the display control unit 24 posts the corrected medical text to the radiology report, and the communication unit 27 transmits the radiology report with the corrected medical text to the report server 7 together with the slice image SL1 (radiology report transmission: step ST8), and the process ends.

[0079] On the other hand, if step ST4 is negated, the display control unit 24 determines whether the confirmation button 58B has been selected (step ST9). If step ST9 is negated, the process returns to step ST4. If step ST9 is affirmed, the process proceeds to step ST8, the display control unit 24 posts the medical text to the radiology report, and the communication unit 27 transmits the radiology report with the medical text to the report server 7 together with the slice image SL1, and the process ends.

[0080] Next, the process performed when the stored information in which the derived property information and the corrected property information are stored separately is stored in the storage 13 will be described. FIG. 12 is a flowchart of the analysis record generation process performed in the present embodiment. It is assumed that the medical image to be radiographed is acquired from the image server 5 by the image acquisition unit 21 and stored in the storage 13. Also, it is assumed that the stored information 45 in which the derived property information and the corrected property information are stored separately is also stored in the storage 13.

[0081] The process is started when the reading doctor gives instructions for creating a reading report. The analysis unit 22 analyzes the medical image to derive property information representing the properties of the structure of interest such as abnormal shadow candidates included in the medical image (step ST11). Next, the analysis record generation unit 23 generates a target medical text related to the target medical image as an image analysis record based on the property information derived by the analysis unit 22 without referring to the storage information 45, that is, without referring to the storage information 45 (step ST12). Further, the analysis record generation unit 23 generates a target medical text related to the target medical image as an alternative plan by referring to the storage information 45 stored in the report server 7 (step ST13). Subsequently, the display control unit 24 displays the target medical text 75 and the alternative plan 76 generated by the analysis record generation unit 23 in the text display area 52 of the display screen 50 displayed on the display 14 (medical text display: step ST14).

[0082] Next, the display control unit 24 accepts a selection of either the target medical text 75 or the alternative plan 76 (step ST15). Further, the display control unit 24 determines whether or not the correction button 78A displayed on the display screen has been selected (step ST16). If step ST16 is affirmed, the correction unit 25 accepts correction of the property information included in the selected medical text using the input device 15 (step ST17). Subsequently, the correction unit 25 determines whether or not the confirmation button 78B has been selected (step ST18). If step ST18 is negated, the process returns to step ST17 and correction continues to be accepted. If step ST18 is affirmed, the storage control unit 26 stores the derived property information and the corrected property information separately in the storage 13 (step ST19). Then, the display control unit 24 posts the selected and corrected medical text to the reading report, and the communication unit 27 transmits the reading report with the medical text posted thereto to the report server 7 together with the slice image SL1 (reading report transmission: step ST20), and the process ends.

[0083] On the other hand, when step ST16 is negated, the display control unit 24 determines whether or not the confirmation button 78B has been selected (step ST21). If step ST21 is negated, the process returns to step ST16. If step ST21 is affirmed, the process proceeds to step ST20, where the display control unit 24 posts the selected medical text to the radiography report, and the communication unit 27 transmits the radiography report with the medical text posted thereto to the report server 7 together with the slice image SL1, and the process ends.

[0084] As described above, in the present embodiment, a plurality of property information representing the properties of the region of interest included in the image is derived, an image analysis record including at least a part of the plurality of property information is generated, user modification to the property information is accepted, and the derived property information and the modified property information are stored separately. For this reason, by referring to the stored storage information, when the image analysis record generated from the image is modified, it is possible to recognize which property information derived by analyzing the image has been modified.

[0085] In addition, regarding the target medical image, in addition to the target medical text generated based on the property information derived from the target medical image without referring to the storage information 45, by displaying an alternative plan referring to the storage information, the options for the text to be posted to the radiography report can be increased. For this reason, the radiologist can post the medical text describing the desired property information to the radiography report.

[0086] Note that in the above embodiment, the target medical text is generated using only the property information derived by the analysis unit 22 for the target medical image, but the present invention is not limited to this. Instead of generating the target medical text using only the property information derived by the analysis unit 22 for the target medical image, an alternative plan generated by referring only to the storage information 45 may be used as the target medical text.

[0087] In the above-described embodiment, although the medical text 55 displayed in the text display area 52 of the display screen 50 is accepted for modification using the input device 15, it is not limited thereto. FIG. 13 is a diagram showing another example of the display screen of the medical text in the present embodiment. As shown in FIG. 13, the display screen 80 includes an image display area 81, a property information display area 82, and a text display area 83. In the image display area 81, a slice image SL3 that most easily identifies the abnormal shadow candidate detected by the analysis unit 22 is displayed. The slice image SL3 includes an abnormal shadow candidate 84, and the abnormal shadow candidate 84 is surrounded by a rectangular area 85.

[0088] In the property information display area 82, buttons 82A to 82I for specifying the shape of the boundary (clear and irregular), the type of absorption value (solid type and ground glass type), the presence or absence of spicules, whether it is a tumor or a nodule, the presence or absence of pleural contact, the presence or absence of pleural indentation, the presence or absence of pleural invasion, the presence or absence of a cavity, and the presence or absence of calcification are displayed.

[0089] In the text display area 83, a medical text 86 generated by the analysis record generation unit 23 is displayed. The medical text 86 is "A tumor with an irregular maximum diameter of 4.2 cm is recognized under the pleura of the left lung. It is in contact with the chest wall, and pleural indentation is recognized, but infiltration is not recognized." Note that the property information used in the medical text 86 is "under the pleura of the left lung", "irregular", "4.2 cm", "tumor", "presence of chest wall contact", "presence of pleural indentation", and "absence of pleural invasion" among the property information derived by the analysis unit 22.

[0090] Note that a modification button 88A and a confirmation button 88B are displayed below the image display area 81. Since the functions of the modification button 88A and the confirmation button 88B are the same as those of the above-described modification buttons 58A, 78A and confirmation buttons 58B, 78B, detailed description thereof is omitted here.

[0091] The radiologist can correct the medical text 86 by selecting a desired button for the property information displayed in the property information display area 82. For example, by selecting button 82C, "spicula absent" can be corrected to "spicula present". Also, by selecting button 82E, "pleural contact present" can be corrected to "pleural contact absent". As a result, in the text display area 83, the corrected medical text "A tumor with a maximum transverse diameter of 4.2 cm, irregular in shape and having spicula, is observed under the pleura of the left lung. Pleural indentation is observed, but infiltration is not observed." will be displayed.

[0092] By correcting the property information included in the medical text on such a display screen, it becomes possible to distinguish and save the derived property information and the corrected property information, similar to the above-described embodiment.

[0093] In the above-described embodiment, by generating a medical text using a medical image of the lung as the diagnosis target, support processing for creating a medical document such as a radiology report is performed. However, the diagnosis target is not limited to the lung. In addition to the lung, any part of the human body such as the heart, liver, brain, and limbs can be used as the diagnosis target. In this case, each learning model of the analysis unit 22 and the analysis record generation unit 23 is prepared to perform analysis processing and analysis record processing according to the diagnosis target, a learning model that performs analysis processing and analysis record generation processing according to the diagnosis target is selected, and the generation processing of the analysis record is executed.

[0094] Also, in the above-described embodiment, when creating a radiology report as an analysis record, the technology of the present disclosure is applied. However, it goes without saying that the technology of the present disclosure can also be applied when creating a medical document other than a radiology report, such as an electronic medical record and a diagnosis report, as an analysis record.

[0095] In addition, in the above-described embodiment, an image analysis record is generated using a medical image, but the present invention is not limited thereto. Of course, the technology of the present disclosure can also be applied when generating an image analysis record for any image other than a medical image. For example, even when analyzing an image of a chemical formula of a compound, deriving the types of cyclic hydrocarbons and functional groups as trait information, and generating the name of the compound as an image analysis record from the derived trait information, the technology of the present disclosure can be applied.

[0096] In addition, in each of the above embodiments, for example, as a hardware structure of a processing unit (Processing Unit) that executes various processes such as an image acquisition unit 21, an analysis unit 22, an analysis record generation unit 23, a display control unit 24, a correction unit 25, a storage control unit 26, and a communication unit 27, the following various processors (Processor) can be used. As described above, in addition to a CPU, which is a general-purpose processor that executes software (program) and functions as various processing units, the above various processors include a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration specifically designed to execute specific processes, such as an ASIC (Application Specific Integrated Circuit).

[0097] One processing unit may be configured by one of these various processors, or may be configured by a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). Further, a plurality of processing units may be configured by one processor.

[0098] As an example of configuring a plurality of processing units with a single processor, first, as represented by computers such as clients and servers, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Second, as represented by a System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with a single integrated circuit (IC) chip is used. Thus, various processing units are configured using one or more of the above various processors as a hardware structure.

[0099] Furthermore, as a hardware structure of these various processors, more specifically, circuitry combining circuit elements such as semiconductor elements can be used. Hereinafter, the appended claims of the present application will be described. (Appended Claim 1) An information storage device comprising at least one processor, wherein the processor derives a plurality of property information representing the properties of the structure of interest included in the image by analyzing the image, generates an image analysis record including at least a part of the plurality of property information, receives a user's modification to the property information, and is configured to separately store the derived property information and the modified property information. (Appended Claim 2) The information storage device according to appended claim 1, wherein the processor is configured to display the image analysis record on a display. (Appended Claim 3) The information storage device according to appended claim 2, wherein the processor is configured to receive, as the modification, at least one of deletion of property information included in the displayed image analysis record and addition of property information not included in the image analysis record. (Appended Claim 4) The processor displays all or part of the derived property information on the display, The information storage device according to appended claim 2, configured to receive the modification based on the selection of the user of the presented trait information. (Appended claim 5) The information storage device according to any one of appended claims 1 to 4, further comprising a learning model trained to output the image analysis record when the trait information is input. (Appended claim 6) The information storage device according to any one of appended claims 1 to 5, wherein the processor generates a sentence including at least a part of the trait information as the image analysis record. (Appended claim 7) The information storage device according to appended claim 6, wherein the image is a medical image, and the sentence is a medical sentence regarding the structure of interest included in the medical image. (Appended claim 8) Comprising at least one processor, The processor derives a plurality of trait information representing the traits of the structure of interest included in the target image to be analyzed, An analysis record generation device configured to generate a target image analysis record including at least a part of the trait information with reference to the information stored by the information storage device according to any one of appended claims 1 to 7. (Appended claim 9) The analysis record generation device according to appended claim 8, wherein the processor identifies the stored information including the trait information that matches the trait information derived from the target image, and generates the image analysis record associated with the identified stored information as the target image analysis record. (Appended claim 10) The analysis record generation device according to appended claim 8 or 9, wherein the processor is further configured to generate another target image analysis record including at least a part of the derived trait information without referring to the stored information. (Appended claim 11) The analysis record generation device according to appended claim 10, wherein the processor is configured to display the target image analysis record and the other target image analysis record on a display. (Appended claim 12) The analysis record generation device according to appended claim 11, wherein the processor is configured to receive a selection of either the displayed target image analysis record or another target image analysis record. (Appended claim 13) The analysis record generation device according to any one of appended claims 8 to 12, wherein the processor is configured to generate, as the target image analysis record, a sentence including at least a part of the property information. (Appended claim 14) The analysis record generation device according to appended claim 13, wherein the image is a medical image and the sentence is a medical sentence regarding the structure of interest included in the medical image. (Appended claim 15) By analyzing an image, a plurality of property information representing the properties of the structure of interest included in the image is derived, An image analysis record including at least a part of the plurality of property information is generated, User modification to the property information is received, An information storage method for separately storing the derived property information and the modified property information. (Appended claim 16) A plurality of property information representing the properties of the structure of interest included in the target image to be analyzed is derived, An analysis record generation method for generating a target image analysis record including at least a part of the property information with reference to the information stored by the information storage device according to any one of appended claims 1 to 7. (Appended claim 17) A procedure for deriving, by analyzing an image, a plurality of property information representing the properties of the structure of interest included in the image, and A procedure for generating an image analysis record including at least a part of the plurality of property information, and A procedure for receiving user modification to the property information, and An information storage program for causing a computer to execute a procedure for separately storing the derived property information and the modified property information. (Appended claim 18) A procedure for deriving a plurality of property information representing the properties of the structure of interest included in the target image to be analyzed, and An analysis record generation program for causing a computer to execute a procedure of generating an object image analysis record including at least a part of the property information by referring to the information stored by the information storage device according to any one of Supplementary Items 1 to 7.

Explanation of Signs

[0100] 1 Medical information system 2 Imaging device 3 Reading WS 4 Diagnosis WS 5 Image server 6 Image DB 7 Report server 8 Report DB 10 Network 11 CPU 12A Information storage program 12B Analysis record generation program 13 Storage 14 Display 15 Input device 16 Memory 17 Network I / F 18 Bus 20 Information storage device 21 Image acquisition unit 22 Analysis unit 22A First learning model 23 Analysis record generation unit 23A Second learning model 24 Display control unit 25 Correction unit 26 Storage control unit 27 Communication unit 30 Teacher data 31 Abnormal shadow 32 Medical image 33,35 Property information 40 Recurrent neural network 41 Encoder 42 Decoder 45 Stored information 50,70,80 Display screen 51,71,81 Image display area 52,72,83 Article display area 53,73,84 Abnormal shadow candidate 54,74,85 Rectangular area 55,86 Medical article 58A,78A,88A Correction button 58B,78B,88B Confirmation button 59 Revised medical article 75 Target medical article 76 Alternative plan 82 Characteristic information display area 82A~82I Button SL1~SL3 Slice image

Claims

1. comprising at least one processor, the processor derives a plurality of trait information representing the traits of the structure of interest included in the target image to be analyzed as first trait information, by analyzing the image to be read, a plurality of trait information representing the traits of the structure of interest included in the image to be read is derived as second trait information, an image analysis record including at least a part of the plurality of second trait information is generated, a correction by a user to the second trait information is received, and the derived second trait information and the corrected second trait information are distinguished and stored in association with the image analysis record, and referring to the information stored by the information storage device configured to do so, an object image analysis record including at least a part of the first trait information is generated. An analysis record generation device configured to generate the record.

2. The processor of claim 1, wherein the processor identifies the stored information including second trait information that matches the first trait information derived from the target image, and generates an image analysis record associated with the identified stored information as the target image analysis record.

3. The processor of claim 1 or 2, further configured to generate another object image analysis record including at least a part of the derived first trait information with the stored information being non-referred.

4. The processor of claim 3, configured to display the target image analysis record and the other target image analysis record on a display.

5. The processor of claim 4, configured to receive a selection of either the displayed target image analysis record or the other target image analysis record.

6. The processor of any one of claims 1 to 5, configured to generate a text including at least a part of the first trait information as the target image analysis record.

7. The target image to be analyzed and the image to be read are medical images, and the text is a medical text regarding the structure of interest included in the medical image. The analysis record generation device of claim 6.

8. Deriving a plurality of trait information representing the traits of the structure of interest included in the target image to be analyzed as first trait information, By analyzing an image to be read, a plurality of property information representing the properties of an object of interest included in the image to be read is derived as second property information, an image analysis record including at least a part of the plurality of second property information is generated, a user's modification to the second property information is received, and the derived second property information and the modified second property information are distinguished and stored in association with the image analysis record. An analysis record generation method for generating a target image analysis record including at least a part of the first property information by referring to the information stored by an information storage device configured to do so.

9. A procedure for deriving a plurality of property information representing the properties of an object of interest included in a target image to be analyzed as first property information, A procedure for causing a computer to execute: a procedure for deriving, as second property information, a plurality of property information representing the properties of an object of interest included in an image to be read by analyzing the image to be read; generating an image analysis record including at least a part of the plurality of second property information; receiving a user's modification to the second property information; and generating a target image analysis record including at least a part of the first property information by referring to the information stored by an information storage device configured to distinguish and store the derived second property information and the modified second property information in association with the image analysis record. An analysis record generation program.

Images