Information processing device, information processing method, and program

Abstract

To provide an information processor, a method for processing information, and a program which can properly perform processing regarding a key image to be related to a sentence.SOLUTION: An information processor includes a processor and a memory. The information processor receives a sentence including a result of diagnosis of an image, specifies a relation of at least two words in the sentence, and determines at least one of whether relating the key image to the sentence based on the image is necessary and a candidate of the key word to be related to the sentence on the basis of the relation of at least two words.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program, and particularly to a technology for analyzing sentences including image diagnosis results. [Background Art]

[0002] In response to an image diagnosis request from an attending physician in a clinical department, a radiological image interpreter diagnoses medical images and creates an interpretation report describing the presence or absence of abnormalities. In this process, the work of attaching key images to the interpretation report imposes a burden.

[0003] To address such a problem, Patent Document 1 discloses a system that, when a character string is input to a finding display area, extracts a registered term or synonym registered in a dictionary from a plurality of words constituting the character string, and identifies an image related to the extracted character string from a plurality of images related to a patient to be interpreted.

[0004] Further, Patent Document 2 describes that, in a system that generates an interpretation report by voice input, when a preset word is detected, a finding sentence is generated based on the recognized word history, and a slice image displayed when the voice is uttered is associated with the finding sentence. [Prior Art Documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2015-162082 [Patent Document 2] Japanese Unexamined Patent Application Publication No. 2018-028562 [Summary of the Invention] [Problem to be Solved by the Invention]

[0006] However, the technologies described in Patent Documents 1 and 2 do not analyze the entire word sequence and voice input, and therefore have the problem that they may not be able to select an appropriate image.

[0007] Furthermore, the technology described in Patent Document 1 had the problem of always selecting an image even when an image was not needed.

[0008] This invention has been made in view of these circumstances, and aims to provide an information processing device, an information processing method, and a program that appropriately perform processing related to key images associated with text. [Means for solving the problem]

[0009] One embodiment of an information processing device for achieving the above objective comprises at least one processor and at least one memory for storing instructions to be executed by the at least one processor, wherein the at least one processor receives a document containing the diagnostic results of an image, identifies the relationship between two or more words contained in the document, and determines, based on the relationship between the two or more words, at least one of the following: whether or not to associate a key image based on an image with the document, and candidate key images to associate with the document. According to this embodiment, since at least one of whether or not to associate a key image with a document, and candidate key images to associate with the document is determined, processing related to key images to be associated with a document can be performed appropriately.

[0010] Furthermore, another embodiment of the information processing device for achieving the above objectives may include at least one processor and at least one memory for storing instructions to be executed by the at least one processor, wherein the at least one processor is an information processing device that receives a document containing the diagnostic results of an image, identifies the relationship between two or more words contained in the document, determines whether it is necessary to associate a key image based on the image with the document based on the relationship between the two or more words, and determines candidate key images to associate with the document if it is determined that association is necessary. In this embodiment as well, processing related to key images to be associated with a document can be performed appropriately.

[0011] Note that "text" includes one or more sentences.

[0012] Preferably, the two or more words include at least two of the following: words representing areas of interest, words representing facts, words representing change information, words representing location, words representing size, words representing characteristics, and words representing shooting conditions.

[0013] Preferably, two or more words include a word representing the region of interest and a word representing the factual status of the region of interest, and at least one processor determines that association with a key image is necessary when the factual status affirms the existence of the region of interest, and determines that association with a key image is unnecessary when the factual status denies the existence of the region of interest.

[0014] The words representing change information preferably include words representing change information for at least one of size and quantity.

[0015] Preferably, at least one processor extracts candidate key images from the image based on their position.

[0016] Preferably, at least one processor accepts two or more images with different shooting conditions and extracts candidate key images from among the two or more images.

[0017] The images are medical images, and two or more words include words representing a disease name. Preferably, at least one processor extracts candidate key images based on the disease name.

[0018] The image is a medical image, and two or more words include a word representing the region of interest and a word representing the degree of malignancy of the region of interest. Preferably, at least one processor determines that association with a key image is necessary when the degree of malignancy confirms malignancy of the region of interest, and determines that association with a key image is unnecessary when the degree of malignancy negates malignancy of the region of interest.

[0019] Preferably, the at least one processor causes key image candidates to be displayed on a display, accepts an operation by a user, and associates the key image candidate with the sentence as a key image in accordance with the operation.

[0020] The image is a three-dimensional image, and preferably, the at least one processor causes a slice image at any slice position of the three-dimensional image to be displayed on the display as a key image candidate, accepts a change of the slice position of the key image candidate by a user, and associates the slice image at the changed slice position with the sentence as a key image.

[0021] One aspect of an information processing method for achieving the above object is an information processing method comprising: a reception step of receiving a sentence including an image diagnosis result; an extraction step of extracting two or more words from the sentence; a specification step of specifying a relationship between the two or more words; and a determination step of determining at least one of whether association of a key image based on the image with the sentence is necessary and a candidate for a key image to be associated with the sentence, based on the two or more words and the relationship. According to this aspect, since at least one of whether association of a key image with the sentence is necessary and a candidate for a key image to be associated with the sentence is determined, processing related to a key image to be associated with the sentence can be appropriately performed.

[0022] One aspect of a program for achieving the above object is a program for causing a computer to execute the above information processing method. The aspect may also include a computer-readable non-transitory storage medium having the program recorded thereon. According to this aspect, since at least one of whether association of a key image with the sentence is necessary and a candidate for a key image to be associated with the sentence is determined, processing related to a key image to be associated with the sentence can be appropriately performed. [Effects of the Invention]

[0023] According to the present invention, processing related to a key image to be associated with a sentence can be appropriately performed. Brief Description of the Drawings

[0024] [Figure 1] Figure 1 is an overall configuration diagram of a medical information processing system. [Figure 2] Figure 2 is a block diagram showing the configuration of a medical information processing apparatus. [Figure 3] Figure 3 is a flowchart showing a medical information processing method using the medical information processing system 10. [Figure 4] Figure 4 is a diagram showing an example of a description of an interpretation image and an interpretation report. [Figure 5] Figure 5 is a diagram showing a structuring result obtained by structuring a finding sentence of an interpretation report through natural language processing. [Figure 6] Figure 6 is a diagram showing two key images extracted from an image. [Figure 7] Figure 7 is a diagram showing an example of a description of an interpretation image and an interpretation report. [Figure 8] Figure 8 is a diagram showing a structuring result of a finding sentence of an interpretation report. [Figure 9] Figure 9 is a diagram showing an interpretation result. [Figure 10] Figure 10 is a diagram showing an interpretation result. [Figure 11] Figure 11 is a diagram showing an interpretation result. [Figure 12] Figure 12 is a diagram showing an interpretation result. [Figure 13] Figure 13 is a diagram showing an interpretation result. [Figure 14] Figure 14 is a diagram showing an interpretation result. [Figure 15] Figure 15 is a diagram for explaining structuring of a finding sentence. [Figure 16] Figure 16 is a diagram for explaining structuring of a finding sentence. [Figure 17] Figure 17 is a diagram for explaining structuring of a finding sentence. Mode for Carrying Out the Invention

[0025] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0026] <Medical Image Processing System> The medical information processing system according to this embodiment is a system that captures medical images of a subject (patient), receives a report (an example of a "text") containing the diagnostic results of the captured medical images, analyzes the entire received report, and processes key images associated with the report based on the results of the analysis.

[0027] Figure 1 is an overall diagram of the medical information processing system 10. As shown in Figure 1, the medical information processing system 10 is comprised of a medical imaging device 12, a medical image database 14, a medical information processing device 16, an image interpretation report database 18, and a user terminal 20.

[0028] The medical imaging equipment 12, the medical image database 14, the medical information processing device 16, the image interpretation report database 18, and the user terminal 20 are connected via a network 22 to enable data transmission and reception. The network 22 includes a wired or wireless LAN (Local Area Network) for communication connection of various devices within the medical institution. The network 22 may also include a WAN (Wide Area Network) connecting LANs of multiple medical institutions.

[0029] Medical imaging equipment 12 is an imaging device that captures images of the area of ​​a subject to be examined and generates medical images. Examples of medical imaging equipment 12 include X-ray imaging devices, CT (Computed Tomography) devices, MRI (Magnetic Resonance Imaging) devices, PET (Positron Emission Tomography) devices, ultrasound devices, and CR (Computed Radiography) devices using a flat-panel X-ray detector.

[0030] The medical image database 14 is a database for managing medical images taken by medical imaging equipment 12. The medical image database 14 is provided by a computer equipped with a large-capacity storage device for storing medical images. The computer incorporates software that provides the functions of a database management system.

[0031] The format of medical images can conform to the Dicom (Digital Imaging and Communications in Medicine) standard. Medical images may have supplementary information (Dicom tag information) defined in the Dicom standard attached to them. In this specification, the term "image" includes not only the image itself, such as a photograph, but also the image data, which is the signal representing the image.

[0032] The medical information processing device 16 is a device that determines at least one of the following: whether or not a key image needs to be associated with a findings statement, and candidate key images to be associated with the findings statement. The medical information processing device 16 can be a personal computer or a workstation (an example of a "computer"). Figure 2 is a block diagram showing the configuration of the medical information processing device 16. As shown in Figure 2, the medical information processing device 16 comprises a processor 16A, a memory 16B, and a communication interface 16C.

[0033] Processor 16A executes instructions stored in memory 16B. The hardware structure of processor 16A consists of various types of processors as shown below. These types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute software (programs) and act as various functional units; GPUs (Graphics Processing Units), which are processors specialized for image processing; PLDs (Programmable Logic Devices), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing; and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing.

[0034] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different types (for example, multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Alternatively, multiple functional units may be composed of a single processor. Examples of composing multiple functional units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, as is typical of computers such as client or server computers, and this processor acts as multiple functional units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple functional units, on a single IC (Integrated Circuit) chip, as is typical of SoCs (System On Chip). Thus, various functional units are configured as hardware structures using one or more of the above-mentioned various processors.

[0035] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0036] Memory 16B stores instructions for the processor 16A to execute. Memory 16B includes RAM (Random Access Memory) and ROM (Read Only Memory), which are not shown. The processor 16A uses RAM as a working area and executes software using various programs and parameters, including the medical information processing program described later, stored in ROM, and also uses parameters stored in ROM, etc., to execute various processes of the medical information processing device 16.

[0037] The communication interface 16C controls communication with the medical imaging equipment 12, the medical image database 14, the image interpretation report database 18, and the user terminal 20 via the network 22, according to a predetermined protocol.

[0038] The medical information processing device 16 may be a cloud server accessible from multiple medical institutions via the internet. The processing performed by the medical information processing device 16 may be a cloud service with a pay-per-use or fixed-rate system.

[0039] Returning to the explanation of Figure 1, the image interpretation report database 18 is a database that manages image interpretation reports generated by users such as radiologists on the user terminal 20. The image interpretation report includes a report of findings. The report of findings is not limited to sentences separated by periods and other punctuation marks, but may also consist of groups of words. The report of findings may be a single sentence or a text composed of multiple sentences. The image interpretation report may also include key images associated with the report of findings.

[0040] The image interpretation report database 18 is provided by a computer equipped with a large-capacity storage device for storing image interpretation reports. The computer incorporates software that provides the functions of a database management system. The medical image database 14 and the image interpretation report database 18 may be configured on a single computer.

[0041] The user terminal 20 is a terminal device for the user to view and edit the image interpretation report. The user terminal 20 may be, for example, a personal computer. The user terminal 20 may be a workstation or a tablet terminal. The user terminal 20 is equipped with an input device 20A and a display 20B. The user uses the input device 20A to input instructions to the medical information processing system 10. The user terminal 20 also displays medical images and image interpretation reports on the display 20B. Furthermore, the user interprets the medical images displayed on the display 20B (an example of a "diagnosis") and inputs the findings (an example of a "diagnosis result") using the input device 20A.

[0042] <Medical Information Processing Method> Figure 3 is a flowchart illustrating a medical information processing method using the medical information processing system 10. The medical information processing method is realized by the processor 16A executing a medical information processing program stored in memory 16B. The medical information processing program may be provided on a computer-readable non-temporary storage medium. In this case, the medical information processing device 16 may read the medical information processing program from the non-temporary storage medium and store it in memory 16B.

[0043] In the image input process of step ST1, medical images captured by the medical imaging equipment 12 and stored in the medical image database 14 are transmitted to the user terminal 20 operated by the user. The medical information processing device 16 causes the user terminal 20 to receive the medical images (an example of "acceptance") and displays the received medical images on the display 20B. This allows the user to interpret the medical images displayed on the display 20B.

[0044] In the findings input step of step ST2, the user generates findings including the interpretation results for the medical images displayed on the display 20B in the image input step, and inputs them into the user terminal 20 using the input device 20A.

[0045] In the findings structuring process of step ST3, the medical information processing device 16 receives the findings entered in the findings input process (an example of a "reception process"). The receipt of findings may be performed simultaneously with the findings input process (in real time), or past findings from previously performed findings input processes may be received.

[0046] Furthermore, in the findings sentence structuring process, the medical information processing device 16 structures the received findings sentence using known natural language processing (an example of the "extraction process" and an example of the "identification process") and obtains the structuring result. Natural language processing is a technology that allows computers to process natural language used in everyday life, and includes processes such as morphological analysis, which breaks down sentences into words, and syntactic analysis, which analyzes the relationships between words obtained from morphological analysis and constructs a syntactic tree that shows the dependency structure between words. Through structuring, the medical information processing device 16 can extract two or more words from the findings sentence and identify the relationships between two or more words.

[0047] In the attachment necessity determination process of step ST4 (an example of a "determination process"), the medical information processing device 16 determines whether or not to attach a key image to the findings text (an example of an "association") based on the structuring results in the findings text structuring process. A key image is a medical image input in the image input process that is judged to be important for interpretation based on the content of the findings text.

[0048] If the attachment requirement determination process determines that associating a key image with the findings statement is unnecessary, the process in this flowchart is terminated. In other words, the key image will not be associated with the findings statement. On the other hand, if the attachment requirement determination process determines that associating a key image with the findings statement is necessary, the medical information processing device 16 executes the process in step ST5.

[0049] In step ST5, the key image determination step (an example of the "determination step"), the medical information processing device 16 analyzes the medical images input in the image input step based on the structuring results in the findings text structuring step and determines a key image to associate with the findings text. The medical information processing device 16 also associates the determined key image with the findings text.

[0050] The findings statement and the key image associated with the findings statement may be stored as a single image interpretation report in the image interpretation report database 18. Alternatively, only the findings statement may be stored as an image interpretation report in the image interpretation report database 18, and the key image associated with the findings statement may be linked to the image interpretation report by a hyperlink to the medical image database 14 or the like.

[0051] The medical information processing device 16 may analyze the medical images based on the structuring results of the findings text, determine candidate key images to associate with the findings text, and allow the user to confirm the candidate key images. For example, the medical information processing device 16 displays the candidate key images on the display 20B. The user confirms the candidate key images displayed on the display 20B, and if there are no problems, inputs an operation to confirm the candidate key image as the key image using the input device 20A. Through this operation, the medical information processing device 16 associates the candidate key image with the findings text as the key image.

[0052] The medical information processing device 16 may also determine a plurality of key image candidates and display the plurality of key image candidates on the display 20B for the user to select. In this case, the user selects at least one key image from the plurality of key image candidates displayed on the display 20B using the input device 20A. The medical information processing device 16 associates the selected key image candidate with the findings statement as the key image.

[0053] The medical information processing device 16 may display a slice image of any slice position in the 3D image on the display 20B as a candidate key image. In this case, the user may change the slice position of the candidate key image using the input device 20A. The medical information processing device 16 may accept the user's change of slice position and associate the slice image of the changed slice position with the findings statement as the key image.

[0054] Thus, the medical information processing method can determine whether or not it is necessary to associate a key image with a medical report. Furthermore, if it is necessary to associate a key image with a medical report, the medical information processing method can determine candidate key images to associate with the report. In addition, the medical information processing method can associate a key image with a medical report.

[0055] Furthermore, the medical information processing method may omit the step of determining whether or not an attachment is necessary. In other words, the medical information processing device 16 may determine the key image to associate with the findings text without determining whether or not it is necessary to associate the key image with the findings text.

[0056] The following provides further specific examples to explain medical information processing methods.

[0057] [Example 1] This section describes an example of a user interpreting a 3D chest and abdominal CT image. Figure 4 shows an example of the interpreted image and the resulting interpretation report. Diagnostic image ID1, which is the image to be interpreted, is the 3D CT image being interpreted. In other words, diagnostic image ID1 is volume data in which voxels are arranged in a 3D manner, and CT values ​​are stored in each voxel.

[0058] The image interpretation report RP1 includes a findings section and a diagnosis section. The findings section is where the user enters their findings regarding diagnostic image ID 1. The diagnosis section is where the user enters their diagnosis regarding diagnostic image ID 1. In this embodiment, the findings statement includes the text entered in the findings section and the text entered in the diagnosis section.

[0059] As shown in Figure 4, the findings section of the RP1 radiology report states, "A 3 cm lung nodule is observed in the right lung segment 6. Spicula is present. There is no pleural effusion. No lymphadenopathy is observed. A low-attenuation area is observed in liver segment 8." The diagnosis section of the RP1 radiology report states, "Suspected lung cancer."

[0060] Figure 5 shows the structured results obtained by structuring the findings text of the radiology report RP1 using natural language processing. As shown in Figure 5, the structured results show that the findings text, which was written in free text, has been structured as information for each lesion.

[0061] Here, the following are classified based on their relationship: "Lung" under the "Organ" category, "Right Lung" and "S6" under the "Location" category, "Nodule" under the "Lesion" category, "Presence of Lesion" under the "Presence of Lesion" category, "Lung Cancer" under the "Disease Name" category, "Suspected Disease Name" under the "Presence of Disease Name" category, "3cm" under the "Size" category, and "Spicule" under the "Characteristics" category. Note that "presence of existence" refers to the possibility of existence.

[0062] Similarly, the relationship between "Lungs" in the "Organ" category, "Pleural effusion" in the "Lesion" category, and "None" in the "Status of Lesion" category has been identified and classified accordingly. In addition, the relationship between "Liver" in the "Organ" category, "S8" in the "Location" category, "Low-attenuation area" in the "Lesion" category, and "Present" in the "Status of Lesion" category has been identified and classified accordingly. Furthermore, the relationship between "Lymph nodes" in the "Organ" category, "Enlargement" in the "Lesion" category, and "None" in the "Status of Lesion" category has been identified and classified accordingly.

[0063] Thus, the structured results identify two or more words extracted from the findings text and their relationships. The two or more words extracted from the findings text include words representing organs, words representing location, words representing lesions, words representing the fact of lesions, words representing disease names, words representing the fact of disease names, words representing size, and words representing characteristics.

[0064] The medical information processing device 16 determines whether or not to associate the key image with the findings description based on two or more words and their relationships. For example, the medical information processing device 16 determines that an association with the key image is necessary if the factual information confirms the existence of a region of interest, and determines that an association with the key image is unnecessary if the factual information denies the existence of a region of interest. A region of interest is, for example, a lesion, an organ (such as overall deformation), an anatomical region, or an area of ​​characteristic imaging features (such as a low-attenuation area).

[0065] In the example of structured results shown in Figure 5, the factual findings confirm the existence of "nodules" and "low-attenuation areas" in the lesion, which are areas of interest. Therefore, the medical information processing device 16 determines that association with key images is necessary.

[0066] If it is determined that a key image association is necessary, the medical information processing device 16 recognizes organs and lesions from the diagnostic image ID 1 using known image processing based on two or more words and their relationships, and automatically determines a key image to associate with the findings statement.

[0067] Figure 6 shows two key images, IK1 and IK2, extracted from diagnostic image ID 1. From the structured results shown in Figure 5, it can be seen that the lesion is located in two places: a nodule in the lung and a low-attenuation area in the liver. Therefore, the medical information processing device 16 recognizes the nodule in the lung and the low-attenuation area in the liver from diagnostic image ID 1 and automatically determines key images IK1 and IK2 to associate with the findings description.

[0068] If the region of interest is a lesion, the lesion can be extracted using a known CAD (Computer-Aided Diagnosis) method, and the slice image in which the lesion is most prominently displayed should be used as the key image.

[0069] Furthermore, in cases of liver cirrhosis, where the outline becomes uneven, if the region of interest involves organ abnormalities, the liver can be extracted using organ extraction / labeling, and the slice image from the slice position that shows the largest area can be used as the key image.

[0070] If necessary, a Multi-Planar Reconstruction (MPR) image or volume rendering image, different from the slice plane, may be created based on predetermined rules and used as a key image.

[0071] In the example shown in Figure 6, the lung key image IK1 has an arrow-shaped annotation A1 applied to the nodular region, and the liver key image IK2 has a circular frame-shaped annotation A2 applied to the low-attenuation region. In this way, the medical information processing device 16 may perform processing to clearly indicate the region of interest on the key image.

[0072] [Example 2] This section describes an example of a user interpreting images from a dynamic CT scan of the liver. Figure 7 shows an example of the interpreted image and the resulting interpretation report. Diagnostic images ID2, ID3, ID4, and ID5 are cross-sectional images from the dynamic CT scan being interpreted. The imaging phases of diagnostic images ID2, ID3, ID4, and ID5 are "non-contrast," "arterial phase," "portal venous phase," and "equilibrium phase," respectively. Thus, in Example 2, the medical information processing device 16 receives two or more medical images with different imaging phases and displays them on the display 20B.

[0073] The RP2 radiology report, like the RP1 radiology report, includes a findings section and a diagnosis section. The findings section states, "A 35mm early-stage, densely stained mass is observed in liver segment 1. Hepatocellular carcinoma is suspected. Fatty liver is present." The diagnosis section states, "Suspected hepatocellular carcinoma" and "Fatty liver."

[0074] Figure 8 shows the structured findings of the RP2 radiology report. Based on the structured findings shown in Figure 8, the medical information processing device 16 determines the key image. Hepatocellular carcinoma is characterized by appearing white in the arterial phase and black in the equilibrium phase. Therefore, in the case of a "tumor," the medical information processing device 16 determines the key image from the "arterial phase" image. Also, "fatty liver" appears whiter than usual in "non-contrast" images and is not diagnosed using contrast-enhanced images. Therefore, in the case of "fatty liver," the medical information processing device 16 determines the key image from the "non-contrast" image. In this way, the medical information processing device 16 can determine the type of image to be used as the key image from the disease name.

[0075] [Example 3] This section describes an example where a user interpreted images from a dynamic CT scan of the liver, different from that in Example 2. Figure 9 shows the interpretation results. F9A in Figure 9 is the findings statement from the interpretation report. As shown in F9A, the findings statement reads, "A simple, low-attenuation mass is observed in liver segment S1. It shows faint enhancement in the arterial phase and washout in the portal venous phase."

[0076] Figure 9, F9B, shows the structured result of the findings statement from F9A. As shown in F9B, the structured result identifies and classifies the relationships between "tumor" in the "lesion" section and "low attenuation (plain)," "faint enhancement (arterial phase)," and "washout (portal phase)" in the "characteristics" section. In other words, for the tumor, it is extracted that low attenuation was observed on plain CT, faint enhancement was observed in the arterial phase, and washout was observed in the portal phase.

[0077] Thus, two or more words extracted from the observation text include words that represent the shooting time (an example of "shooting conditions").

[0078] Based on the structured results shown in F9B, the medical information processing device 16 determines a key image for "low attenuation" from "simple" images, a key image for "faint staining" from "arterial phase" images, and a key image for "washout" from "portal venous phase" images. Since the disease name is unknown here, the medical information processing device 16 determines multiple images as key images.

[0079] In this manner, the medical information processing device 16 receives two or more medical images taken at different time phases and extracts candidate key images from among the two or more medical images.

[0080] [Example 4] This section describes an example of a user interpreting images from a follow-up examination of a pulmonary nodule. A follow-up examination is an examination performed by a physician on the same patient after a certain period of time has elapsed to check on the patient's progress.

[0081] Figure 10 shows the results of the radiographic interpretation. F10A in Figure 10 is the findings statement from the radiographic interpretation report. As shown in F10A, the findings statement reads, "A solid mass measuring 59 mm in length was found in S4 of the right middle lobe of the lung. Its size has increased compared to the previous examination."

[0082] F10B, shown in Figure 10, is the structured result of the findings text of F10A. As shown in Figure 10A, as a result of the structuring, "tumor" is classified under the "lesion" item and "size increase" is classified under the "comparison" item, with their relationships identified. Based on the "size increase" in this structured result, the medical information processing device 16 determines that it is necessary to associate a key image with the findings text. The medical information processing device 16 may also determine a slice image at the same slice position as the key image from the previous examination's reading report (the key image associated with the findings text of the previous examination) as the key image.

[0083] Here, we use the case where the size is increasing as an example, but the medical information processing device 16 also determines that it is necessary to associate the key image with the findings even when the size is decreasing. Furthermore, in the case of a follow-up examination of a lesion called "pleural effusion," the medical information processing device 16 determines that it is necessary to associate the key image with the findings if the amount of fluid has increased or decreased.

[0084] Thus, the words extracted from the observation text include words representing comparison (an example of "change information"), and the words representing comparison include words representing a comparison of at least one of size and quantity. Furthermore, if the observation text contains words representing comparison and words representing facts, and their relationship is identified, it is determined that an association of a key image with the observation text is necessary.

[0085] If the findings include words representing past areas of interest, words representing comparisons, and words representing factual accuracy, and their relationships are identified, it may be determined that associating a key image with the findings is necessary.

[0086] The words extracted from the findings include words indicating comparison and words indicating malignancy, and it may be determined that a key image should be associated with the findings if the comparison suggests malignancy of the lesion.

[0087] [Example 5] This section describes an example where a user interpreted images from a follow-up examination of a lung nodule different from that in Example 4. Figure 11 shows the results of the interpretation. F11A in Figure 11 is the findings statement from the interpretation report. As shown in F11A, the findings statement reads, "A solid mass measuring 59 mm in length was found in S4 of the right middle lobe of the lung. No significant changes from the previous examination."

[0088] F11B, shown in Figure 11, is the structured result of the findings text of F11A. As shown in Figure 11A, as a result of the structuring, "tumor" is identified and classified under the "lesion" item, and "no significant change" is identified under the "comparison" item. Based on this structured result of "no significant change," the medical information processing device 16 decides that it is not necessary to associate a key image with the findings text.

[0089] Thus, the medical information processing device 16 determines that associating a key image with the findings is unnecessary when the word indicating comparison negates a change over time. However, even if there is no change in the size of the lesion, it may still determine that associating a key image with the findings is necessary.

[0090] [Example 6] This section explains an example of a case where a liver cyst was found as a result of the radiographic interpretation. Figure 12 shows the results of the radiographic interpretation. F12A in Figure 12 is the findings statement from the radiographic interpretation report. As shown in F12A, the findings statement reads, "A cyst was found in the liver."

[0091] Figure 12 shows F12B, which is the structured result of the findings text of F12A. As shown in Figure 12A, as a result of the structuring, the relationship between "organ" (liver), "lesion" (cyst), and "factual" (present) has been identified and classified. Since liver cysts do not require clinical treatment intervention, the medical information processing device 16 determines, based on this structuring result, that it is unnecessary to associate a key image with the findings text.

[0092] Thus, the medical information processing device 16 determines that it is unnecessary to associate a key image with the findings statement when the word describing the lesion negates the need for therapeutic intervention.

[0093] [Example 7] This section describes an example in which a pulmonary nodule exhibiting malignant characteristics was found based on the interpretation of radiographs. Figure 13 shows the results of the radiograph interpretation. F13A in Figure 13 is the findings statement from the radiograph interpretation report. As shown in F13A, the findings statement reads, "A mass measuring 3 cm in its longest diameter was found in S4 of the right middle lobe of the lung. A spicule was observed."

[0094] Figure 13 shows F13B, which is the structured result of the findings text of F13A. As shown in Figure 13A, as a result of the structuring, "tumor" is identified under the "lesion" item, "present" under the "fact of the lesion" item, and "spicules" under the "characteristics" item, and their relationships are identified and classified. Spiculaes are characteristics that indicate malignancy, and the medical information processing device 16 determines that it is necessary to associate a key image with the findings text based on the "spicules" in the structured result.

[0095] Thus, the medical information processing device 16 determines that it is necessary to associate a key image with the findings statement when the word describing the characteristics indicates malignancy.

[0096] [Example 8] This section describes an example in which a pulmonary nodule was found to have benign characteristics based on the interpretation of radiographs. Figure 14 shows the results of the radiograph interpretation. F14A in Figure 14 is the findings statement from the radiograph interpretation report. As shown in F14A, the findings statement reads, "A nodule was found in S4 of the right middle lobe of the lung. Fat was found inside."

[0097] F14B, shown in Figure 14, is the structured result of the findings text of F14A. As shown in Figure 14A, as a result of the structuring, "tumor" is identified and classified under the "lesion" item, "presence of lesion" under the "fact" item, and "fat" under the "characteristics" item. Fat is a characteristic that indicates benignity, and the medical information processing device 16 determines that it is unnecessary to associate a key image with the findings text based on the "fat" in the structured result.

[0098] Thus, the medical information processing device 16 determines that it is unnecessary to associate a key image with the findings statement when the word describing the characteristics indicates benignity.

[0099] <Structure> This section will explain the details of structuring observational texts using natural language processing, with examples.

[0100] [Example 9] Figure 15 is a diagram illustrating the structuring of the findings statement. F15A shown in Figure 15 is the findings statement from the radiology report. As shown in F15A, the findings statement reads: "A solid mass measuring 59 mm in length is observed in S4 of the right middle lobe of the lung. The margin is lobulated and partially serrated. It does not contain calcification, cavities, or bronchial radiolucency. An early-stage, densely stained mass measuring 35 mm in length is observed in S1 of the liver. Hepatocellular carcinoma is suspected. Fatty liver is observed."

[0101] The medical information processing device 16 performs morphological analysis on the findings sentence shown in F15A and converts the findings sentence into a word sequence in which each word is separated. Furthermore, the medical information processing device 16 performs syntactic analysis on the word sequence to identify the relationships between each word.

[0102] Figure 15 shows F15B, which is a diagram of the word sequence converted from the findings sentence shown in F15A, and the relationships extracted from that word sequence. In F15B, the parts separated by " / " represent words, and the lines connecting each word indicate the identified relationships. Here, the findings sentence has been converted into the word sequence "A solid mass measuring 59 mm in length is observed in the right lung / middle lobe / S4. The margins are lobulated and partially serrated. There is no calcification, cavity, or bronchial radiolucency inside. An early-stage, intensely stained mass measuring 35 mm in length is observed in the liver / S1. Hepatocellular carcinoma is suspected. Fatty liver is observed." and the relationships between the words have been identified.

[0103] For example, the relationship between "right lung" and "mum" has been identified. The relationship between "mum" and "right lung," "solid type," and "is observed" within the same sentence has been identified, and furthermore, its relationship with "periphery" in the next sentence has also been identified. In this way, relationships are not limited to words within the same sentence, but can also be identified between words in different sentences.

[0104] Figure 15 shows F15C, which is the structured result of the findings statement using the analysis results shown in F15B. Here, the following items are classified based on their relationships: "Lung" under the "Organ" category, "Right lung," "Middle lobe," and "S4" under the "Location" category, "Mass" under the "Lesion" category, "Presence of lesion" under the "Status" category, "59 mm" under the "Size" category, and "Solid type+", "Lobulated type+", "Serrated (partial)+", "Calcification-", "Cavity-", and "Bronchial radiolucency-" under the "Characteristics" category.

[0105] Furthermore, the following items are classified based on their relationship: "Liver" under the "Organ" category, "S1" under the "Location" category, "Tumor" under the "Lesion" category, "Presence of Lesion" under the "Presence of Lesion" category, "Hepatocellular Carcinoma" under the "Disease Name" category, "Suspected" under the "Disease Name Status" category, "35mm in longest diameter" under the "Size" category, and "Early staining +" under the "Characteristics" category. In addition, the following items are classified based on their relationship: "Liver" under the "Organ" category, "Fatty Liver" under the "Disease Name" category, and "Presence of Disease Name Status" category.

[0106] [Example 10] Figure 16 is a diagram illustrating the structuring of the findings statement. F16A shown in Figure 16 is the findings statement from the radiology report. As shown in F16A, the findings statement reads, "A simple, low-attenuation mass is observed in liver segment S1. It shows faint enhancement in the arterial phase and washout in the portal venous phase. Hepatocellular carcinoma is suspected."

[0107] Figure 16 shows F16B, which is the result of natural language processing analysis of the sentence shown in F16A. Here, it has been transformed into the word sequence: "A simple, low-attenuation mass is found in liver S1. It stains faintly in the arterial phase and shows washout in the portal venous phase. Hepatocellular carcinoma is suspected." The relationships between the words have also been identified.

[0108] For example, the relationship between "simple" and "low attenuation," "arterial phase" and "high staining," and "portal phase" and "washout" have been identified.

[0109] Figure 16 shows F16C, which is the structured result of the findings using the analysis results shown in F16B. Here, the following items are classified based on their relationships: "Liver" under the "Organ" category, "S1" under the "Location" category, "Tumor" under the "Lesion" category, "Presence of Lesion" under the "Disease Name" category, "Hepatocellular Carcinoma" under the "Disease Name" category, "Suspected" under the "Characteristics" category, and "Low Attenuation (Simple)", "Faintly Enhanced (Arterial Phase)", and "Washout (Portal Venous Phase)".

[0110] In this way, the medical information processing device 16 associates the characteristics with the imaging conditions.

[0111] [Example 11] Figure 17 is a diagram illustrating the structuring of the findings statement. F17A in Figure 17 is the findings statement from the radiology report. As shown in F17A, the findings statement reads, "A solid mass measuring 59 mm in its longest diameter was found in S4 of the right middle lobe of the lung. Its size has increased compared to the previous examination."

[0112] Figure 17 shows F17B, which is the result of natural language processing analysis of the sentence shown in F17A. Here, it has been transformed into the word sequence, "A solid mass with a longest diameter of 59 mm was found in the right lung / middle lobe / S4. The size has increased compared to the previous examination." and the relationships between the words have also been identified.

[0113] For example, the relationship between "increase" and "previous examination" and "size" within the same sentence has been identified, and furthermore, its relationship with "tumor" in the preceding sentence has also been identified. Figure 17 shows F17C, which is the structured result of the findings statement using the analysis results shown in F17B. Here, the following items are classified based on their relationships: "Lung" under the "Organ" category, "Right lung," "Middle lobe," and "S4" under the "Location" category, "Mass" under the "Lesion" category, "Presence of lesion" under the "Presence of lesion" category, "Longest diameter 59 mm" under the "Size" category, "Solid type" under the "Characteristics" category, and "Size increase" under the "Comparison" category.

[0114] In this way, the medical information processing device 16 extracts comparative information.

[0115] <Other> In the technologies described in Patent Documents 1 and 2, there is no option to not associate a key image, but according to this embodiment, it is possible to determine whether or not to associate an image-based key image with a text.

[0116] Processing related to key images may be determined using supplementary information of medical images. For example, supplementary information of medical images may be obtained, lesion information may be obtained by analyzing the image interpretation report, and at least one of the following may be determined based on the supplementary information and the lesion information: whether or not to associate the key image with the findings text, and candidate key images to associate with the findings text. Supplementary information includes slice interval, contrast information (non-contrast / contrast, arterial phase / portal phase / equilibrium phase), etc. Slice interval is the distance between adjacent slice images in a direction perpendicular to the slice direction.

[0117] The processing of key images according to this embodiment is applicable to images other than medical images. For example, it can receive images of social infrastructure facilities such as transportation, electricity, gas, and water, along with their corresponding text, identify the relationship between two or more words in the text, and, based on the relationship between the two or more words, determine at least one of the following: whether or not to associate an image-based key image with the text, and select candidate key images to associate with the text.

[0118] The technical scope of the present invention is not limited to the scope described in the embodiments above. The configurations and other elements in each embodiment can be appropriately combined with those in each embodiment without departing from the spirit of the present invention. [Explanation of Symbols]

[0119] 10…Medical Information Processing System 12…Medical imaging equipment 14…Medical Image Database 16…Medical information processing device 16A…Processor 16B…Memory 16C…Communication Interface 18…Image Interpretation Report Database 20…User terminal 20A…Input device 20B…Display 22…Network A1... Annotation A2... Annotation ID1~ID5...Diagnostic images IK1...Key image IK2...Key image RP1…Image Interpretation Report RP2…Image Interpretation Report ST1~ST5…Steps in the medical information processing method

Claims

1. At least one processor, At least one memory for storing instructions to be executed by the aforementioned at least one processor, Equipped with, The aforementioned at least one processor is We accept text containing the diagnostic results of the images. Identify the relationship between two or more words contained in the aforementioned text, Based on the relationship between the two or more words, it is determined whether or not to associate the key image based on the image with the text. The two or more words mentioned above include words representing lesions and words representing information about changes. The word representing the change information includes a word representing at least one of the change information of size and quantity. The aforementioned at least one processor is If the size of the lesion increases, the size of the lesion decreases, the amount of the lesion increases, or the amount of the lesion decreases, it is determined that association with the key image is necessary. If there is no change in the size and amount of the lesion, it is determined that the association of the key image is unnecessary. Information processing device.

2. The two or more words mentioned above include at least two words from among words representing areas of interest, words representing facts, words representing locations, words representing sizes, words representing characteristics, and words representing shooting conditions. The information processing apparatus according to claim 1.

3. The two or more words mentioned above include a word representing a domain of interest and a word representing the factual nature of the domain of interest. The aforementioned at least one processor is If the aforementioned facts affirm the existence of the aforementioned area of ​​interest, then it is determined that the association of the key image is necessary. If the aforementioned facts negate the existence of the aforementioned region of interest, it is determined that the association of the key image is unnecessary. The information processing apparatus according to claim 1.

4. The aforementioned at least one processor is Based on the aforementioned position, candidates for the key image are extracted from the image. The information processing apparatus according to claim 2.

5. The aforementioned at least one processor is The system accepts two or more images with different shooting conditions. Candidates for the key image are extracted from the two or more images mentioned above. The information processing apparatus according to claim 2.

6. The aforementioned image is a medical image. The two or more words mentioned above include words that represent disease names. The aforementioned at least one processor is Candidate key images are extracted based on the aforementioned disease name. The information processing apparatus according to any one of claims 1 to 5.

7. The two or more words mentioned above include a word representing the area of ​​interest and a word representing the degree of malignancy of the area of ​​interest. The aforementioned at least one processor is If the aforementioned malignancy level confirms malignancy in the area of ​​interest, it is determined that association with the key image is necessary. If the malignancy level negates the malignancy of the region of interest, it is determined that the association of the key image is unnecessary. The information processing apparatus according to any one of claims 1 to 6.

8. The aforementioned at least one processor is The candidate key images are displayed on the screen. Accepts user input, In accordance with the above operation, the candidate key image is associated with the text as the key image. The information processing apparatus according to any one of claims 1 to 7.

9. The aforementioned image is a three-dimensional image. The aforementioned at least one processor is As a candidate for the key image, a slice image of any slice position of the three-dimensional image is displayed on the display. The user can change the slice position of the candidate key image. The slice image of the modified slice position is associated with the text as a key image. The information processing apparatus according to any one of claims 1 to 8.

10. Computers A reception process that accepts text containing the diagnostic results of the images, A process of identifying the relationship between two or more words contained in the aforementioned text, A decision step of determining whether or not to associate the key image based on the image with the text, based on the relationship between the two or more words, Execute, The two or more words mentioned above include words representing lesions and words representing information about changes. The word representing the change information includes a word representing at least one of the change information of size and quantity. The aforementioned decision-making process is, If the size of the lesion increases, the size of the lesion decreases, the amount of the lesion increases, or the amount of the lesion decreases, it is determined that association with the key image is necessary. If there is no change in the size and amount of the lesion, it is determined that the association of the key image is unnecessary. Information processing methods.

11. A program for causing a computer to execute the information processing method described in claim 10.

Images