Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program capable of reducing errors on the associations between images of documents containing sentences and the images and the sentences and / or comparing the documents with an original document and creating an easy-to-read document.SOLUTION: An information processing device comprises at least one processor, and at least one memory that stores instructions executed by the at least one processor. The at least one processor acquires information on an object shown in one or more accepted images, acquires information described in one or more accepted sentences, determines whether there is a correspondence between images and sentences on the basis of information on the object and the described information, and performs processing for assisting the creation of a document containing images and sentences on the basis of presence or absence of the correspondence.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program, and more particularly to a technique for assisting in the creation of a document including text and images. [Background technology]

[0002] When preparing an image interpretation report, a doctor creates key images for important findings. Key images are created so that other doctors can quickly check the condition and location of lesions when reading the image interpretation report, and as a reference when interpreting images of the same patient in the future.

[0003] When creating key images, doctors need to check whether key images of important findings have been created, rearrange the order of the image reading report so that it matches the order of the key images, and insert image numbers into the image reading report.

[0004] However, these tasks are burdensome and can result in problems such as the order of the radiology report not matching the order of the key images, unnecessary key images being present, and key images being omitted.

[0005] Patent Document 1 discloses a technology for acquiring the area of ​​interest during image interpretation and the contents of the image interpretation report, and judging the consistency. The technology of Patent Document 1 does not judge the presence or absence of a key image, and there is a possibility of a defect occurring in which a key image is not attached to an important finding in the image interpretation report. In addition, the technology of Patent Document 1 does not associate a key image with a statement of a finding, and cannot assign a figure number, etc.

[0006] Furthermore, Patent Document 2 discloses a technique for rearranging images in the order of findings. However, the technique of Patent Document 2 is a technique in which a user associates images with findings, and does not reduce the burden on doctors. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2016-057695 A [Patent Document 2] Patent No. 6923863 Summary of the Invention [Problem to be solved by the invention]

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide an information processing device, an information processing method, and a program that reduce errors in matching images and text in documents that contain text and images, and / or create documents that are easier to read by comparing them with the original document. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, an information processing device according to a first aspect of the present disclosure is an information processing device that includes at least one processor and at least one memory that stores instructions to be executed by the at least one processor, wherein the at least one processor acquires information about an object appearing in one or more received images, acquires information described in one or more received sentences, determines whether or not there is a correspondence between the image and the sentences based on the information about the object and the described information, and executes a process to assist in creating a document including the image and the sentences based on the presence or absence of the correspondence.

[0010] According to the first aspect, it is possible to reduce errors in associating images and text in a document that includes text and images, and / or to create a document that is easier to read by comparing it with the original document.

[0011] The information processing device according to the second aspect of the present disclosure may be the information processing device according to the first aspect, in which the process is a process of issuing a warning when an image corresponding to a text does not exist. This makes it possible to prevent an image attachment omission caused by an image not existing corresponding to a text.

[0012] The information processing device according to the third aspect of the present disclosure may be the information processing device according to the first or second aspect, and the process may be a process of issuing a warning when an image corresponding to a sentence does not exist and the importance of the sentence, which indicates the degree to which an image corresponding to the sentence is necessary, is equal to or greater than a threshold. This makes it possible to prevent an image attachment omission due to an absence of an image corresponding to an important sentence.

[0013] An information processing device according to a fourth aspect of the present disclosure is the information processing device according to any one of the first to third aspects, and the process may be a process of issuing a warning when there is no text corresponding to the image. This makes it possible to eliminate text omissions that result in no text corresponding to the image.

[0014] An information processing device according to a fifth aspect of the present disclosure may be the information processing device according to any one of the first to fourth aspects, in which the process is a process of rearranging the order of one of the sentences and the images that are determined to correspond based on the order of the other, thereby making it possible to match the order of the sentences and the order of the images to create an easy-to-read document.

[0015] An information processing device according to a sixth aspect of the present disclosure is the information processing device according to any one of the first to fifth aspects, and the process may be a process of rearranging the order of images based on the order of text, thereby making it possible to match the order of text with the order of images.

[0016] An information processing device according to a seventh aspect of the present disclosure is the information processing device according to any one of the first to sixth aspects, and the process may be a process of assigning figure numbers to images among the sentences and images that are determined to have a correspondence. This allows the image to be assigned a figure number, thereby enriching the description of the document.

[0017] An information processing device according to an eighth aspect of the present disclosure is the information processing device according to the seventh aspect, and the process of assigning figure numbers may be a process of assigning figure numbers to objects in the image in the order in which they appear in the corresponding sentence. This makes it possible to assign figure numbers in the order in which they appear in the sentence.

[0018] An information processing device according to a ninth aspect of the present disclosure is the information processing device according to the eighth aspect, and the process may be a process of assigning a figure number assigned to the image to the sentence among the sentence and the image that are determined to have a correspondence. This allows the sentence to be assigned a figure number, thereby enriching the description of the document.

[0019] An information processing device according to a tenth aspect of the present disclosure is the information processing device according to the ninth aspect, and the process may be a process of adding figure numbers to corresponding parts of a sentence. This makes it possible to make the document easier to read.

[0020] In an information processing device according to an eleventh aspect of the present disclosure, in the information processing device according to any one of the first to tenth aspects, at least one processor may execute a process each time it receives either an image input or a text input. This allows the process to be executed at any time.

[0021] The information processing device according to the twelfth aspect of the present disclosure may be the information processing device according to any one of the first to eleventh aspects, and may include a first mode in which processing is executed every time an image input or a text input is received, and a second mode in which processing is executed after all images and all text inputs are received. This allows processing to be executed in either mode.

[0022] In the information processing device according to a thirteenth aspect of the present disclosure, in the information processing device according to any one of the first to twelfth aspects, at least one processor may obtain a degree of certainty indicating the likelihood that the information about the object and the written information correspond to each other, and determine whether or not the image and the text correspond to each other based on the degree of certainty. This makes it possible to appropriately determine whether or not the image and the text correspond to each other.

[0023] In the information processing device according to a fourteenth aspect of the present disclosure, in the information processing device according to any one of the first to thirteenth aspects, at least one processor may analyze the received image or an original image from which the received image was created to acquire information about the object. This makes it possible to appropriately acquire information about the object.

[0024] An information processing device according to a fifteenth aspect of the present disclosure is the information processing device according to any one of the first to fourteenth aspects, in which the image may be a key image based on a medical image.

[0025] An information processing device according to a sixteenth aspect of the present disclosure is an information processing device according to the fifteenth aspect, wherein the object includes at least one of an organ and a tumor, and the information about the object includes at least one of size, characteristics, disease name, location, and feature amount.

[0026] In order to achieve the above-mentioned object, an information processing method according to a seventeenth aspect of the present disclosure is an information processing method in which at least one processor acquires information about an object appearing in one or more received images, acquires information described in one or more received sentences, determines whether or not there is a correspondence between the image and the sentences based on the information about the object and the described information, and executes a process to assist in creating a document including the image and the sentences based on the presence or absence of the correspondence.

[0027] According to the seventeenth aspect, it is possible to reduce errors in matching images and text in a document that includes text and images, and / or to create a document that is easier to read by comparing it with the original document. In the seventeenth aspect, the same matters as those specified in the second to sixteenth aspects can be appropriately combined.

[0028] In order to achieve the above object, a program according to an 18th aspect of the present disclosure is a program for causing a computer to execute the information processing method according to the 17th aspect. A non-transitory computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) storing the program according to the 18th aspect is also included in the present disclosure.

[0029] According to the eighteenth aspect, it is possible to reduce errors in associating images and text in a document that includes text and images, and / or to create a document that is easier to read by comparing it with the original document. Effect of the Invention

[0030] According to the present invention, it is possible to reduce errors in matching images and text in a document that includes text and images, and / or to create a document that is easier to read by comparing it with the original document. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a medical information processing system. [Diagram 2] FIG. 2 is a block diagram showing the electrical configuration of the medical information processing device. [Diagram 3] FIG. 3 is a block diagram showing the functional configuration of the medical information processing device. [Figure 4] FIG. 4 is a diagram showing an image interpretation report. [Diagram 5] FIG. 5 is a flowchart showing the medical information processing method. [Figure 6] FIG. 6 is a diagram showing an image interpretation report. [Figure 7] FIG. 7 is a diagram showing an image interpretation report. [Figure 8] FIG. 8 is a diagram for explaining the correspondence between the finding sentence and the key image using the confidence factor. [Figure 9] FIG. 9 is a diagram for explaining a case where two or more key images correspond to one finding sentence. [Figure 10] FIG. 10 is a diagram for explaining a case where one key image is associated with a plurality of finding sentences. [Figure 11] FIG. 11 is an explanatory diagram showing an example of learning data used in the method for generating a language feature extraction model. [Figure 12]FIG. 12 is a block diagram showing a schematic functional configuration of a machine learning device that trains a language feature extraction model. [Figure 13] FIG. 13 is a block diagram illustrating an example of a hardware configuration of the machine learning device. [Figure 14] FIG. 14 is a flowchart showing an example of a machine learning method executed by the machine learning device. [Figure 15] FIG. 15 is a block diagram showing a schematic functional configuration of a machine learning device that trains an image feature extraction model and a cross-modal feature integration model. [Figure 16] FIG. 16 is a flowchart illustrating an example of a machine learning method executed by the machine learning device. [Figure 17] FIG. 17 is a block diagram showing a functional configuration of the object information acquisition unit. [Figure 18] FIG. 18 is a flowchart showing a medical image analysis method by the object information acquisition unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Here, a medical information processing system will be described as an example of an information processing apparatus, an information processing method, and a program according to the present invention.

[0033] <Medical information processing system> The medical information processing system according to this embodiment is a system that executes auxiliary processing to acquire information about an object appearing in a key image based on a medical image, acquire information described in a finding text, determine whether or not there is a correspondence between the key image and the finding text based on the information about the object and the described information, and assist in the creation of an image interpretation report including the finding text and the key image based on the presence or absence of the correspondence.The medical information processing system can reduce errors in matching medical images and findings in an image interpretation report (an example of a "document") and / or create an image interpretation report that is easier to read compared to the original image interpretation report.

[0034] Fig. 1 is an overall configuration diagram of a medical information processing system 10. As shown in Fig. 1, the medical information processing system 10 includes a medical image inspection device 12, a medical image database 14, a user terminal device 16, an image interpretation report database 18, and a medical information processing device 20.

[0035] The medical image inspection equipment 12, the medical image database 14, the user terminal device 16, the image interpretation report database 18, and the medical information processing device 20 are connected to each other via a network 22 so as to be able to transmit and receive data. The network 22 includes a wired or wireless LAN (Local Area Network) that communicatively connects various devices in the medical institution. The network 22 may also include a WAN (Wide Area Network) that connects the LANs of multiple medical institutions.

[0036] The medical imaging inspection equipment 12 is an imaging device that captures an image of an inspection target part of a subject and generates a medical image. Examples of the medical imaging inspection equipment 12 include an X-ray imaging device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, an ultrasound device, a CR (Computed Radiography) device using a flat X-ray detector, and an endoscope device.

[0037] The medical image database 14 is a database that manages medical images captured by the medical image inspection equipment 12. The medical image database 14 is implemented as a computer equipped with a large-capacity storage device for storing medical images. Software that provides the functions of a database management system is installed in the computer.

[0038] The medical image may be a plurality of tomographic images captured by a CT device, an MRI device, etc., or a three-dimensional reconstructed image reconstructed using a plurality of tomographic images. It may also be a cross-sectional image in any direction of the three-dimensional reconstructed image.

[0039] The format of the medical image can be in accordance with the Dicom (Digital Imaging and Communications in Medicine) standard. Supplementary information (Dicom tag information) defined in the Dicom standard may be added to the medical image. Note that the term "image" in this specification includes not only the image itself, such as a photograph, but also image data, which is a signal representing an image.

[0040] The user terminal device 16 is a terminal device for a doctor, who is a user, to create and view an interpretation report, and includes viewer software for the doctor to view medical images. The user terminal device 16 is, for example, a personal computer. The user terminal device 16 may be a workstation or a tablet terminal. The user terminal device 16 includes an input device 16A and a display device 16B. The input device 16A may include a mouse and a keyboard.

[0041] The doctor uses the input device 16A to input an instruction to display a medical image. The user terminal device 16 displays the medical image on the display 16B in accordance with the instruction. The doctor uses the input device 16A to create a key image from the displayed medical image. Furthermore, the doctor uses the input device 16A to input a finding statement, which is a sentence indicating the result of the interpretation of the medical image. In this way, the doctor uses the user terminal device 16 to create an interpretation report including the key image and the finding statement.

[0042] The key image is an image that is determined to be important in image interpretation based on the contents of the findings, among medical images captured during an examination of a subject that is the subject of an image interpretation report.

[0043] The image interpretation report database 18 is a database that manages image interpretation reports created by doctors. The image interpretation report database 18 is implemented by a computer equipped with a large-capacity storage device for storing image interpretation reports. Software that provides the functions of a database management system is installed in the computer. The medical image database 14 and the image interpretation report database 18 may be configured in a single computer.

[0044] The medical information processing device 20 is a device that executes auxiliary processing, which is processing that assists in creating an interpretation report. A personal computer or a workstation (an example of a "computer") can be applied to the medical information processing device 20. FIG. 2 is a block diagram showing the electrical configuration of the medical information processing device 20. As shown in FIG. 2, the medical information processing device 20 includes a processor 20A, a memory 20B, and a communication interface 20C.

[0045] The processor 20A executes instructions stored in the memory 20B. The hardware structure of the processor 20A is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and acts as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0046] A processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functional units may be configured with one processor. As an example of configuring multiple functional units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor acts as multiple functional units. Second, as represented by a SoC (System On Chip), there is a form in which a processor is used that realizes the functions of the entire system including multiple functional units with one IC (Integrated Circuit) chip. In this way, the various functional units are configured using one or more of the above various processors as a hardware structure.

[0047] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0048] The memory 20B stores instructions to be executed by the processor 20A. The memory 20B includes a RAM (Random Access Memory) and a ROM (Read Only Memory), not shown. The processor 20A uses the RAM as a working area, executes software using various programs and parameters, including a medical information processing program described later, stored in the ROM, and executes various processes of the medical information processing device 20 by using the parameters stored in the ROM, etc.

[0049] The communication interface 20C controls communications with the medical image inspection equipment 12, the medical image database 14, the user terminal device 16, and the image interpretation report database 18 via the network 22 in accordance with a predetermined protocol.

[0050] The medical information processing device 20 may be a cloud server accessible from a plurality of medical institutions via the Internet. The processing performed by the medical information processing device 20 may be a cloud service with a fee-based or flat-rate fee system.

[0051] [Functional configuration of medical information processing device] Fig. 3 is a block diagram showing the functional configuration of the medical information processing device 20. Each function of the medical information processing device 20 is realized by the processor 20A executing a medical information processing program stored in the memory 20B. As shown in Fig. 3, the medical information processing device 20 includes an image acquisition unit 32, an object information acquisition unit 34, a text acquisition unit 36, a written information acquisition unit 38, a correspondence determination unit 40, and an auxiliary processing execution unit 42.

[0052] The image acquiring unit 32 acquires one or more key images to be used in the image interpretation report. For example, the image acquiring unit 32 acquires a medical image from the medical image database 14 and displays it on the display 16B. The doctor creates a key image by trimming an area determined to be important for image interpretation from the medical image displayed on the display 16B using the input device 16A. The image acquiring unit 32 accepts the key image created by the doctor. The image acquiring unit 32 may acquire a key image that has already been written in the image interpretation report.

[0053] The object information acquisition unit 34 acquires object information on objects appearing in one or more key images acquired by the image acquisition unit 32. The object information acquisition unit 34 may acquire object information by analyzing the key image, or may acquire object information by analyzing a medical image that is an original image from which the key image was created. The object includes at least one of an organ and a tumor. The object information includes at least one of size, properties, disease name, position, and feature amount. The object information acquisition unit 34 may acquire image features indicating feature amounts of the key image as the object information. The image features may be expressed by an image feature vector obtained by feature vectorizing the key image, or may be a feature map of multiple channels.

[0054] The text acquisition unit 36 ​​acquires one or more finding sentences to be used in the image interpretation report. For example, a doctor inputs a finding sentence related to a key image using the input device 16A. The text acquisition unit 36 ​​accepts the finding sentences input by the doctor. The text acquisition unit 36 ​​may accept structured data obtained by structuring the finding sentences input by the doctor through structural analysis.

[0055] The described information acquiring unit 38 acquires described information that is described in one or more finding sentences acquired by the text acquiring unit 36. The described information acquiring unit 38 may acquire, as the described information, a language feature vector in which a feature amount corresponding to the finding sentence is extracted.

[0056] The correspondence determination unit 40 determines whether or not there is a correspondence between each combination of the key image acquired by the image acquisition unit 32 and the finding sentence acquired by the text acquisition unit 36, based on the object information acquired by the object information acquisition unit 34 and the description information acquired by the description information acquisition unit 38. The correspondence determination unit 40 may obtain a confidence level indicating the likelihood that the object information and the description information correspond to each other, and may determine whether or not there is a correspondence between the key image and the finding sentence based on the obtained confidence level. For example, the correspondence determination unit 40 may determine that there is a correspondence between the key image and the finding sentence when the confidence level between the object information of the key image and the description information of the finding sentence is equal to or higher than a threshold, and may determine that there is no correspondence between the key image and the finding sentence when the confidence level is less than the threshold.

[0057] The auxiliary process execution unit 42 executes auxiliary process based on the presence or absence of correspondence determined by the correspondence determination unit 40. The auxiliary process may be a process of issuing a warning when a key image corresponding to the finding sentence does not exist. The auxiliary process may be a process of issuing a warning when a key image corresponding to the finding sentence does not exist and the finding sentence is an important finding. When a key image corresponding to the finding sentence does not exist and the finding sentence is not an important finding, the auxiliary process does not need to be executed.

[0058] An important finding is a finding for which a key image needs to be created. A finding for which a key image needs to be created is, for example, a finding describing a specific disease. The auxiliary process execution unit 42 may calculate the importance of the finding sentence, which indicates the degree to which a key image corresponding to the finding sentence is necessary, and may determine that the finding sentence is an important finding when the importance of the finding sentence is equal to or greater than a threshold value.

[0059] The auxiliary process may be a process of issuing a warning when a finding sentence corresponding to a key image does not exist. The auxiliary process may be a process of rearranging the order of one of a finding sentence and a key image that are determined to have a correspondence based on the order of the other. The auxiliary process may be a process of rearranging the order of the key images based on the order of the finding sentences.

[0060] The auxiliary process may be a process of assigning figure numbers to key images among the findings and key images that are determined to have a correspondence. The process of assigning figure numbers may be a process of assigning figure numbers to objects in the key images in the order that they appear in the corresponding findings.

[0061] The auxiliary process may be a process of assigning the figure number assigned to the key image to the finding sentence out of the finding sentence and the key image that are determined to have a correspondence. The auxiliary process may be a process of assigning the figure number to the corresponding part of the finding sentence.

[0062] The auxiliary process execution unit 42 may execute the auxiliary process each time it receives any one of a key image in the image acquisition unit 32 and a finding sentence in the sentence acquisition unit 36. The auxiliary process execution unit 42 may execute the auxiliary process after it receives all the key images in the image acquisition unit 32 and all the finding sentences in the sentence acquisition unit 36. The medical information processing device 20 may have a first mode in which it executes the auxiliary process each time it receives any one of a key image and a finding sentence, and a second mode in which it executes the auxiliary process after it receives all the key images and all the finding sentences. The first mode and the second mode may be switchable by a doctor's operation of the input device 16A.

[0063] <Medical information processing method> 4 is a diagram showing the image interpretation report RP1 displayed on the display 16B of the user terminal device 16. The image interpretation report RP1 is created by a doctor.

[0064] The display area D1 of the image interpretation report RP1 is an area where the findings are displayed. In the display area D1, from the top, the findings SN1 "No significant lymph node enlargement is observed," SN2 "Calcification is observed in coronary a," SN3 "A small amount of right pleural effusion is observed," SN4 "An 11 mm-sized low attenuation is observed in the left lobe of the thyroid," and SN5 "Changes following fracture are observed in the distal right clavicle and right third rib." are displayed.

[0065] Display area D2 of image interpretation report RP1 is an area in which key images are displayed. In display area D2, key image IK1 including lesion area A1 and key image IK2 including lesion area A2 are displayed in this order from the left.

[0066] FIG. 5 is a flowchart showing a medical information processing method using the medical information processing device 20. The medical information processing method is a method of executing auxiliary processing based on the presence or absence of correspondence between a key image and a finding sentence, based on object information shown in a key image and description information described in the finding sentence. The medical information processing method is realized by the processor 20A executing a medical information processing program stored in the memory 20B. The medical information processing program may be provided to the medical information processing device 20 by a computer-readable non-transitory storage medium, or may be provided to the medical information processing device 20 via the Internet. Here, an example of executing auxiliary processing on the image interpretation report RP1 shown in FIG. 4 will be described.

[0067] In step S1, the image acquisition unit 32 acquires one or more key images to be used in the image interpretation report. Here, the image acquisition unit 32 acquires a key image IK1 and a key image IK2.

[0068] In step S2, the object information acquisition unit 34 acquires object information regarding the objects captured in the key images IK1 and IK2 acquired in step S1. Here, the object information acquisition unit 34 acquires, as the object information, information on the lesion area A1 from the key image IK1 and information on the lesion area A2 from the key image IK2.

[0069] In step S3, the text acquisition unit 36 ​​acquires one or more finding texts to be used in the image interpretation report. Here, the text acquisition unit 36 ​​acquires finding texts SN1 to SN5.

[0070] In step S4, the described information acquisition unit 38 acquires the described information described in each of the finding sentences SN1 to SN5 acquired in step S3. Here, the described information acquisition unit 38 acquires, as the described information, "not found" from the finding sentence SN1, "coronary a." and "calcification" from the finding sentence SN2, "small amount" and "right pleural effusion" from the finding sentence SN3, "left lobe of thyroid" and "low absorption of 11 mm" from the finding sentence SN4, and "distal right clavicle", "right third rib", and "changes after fracture" from the finding sentence SN5.

[0071] In step S5, the correspondence determination unit 40 determines whether or not there is a correspondence between the key image IK1 and the key image IK2 and the findings SN1 to SN5 in a brute force manner based on the object information acquired in step S2 and the description information acquired in step S4. That is, the correspondence determination unit 40 determines whether or not there is a correspondence between each combination of the key image IK1 and the findings SN1, the key image IK1 and the findings SN2, ..., the key image IK1 and the findings SN5, the key image IK2 and the findings SN1, the key image IK2 and the findings SN2, ..., the key image IK2 and the findings SN5. Here, it is assumed that the correspondence determination unit 40 has determined that there is a correspondence between the combination of the key image IK1 and the findings SN5 and the combination of the key image IK2 and the findings SN4.

[0072] In step S6, the auxiliary process execution unit 42 executes auxiliary process based on the presence or absence of correspondence determined in step S5. Here, the auxiliary process is assumed to be a process of rearranging the order of key images based on the order of the finding sentences in a plurality of combinations of finding sentences and key images determined to have a correspondence.

[0073] 6 is a diagram showing an image interpretation report RP2 displayed on the display 16B of the user terminal device 16. The image interpretation report RP2 is obtained by executing auxiliary processing on the image interpretation report RP1. In the image interpretation report RP2, the display of the findings SN1 to SN5 in the display area D1 is the same as that of the image interpretation report RP1. In the image interpretation report RP2, the position of the key image in the display area D2 is changed from that of the image interpretation report RP1, and key image IK2 and key image IK1 are displayed in this order from the left.

[0074] In other words, in the combination of key image IK1 and finding sentence SN5 that are determined to have a correspondence, and in the combination of key image IK2 and finding sentence SN4, the auxiliary processing execution unit 42 places the key image IK2 corresponding to finding sentence SN4, which comes first from the top of the display area D1, on the left side, and the key image IK1 corresponding to finding sentence SN5, which comes last from the top, on the right side.

[0075] It is believed that the findings in a radiology report are read in order from top to bottom, and the key images are viewed in order from left to right. Therefore, by rearranging the left and right order of the key images based on the top-down order of the findings, it is possible to create an easy-to-read radiology report.

[0076] The auxiliary process is not limited to the process of rearranging the order of key images. Fig. 7 is a diagram showing an image interpretation report RP3 displayed on the display 16B of the user terminal device 16. The image interpretation report RP3 is a report after the auxiliary process has been executed. In this example, in addition to the same findings SN1 to SN5 and key images IK1 to IK2 as in Fig. 5, a finding SN6 "A marginally lobulated solid mass measuring approximately 68 mm in diameter was observed in the right lung S10. Lung cancer suspected" and a key image IK3 including a lesion area A3 are shown.

[0077] In the image interpretation report RP3, the fact that there is no key image corresponding to the finding SN6 (an example of "non-existence") is warned by a marker MK1 that highlights the finding SN6. The finding SN6 may be warned only when there is no corresponding key image and the finding SN6 is an important finding. Here, the information described in the finding SN6, "solid mass," is a description of the disease, so the finding SN6 is determined to be an important finding.

[0078] In addition, in the display area D2 of the image interpretation report RP3, the key image IK2 and the key image IK1 are displayed in order from the left based on the order of the corresponding finding sentences SN4 and SN5, similar to the image interpretation report RP2. Furthermore, the key images IK2 and IK1 are assigned figure numbers in order from the left. Here, the key image IK2 is assigned "Figure 1" as the figure number NF1, and the key image IK1 is assigned "Figure 2" as the figure number NF2. In this way, the figure numbers NF1 and NF2 are assigned in the order in which the lesion area A2 shown in the key image IK2 and the lesion area A1 shown in the key image IK1 appear in the corresponding finding sentences SN4 and SN5, respectively.

[0079] The finding sentences SN4 and SN5 are assigned the figure numbers NF1 and NF2 of the corresponding key images, respectively. The figure numbers NF1 and NF2 assigned to the finding sentences SN4 and SN5 may be assigned to the corresponding parts of the finding sentences SN4 and SN5.

[0080] In the image interpretation report RP3, the fact that there is no finding statement corresponding to the key image IK3 is warned by a frame FL1 that highlights the key image IK3.

[0081] <Details of the correspondence between key images and findings> 8 is a diagram for explaining the correspondence between the finding sentence and the key image using the confidence level. In the example shown in FIG. 8, the image acquisition unit 32 acquires the key image IK11, the key image IK12, and the key image IK13, and the text acquisition unit 36 ​​acquires the finding sentence SN11 "Changes after fracture are observed in the right distal clavicle and the right third rib." and the finding sentence SN12 "Low absorption of 11 mm in size is observed in the left lobe of the thyroid gland." In this case, the correspondence determination unit 40 calculates the confidence level for each combination of the finding sentence SN11 and the key image IK11, the finding sentence SN11 and the key image IK12, the finding sentence SN11 and the key image IK13, the finding sentence SN12 and the key image IK11, the finding sentence SN12 and the key image IK12, and the finding sentence SN12 and the key image IK13 based on the object information and the description information, and determines whether or not there is a correspondence.

[0082] The confidence level is an index indicating that the greater the value, the higher the likelihood that the object information and the description information correspond to each other, and is, for example, a value between 0 and 1. The threshold value is, for example, 0.90. That is, when the confidence level is 0.90 or more, the correspondence determination unit 40 determines that there is a correspondence between the key image and the finding text, and corresponds the key image to the finding text.

[0083] FIG. 9 is a diagram for explaining a case where two or more key images are associated with one finding sentence. In the example shown in FIG. 9, the confidence level of the finding sentence SN21 "A nodular early staining area with a diameter of 1 cm is observed in S3. Washout is observed in the equilibrium phase, and HCC recurrence is considered" and the key image IK21 is 0.95, and the confidence level of the finding sentence SN21 and the key image IK22 is 0.96. The key images IK21 and IK22 are images of the same anatomical position, and both have confidence levels higher than the threshold value of 0.90, so both the key images IK21 and IK22 are associated with the finding sentence SN21. In this way, two or more key images may correspond to one finding sentence.

[0084] In addition, the finding sentence SN21 contains sentence L1 "A nodular early staining area measuring 1 cm in diameter was observed in S3." and sentence L2 "Washout was observed in the equilibrium phase, suggesting HCC recurrence." However, sentences L1 and L2 may be treated as separate finding sentences.

[0085] FIG. 10 is a diagram for explaining a case where the confidence level between one key image and multiple finding sentences is equal to or higher than a threshold. In the example shown in FIG. 10, the confidence level between the finding sentence SN31 "Changes after fracture are observed in the right distal clavicle and the right rib" and the key image IK31 is 0.95, which is equal to or higher than the threshold value of 0.90. Also, the confidence level between the finding sentence SN32 "Calcification is observed in coronary a." and the key image IK31 is 0.90, which is equal to or higher than the threshold value of 0.90. In this case, the finding sentence is associated with a finding sentence having a relatively high confidence level. That is, the correspondence determination unit 40 associates the key image IK31 with the finding sentence SN31. In this manner, one finding sentence corresponds to one key image.

[0086] <Language feature extraction model> The description information acquiring unit 38 may include a language feature extraction model. The language feature extraction model is a learned model that is trained to receive an input of a finding sentence and output a corresponding finding feature.

[0087] [Examples of data used for machine learning] FIG. 11 is an explanatory diagram showing an example of learning (training) data used in the method for generating a language feature extraction model. Here, an example of training data TDj including a key image IMj, position information TPj regarding a region of interest ROIj in the key image IMj, and a finding sentence TXj described in the region of interest ROIj will be described. The key image IMj, the position information TPj regarding the region of interest ROIj, and the finding sentence TXj are associated (linked) with each other. The subscript j represents an index number as an identification code of the associated data set. The region of interest ROIj is mainly a lesion area.

[0088] The position information TPj regarding the region of interest ROIj is information that can specify the position of the region of interest ROIj in the key image IMj. The position information TPj may be coordinate information indicating coordinates in the key image IMj, information indicating an area or range in the key image IMj, or a combination of these. The position information TPj may be information added as annotation information for the key image IMj, or meta information attached to the key image IMj such as a DICOM tag.

[0089] The finding sentence TXj may be, for example, a sentence written in a radiology report. Here, as the finding sentence TXj, a text that is unstructured data in a free description type sentence format before structuring is exemplified, but structured data that is structured by a structural analysis of the sentence can also be used.

[0090] Such training data TDj can be generated by sampling appropriate data from a database in which data on medical images and interpretation reports relating to past examination cases in a medical institution such as a hospital are associated with each other and stored.

[0091] [Example of machine learning device configuration] 12 is a block diagram showing a schematic functional configuration of a machine learning device 100 that trains a language feature extraction model. The machine learning device 100 may be the same device as the medical information processing device 20, or may be a different device.

[0092] The machine learning device 100 includes a language feature extraction model 102, a domain estimation model 104, a loss calculation unit 106, and a parameter update unit 108. The functions of each unit of the machine learning device 100 can be realized by a combination of computer hardware and software.

[0093] For example, a natural language processing model called BERT (Bidirectional Encoder Representations from Transformers) is applied to the language feature extraction model 102. The language feature extraction model 102 receives an input of a finding sentence TXj, which is a text, extracts a feature amount corresponding to the input finding sentence TXj, and outputs a finding feature LFVj, which is a language feature vector (finding feature vector).

[0094] For example, a convolutional neural network (CNN) is applied to the region estimation model 104. The region estimation model 104 receives an input of a key image IMj and a finding feature LFVj, estimates a lesion region in the key image IMj mentioned in the input finding sentence TXj, and outputs estimated region information PAj indicating the position of the estimated lesion region. The estimated region information PAj may be, for example, coordinate information specifying the position of a rectangle (bounding box) surrounding the range of the estimated lesion region, or may be a segmentation mask image or the like specifying the estimated lesion region in pixel units.

[0095] The loss calculation unit 106 calculates a loss indicating the error between the estimated lesion area indicated in the estimated area information PAj output from the area estimation model 104 and the correct region of interest ROIj indicated by the correct position information TPj linked to the key image IMj.

[0096] Based on the loss calculated by the loss calculation unit 106, the parameter update unit 108 calculates the amount of update for the parameters of each model of the domain estimation model 104 and the language feature extraction model 102 so as to reduce the loss, and updates the parameters of each model according to the calculated amount of update. The parameters of each model include the filter coefficients (weights of connections between nodes) of filters used in processing each layer of the neural network, and the biases of the nodes. The parameter update unit 108 optimizes the parameters of each model by a method such as Stochastic Gradient Descent (SGD).

[0097] 13 is a block diagram showing an example of a hardware configuration of the machine learning device 100. The machine learning device 100 includes a processor 112, a computer-readable medium 114 which is a non-transient tangible entity, a communication interface 116, an input / output interface 118, and a bus 119. The processor 112 is connected to the computer-readable medium 114, the communication interface 116, and the input / output interface 118 via the bus 119.

[0098] The form of the machine learning device 100 is not particularly limited, and may be a server, a workstation, a personal computer, or the like.

[0099] The processor 112 includes a central processing unit (CPU). The processor 112 may include a graphics processing unit (GPU). The computer-readable medium 114 includes a memory 114A which is a main storage device and a storage 114B which is an auxiliary storage device. The computer-readable medium 114 may be, for example, a semiconductor memory, a hard disk drive (HDD) device, a solid state drive (SSD) device, or a combination of a plurality of these.

[0100] The machine learning device 100 may further include an input device 142 and a display device 144. The input device 142 is, for example, a keyboard, a mouse, a multi-touch panel, or other pointing device, or a voice input device, or an appropriate combination of these. The display device 144 is, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, or a projector, or an appropriate combination of these. The input device 142 and the display device 144 are connected to the processor 112 via the input / output interface 118.

[0101] The machine learning device 100 can be connected to an electric communication line (not shown) via the communication interface 116. The electric communication line may be a wide area communication line, a private network communication line, or a combination of these.

[0102] The machine learning device 100 is communicatively connected to an external device such as a training data storage unit 150 via the communication interface 116. The training data storage unit 150 includes a storage in which a training data set including a plurality of training data TDj is stored. The training data storage unit 150 may be built in a storage 114B in the machine learning device 100.

[0103] The computer-readable medium 114 stores a plurality of programs, data, and the like, including a learning processing program 120 and a display control program 130. The term "program" includes the concept of a program module. The processor 112 executes the instructions of the programs stored in the computer-readable medium 114 to function as various processing units.

[0104] The learning process program 120 includes instructions for acquiring training data TDj and executing learning processes for the language feature extraction model 102 and the domain estimation model 104. That is, the learning process program 120 includes a data acquisition program 122, the language feature extraction model 102, the domain estimation model 104, a loss calculation program 126, and an optimizer 128. The data acquisition program 122 includes instructions for executing a process for acquiring training data TDj from the training data storage unit 150.

[0105] The loss calculation program 126 includes an instruction to execute a process of calculating a loss indicating an error between estimated area information indicated by information indicating the position of the lesion area output from the area estimation model 104 and correct position information TPj corresponding to the finding sentence TXj input to the language feature extraction model 102. The optimizer 128 includes an instruction to execute a process of calculating an update amount of the parameters of each model of the area estimation model 104 and the language feature extraction model 102 from the calculated loss and updating the parameters of each model.

[0106] The display control program 130 generates a display signal required for display output to the display device 144 and includes instructions for executing display control of the display device 144 .

[0107] [Overview of machine learning methods] Fig. 14 is a flowchart showing an example of a machine learning method executed by the machine learning device 100. Before executing the flowchart of Fig. 14, a training dataset is prepared by preparing a plurality of sets of training data TDj, which is a set of data in which a training key image IMj, a finding sentence TXj that is a text explaining a certain region of interest ROIj in the key image IMj, and position information TPj related to the region of interest ROIj are linked together.

[0108] In step S100, the processor 112 acquires a data set including a key image IMj, position information TPj on a region of interest ROIj in the key image IMj, and a comment statement TXj describing the region of interest ROIj from a training data set.

[0109] In step S110, the processor 112 inputs the finding sentence TXj to the language feature extraction model 102, causes the language feature extraction model 102 to extract finding features LFVj indicating the feature amount of the finding sentence TXj, and obtains an output of the finding features LFVj from the language feature extraction model 102. The finding features LFVj are expressed by a language feature vector obtained by making the finding sentence TXj into a feature vector.

[0110] In step S120, the processor 112 inputs the finding feature LFVj output by the language feature extraction model 102 and the key image IMj linked to the finding sentence TXj to the region estimation model 104, and causes the region estimation model 104 to estimate the region of interest (lesion region) in the key image IMj mentioned in the finding sentence TXj. The region estimation model 104 outputs estimated region information PAj estimated from the input finding feature LFVj and key image IMj.

[0111] In step S130, processor 112 calculates a loss indicating an error between estimated region information PAj of the lesion region estimated by region estimation model 104 and position information TPj of the correct region of interest ROIj.

[0112] In step S140, the processor 112 calculates parameter update amounts for each of the language feature extraction model 102 and the region estimation model 104 so as to minimize the loss.

[0113] Then, in step S150, processor 112 updates the parameters of each model of language feature extraction model 102 and region estimation model 104 according to the calculated parameter update amount. Note that training each model to minimize the loss means training each model so that the estimated lesion region estimated by region estimation model 104 matches the correct region of interest ROIj (so that the error between them becomes small). The operations of steps S100 to S150 described above may be performed in mini-batch units.

[0114] After step S150, in step S160, the processor 112 determines whether or not to end the learning. The learning end condition may be determined based on the loss value or the number of parameter updates. As a method based on the loss value, for example, the learning end condition may be that the loss has converged within a specified range. As a method based on the number of updates, for example, the learning end condition may be that the number of updates has reached a specified number. Alternatively, a data set for evaluating the performance of the model may be prepared separately from the training data, and whether or not to end the learning may be determined based on an evaluation value using the evaluation data.

[0115] If the determination result in step S160 is a No determination, processor 112 returns to step S100 and continues the learning process. On the other hand, if the determination result in step S160 is a Yes determination, processor 112 ends the flowchart of FIG.

[0116] The thus generated learned (trained) language feature extraction model 102 is a model that can receive an input of a finding sentence and output a finding feature (feature vector) in which information on the position of a lesion area (area of ​​interest) in an image referred to by the finding sentence is embedded. In other words, information necessary for identifying the position of a lesion area in an image is embedded in the finding feature output by the language feature extraction model 102. The machine learning method executed by the machine learning device 100 can be understood as a method for generating a language feature extraction model 102 that outputs a language feature vector including information for identifying the position of a lesion area in an image described in a finding sentence.

[0117] <Image feature extraction model and cross-modal feature integration model> The object information acquisition unit 34 may include an image feature extraction model. The image feature extraction model is a learned model trained to receive an input of a key image and position information related to a region of interest in the key image, and to output image features indicating the feature amount of the key image.

[0118] The correspondence determination unit 40 may include a cross-modal feature integration model. The cross-modal feature integration model is a learned model trained to determine the correspondence between an image having position information regarding a region of interest in the image and a finding statement describing the region of interest.

[0119] 15 is a block diagram showing a schematic functional configuration of a machine learning device 160 that trains an image feature extraction model and a cross-modal feature integration model using a trained language feature extraction model. The machine learning device 160 may be the same device as the machine learning device 100, or may be the same device as the medical information processing device 20.

[0120] The machine learning device 160 includes a language feature extraction model 102E, an image feature extraction model 162, a cross-modal feature integration model 164, a loss calculation unit 166, and a parameter update unit 168.

[0121] The training dataset may be the same as the dataset used in the machine learning device 100. For example, CNN is applied to the image feature extraction model 162. The image feature extraction model 162 receives input of a key image IMj and position information TPj related to a region of interest ROIj in the key image, and outputs image features IFVj indicating the feature amount of the key image IMj. The image features IFVj may be expressed by an image feature vector obtained by converting the key image IMj into a feature vector. The image features IFVj may be a feature map of multiple channels.

[0122] The language feature extraction model 102E is a learned model trained to receive an input of a finding sentence TXi and output a corresponding finding feature LFVi. The finding sentence TXi input to the language feature extraction model 102E is not limited to a finding sentence TXj (i=j) linked to a key image IMj, and may be a finding sentence (i≠j) not linked to a key image IMj.

[0123] The cross-modal feature integration model 164 receives the image feature IFVj and the finding feature LFVj as input, and outputs a relevance score indicating the relevance between the two. The relevance score may be a numerical value indicating the degree of relevance, and may indicate the degree of certainty of the relevance by a numerical value ranging from 0 to 1, with "0" indicating no relevance and "1" indicating relevance.

[0124] The loss calculation unit 166 calculates a loss indicating an error between the relevance score output from the cross-modal feature integration model 164 and the correct relevance score. When a combination of a key image IMj and a finding sentence TXi (i=j) linked thereto is input to the image feature extraction model 162 and the language feature extraction model 102E, the correct relevance score may be determined as "1". On the other hand, when a combination of a key image IMj and an unrelated finding sentence TXi (i≠j) not linked thereto is input to the image feature extraction model 162 and the language feature extraction model 102E, the correct relevance score may be determined as "0".

[0125] The parameter update unit 168 calculates the update amount of the parameters of each model, the cross-modal feature integration model 164 and the image feature extraction model 162, so as to minimize the loss calculated by the loss calculation unit 166, and updates the parameters of each model according to the calculated update amount.

[0126] The hardware configuration of the machine learning device 160 may be the same as that of the machine learning device 100 shown in Fig. 13. The machine learning device 160 includes a cross-modal feature integration model 164 instead of the region estimation model 104 in Fig. 13. The machine learning device 160 differs from the machine learning device 100 in the loss function of the loss calculated by the loss calculation program 126 and the model whose parameters are updated by the optimizer 128.

[0127] [Overview of machine learning methods] 16 is a flowchart showing an example of a machine learning method executed by the machine learning device 160. In step S101, the processor 112 acquires a data set of a key image IMj, position information TPj on a region of interest ROIj in the key image IMj, and a finding sentence TXi explaining (describes) the region of interest ROIi from a training data set. If i=j in the acquired data set at this time, the processor 112 acquires "1" as a correct relevance score, and if i≠j, acquires "0" as a correct relevance score.

[0128] In step S111, the processor 112 inputs the finding sentence TXi to the language feature extraction model 102E, and causes the language feature extraction model 102E to extract the finding feature LFVi.

[0129] In step S112, the processor 112 inputs the key image IMj and position information TPj relating to the region of interest ROIj in the key image IMj to the image feature extraction model 162, and causes the image feature extraction model 162 to extract image features IFVj.

[0130] In step S114, the processor 112 inputs the image features IFVj output from the image feature extraction model 162 and the finding features LFVi output from the language feature extraction model 102E to the cross-modal feature integration model 164, and causes the cross-modal feature integration model 164 to estimate a relevance score. The processor 112 causes the image feature extraction model 162 to extract the image features IFVj.

[0131] Then, in step S128, the processor 112 calculates a loss indicating the error between the relevance score (estimated value) output from the cross-modal feature integration model 164 and the correct relevance score.

[0132] Then, in step S142, the processor 112 calculates parameter update amounts for each of the image feature extraction model 162 and the cross-modal feature integration model 164 so as to minimize the calculated loss.

[0133] In step S152, the processor 112 updates the parameters of the image feature extraction model 162 and the cross-modal feature integration model 164 in accordance with the calculated parameter update amounts.

[0134] The operations of steps S101 to S152 shown in FIG. 16 may be performed in units of mini-batches.

[0135] After step S152, in step S160, processor 112 determines whether or not to end the learning.

[0136] If the determination result in step S160 is a No determination, processor 112 returns to step S101 and continues the learning process. On the other hand, if the determination result in step S160 is a Yes determination, processor 112 ends the flowchart in FIG.

[0137] By training each model in this way, it is possible to build an AI (Artificial Intelligence) that can accurately determine whether an input image and a finding correspond (are related or not).

[0138] <Identifying regions of interest in medical images> [Configuration of object information acquisition unit] The object information acquisition unit 34 may acquire the object information by analyzing the medical image that is the original image from which the key image is created. Here, a method for identifying the region of interest of the medical image that is the original image from which the key image is created based on the region of interest that is the lesion region of the key image will be described.

[0139] Fig. 17 is a block diagram showing the functional configuration of the object information acquisition unit 34. As shown in Fig. 17, the object information acquisition unit 34 includes a key image acquisition unit 170, a linked information extraction unit 172, a region of interest identification unit 182, and an output unit 190.

[0140] The key image acquisition unit 170 acquires the key image accepted by the image acquisition unit 32 of the medical image processing device 20 .

[0141] The linking information extraction unit 172 analyzes the key image and extracts linking information with the medical image from which the key image was created. That is, the linking information is information for linking the key image with the medical image from which the key image was created. The linking information is, for example, information that is captured in the key image separately from the subject. The linking information includes, for example, at least one of the series number, slice number, window width, window level, and annotation of the medical image from which the key image was created. The linking information may be the result of alignment between the key image and the medical image from which the key image was created. The linking information extraction unit 172 includes a character recognition unit 174, an image recognition unit 176, and an alignment result acquisition unit 180.

[0142] The character recognition unit 174 analyzes characters in the key image by a known character recognition method such as OCR (Optical Character Recognition) to extract the linking information. The linking information extracted by the character recognition unit 174 may include at least one of the window width, window level, slice number, and series number of the key image.

[0143] The image recognition unit 176 extracts the linking information by performing image recognition on the key image. The linking information extracted by the image recognition unit 176 may include at least one of the window width, the window level, and the annotation of the key image. The image recognition unit 176 includes an image recognition model 178. The image recognition model 178 that extracts the window width or the window level of the key image is a classification model using CNN or a regression model. The image recognition model 178 that recognizes the annotation of the key image is a segmentation model to which a convolutional neural network is applied or a detection model. The image recognition unit 176 may include a plurality of image recognition models 178 among the classification model, the regression model, the segmentation model, and the detection model. The image recognition model 178 is stored in the memory 20B.

[0144] Furthermore, the image recognition unit 176 detects annotations added to the key image. The annotations detected by the image recognition unit 176 may include at least one of a circle, a rectangle, an arrow, a line segment, a point, and a scribble.

[0145] The registration result acquisition unit 180 acquires the result of registration between the key image and the medical image performed by the registration unit 186, which will be described later.

[0146] The region of interest identifying section 182 identifies a region of interest based on the linking information extracted by the linking information extracting section 172. Using the linking information, the region of interest identifying section 182, for example, first estimates a position corresponding to the key image in the medical image from which the key image was created, and then identifies the region of interest in the medical image.

[0147] The region of interest identifying section 182 may identify a region of interest from a two-dimensional image, or may identify a region of interest from a three-dimensional image. The identified region of interest may be a two-dimensional region, or may be a three-dimensional region.

[0148] Region of interest identification unit 182 includes region of interest estimation model 184, alignment unit 186, and annotation addition unit 188. Region of interest estimation model 184 is a deep learning model that outputs the position of a region of interest in an input image when an image is given as input. Region of interest estimation model 184 may be a trained model to which CNN is applied. Region of interest estimation model 184 is stored in memory 20B.

[0149] The alignment unit 186 aligns the key image with the original medical image from which the key image was created. Aligning the key image with the original medical image from which the key image was created means matching the pixels of both images showing the same subject, such as an organ. The result of the alignment between the key image and the medical image by the alignment unit 186 includes the correspondence between the pixels of the key image and the pixels of the medical image. The annotation addition unit 188 adds an annotation to the original medical image from which the key image was created.

[0150] The output unit 190 outputs the region of interest identified by the region of interest identifying unit 182. The output region of interest may be at least one of a mask, a bounding box, and a heat map that is added to the medical image from which the key image is created.

[0151] [Medical image analysis method] 18 is a flowchart showing a medical image analysis method by the object information acquisition unit 34. The medical image analysis method is a method for identifying a region of interest in a medical image from which a key image is created.

[0152] In step S171, the key image acquisition unit 170 acquires a key image accepted by the image acquisition unit 32 of the medical image processing device 20. The linking information extraction unit 172 extracts linking information required to link the key image with the medical image from which the key image was created, from the key image. Here, the image recognition unit 176 extracts the linking information from the key image using the image recognition model 178. In addition, the character recognition unit 174 extracts the linking information from the key image using OCR.

[0153] In step S172, if an annotation has been added to the key image acquired in step S171, the image recognition unit 176 detects the annotation from the key image.

[0154] In step S173, the region of interest identifying unit 182 identifies a slice image of the original medical image that is at the same position as the key image, based on the slice number in the linking information extracted in step S171. If the slice number cannot be extracted in step S171, the region of interest identifying unit 182 identifies a slice image at the same position as the key image by a known method.

[0155] In step S174, the registration unit 186 performs registration between the key image and the slice image specified in step S173. Since the key image may be cropped or rotated from the slice image of the original medical image, registration may be necessary.

[0156] In step S175, if an annotation has been added to the key image acquired in step S171, the annotation adding unit 188 adds the annotation to the slice image identified in step S173. By performing the alignment in step S174, the annotation adding unit 188 can add the annotation to the same position of the slice image as the position of the annotation of the key image.

[0157] In step S176, the region of interest identifying unit 182 identifies a region of interest in the slice image based on the annotation added in step S175. Here, the region of interest identifying unit 182 identifies the region of interest using the region of interest estimation model 184. The result of identifying the region of interest may be at least one of a mask, a bounding box, and a heat map. The output unit 190 outputs the identified region of interest.

[0158] In this way, the annotation of the key image is added to the slice image from which the key image is created, and the region of interest is estimated based on the annotation, so that the region of interest of the slice image can be identified, and therefore the region of interest of the original medical image can be identified.

[0159] Here, a case has been described in which annotations have been added to the key image acquired in step S171, but region of interest estimation model 184 can also estimate a region of interest from a key image that does not include annotations.

[0160] <Other> The medical information processing apparatus, the medical information processing method, and the medical information processing program according to the present embodiment can also be applied to an information processing apparatus, an information processing method, and an information processing program that use natural images other than medical images.

[0161] For example, the present invention can be applied to the creation of reports on social infrastructure facilities such as transportation, electricity, gas, and water, etc. In this case, information on objects shown in the received images of one or more infrastructure facilities is obtained, information described in text on the received one or more infrastructure facilities is obtained, and a process can be executed to determine whether or not there is a correspondence between the image and the text based on the information on the objects and the described information, and to assist in the creation of a document including the image and the text based on the presence or absence of the correspondence.

[0162] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other without departing from the spirit of the present invention. [Explanation of symbols]

[0163] 10. Medical information processing systems 12. Medical imaging equipment 14. Medical image database 16...User terminal device 16A…Input device 16B…Display 18...Radiography report database 20...Medical information processing device 20A…Processor 20B…Memory 20C…Communication interface 22…Network 32...Image acquisition section 34…Object information acquisition unit 36…Text acquisition part 38...Recorded information acquisition section 40…Response Judgment Department 42...Auxiliary processing execution unit 100...Machine learning device 102…Language feature extraction model 102E…Linguistic feature extraction model 104…Area estimation model 106...Loss calculation section 108...Parameter update section 112…Processor 114...Computer-readable medium 114A…Memory 114B…Storage 116…Communication interface 118... Input / Output Interface 119…Bus 120...Learning processing program 122...Data acquisition program 126...Loss calculation program 128…Optimizer 130...Display control program 142...Input device 144...Display device 150…Training data storage section 160...Machine learning device 162...Image feature extraction model 164…Cross-modal feature integration model 166...Loss calculation section 168…Parameter update section 170…Key image acquisition section 172...Information extraction section 174...Character recognition section 176…Image recognition section 178…Image recognition model 180…Result acquisition part 182...Region of interest identification unit 184…Region of interest estimation model 186…Alignment section 188…Annotation addition section 190...Output section A1: Lesion area A2: Lesion area A3: Lesion area D1…display area D2…display area FL1…frame IK1…Key image IK2…Key image IK3…Key image IK11…Key image IK12…Key image IK13…Key image IK21…Key image IK22…Key image IK31…Key image IMj…Key image L1…sentence L2…sentence LFVj…Findings and characteristics MK1…Marker NF1…Figure number NF2…Figure number PAj…Estimated area information ROIi…region of interest ROIj…region of interest RP1: Image interpretation report RP2: Image interpretation report RP3: Image interpretation report SN1…Remarks SN2: Observations SN3…Observations SN4…Observations SN5…Remarks SN6…Remarks SN11…Remarks SN12…Remarks SN21…Remarks SN31…Remarks SN32…Remarks S1 to S6: Steps of the medical information processing method S101~S160...Steps of machine learning method S171~S176……Steps of medical image analysis method TDj...training data TPj…location information TXi…Observations TXj…Observations

Claims

1. At least one processor; at least one memory storing instructions for execution by said at least one processor; Equipped with The at least one processor Obtaining information about objects appearing in the one or more received images; Obtain information contained in one or more received documents; determining whether or not there is a correspondence between the image and the text based on information about the object and the written information; executing a process for assisting in the creation of a document including the image and the text based on the presence or absence of the correspondence; Information processing device.

2. The process is a process of issuing a warning when the image corresponding to the text does not exist. The information processing device according to claim 1 .

3. The process is a process of issuing a warning when the image corresponding to the sentence does not exist and the importance of the sentence, which indicates the degree to which an image corresponding to the sentence is necessary, is equal to or greater than a threshold value. The information processing device according to claim 1 .

4. The process is a process of issuing a warning when the text corresponding to the image does not exist. The information processing device according to claim 1 .

5. the process is a process of rearranging the order of one of the sentences and the images that are determined to have a correspondence based on the order of the other. The information processing device according to claim 1 .

6. The process is a process of rearranging the order of the images based on the order of the sentences. The information processing device according to claim 5 .

7. the process is a process of assigning a figure number to the image among the text and the image determined to have the correspondence; The information processing device according to claim 1 .

8. The process of assigning figure numbers is a process of assigning figure numbers to objects in the image in the order in which they appear in the corresponding sentence. The information processing device according to claim 7.

9. the process is a process of assigning the figure number assigned to the image to the sentence among the sentences and the image determined to have the correspondence; The information processing device according to claim 8.

10. The process is a process of assigning figure numbers to corresponding parts of the text. The information processing device according to claim 9.

11. The at least one processor executing the process each time the input of the image or the input of the text is received; The information processing device according to claim 1 .

12. a first mode in which the process is executed each time the input of the image or the input of the text is received, and a second mode in which the process is executed after the input of all the images and all the text is received; Equipped with The information processing device according to claim 11.

13. The at least one processor obtaining a degree of certainty indicating the likelihood that the information about the object corresponds to the described information; determining whether or not there is a correspondence between the image and the text based on the degree of certainty; The information processing device according to claim 1 .

14. The at least one processor analyzing the received image or an original image from which the received image was created to obtain information about the object; The information processing device according to claim 1 .

15. The image is a key image based on a medical image. The information processing device according to claim 1 .

16. the object includes at least one of an organ and a tumor; The information about the object includes at least one of a size, a property, a disease name, a position, and a feature amount. The information processing device according to claim 15.

17. At least one processor Obtaining information about objects appearing in the one or more received images; Obtain information contained in one or more received documents; determining whether or not there is a correspondence between the image and the text based on information about the object and the written information; executing a process for assisting in the creation of a document including the image and the text based on the presence or absence of the correspondence; Information processing methods.

18. A program causing a computer to execute the information processing method according to claim 17.

Images