Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, information processing method and information processing program which can support reading of an image.SOLUTION: An information processing device 10 includes at least one processor. The processor acquires a document described for a specimen, extracts document findings information indicating findings of a subject included in a document, and specifies findings extraction processing for extracting image findings information indicating the findings indicated by the document findings information from a first image in a plurality of types of findings extraction processing for extracting the image findings information indicating the plural types of mutually-different findings that may be included in the first image obtained by imaging the subject.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, image diagnosis has been performed using medical images obtained by imaging devices such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices. Also, medical images have been analyzed by CAD (Computer Aided Detection / Diagnosis) using a discriminator learned by deep learning or the like to detect and / or diagnose regions of interest including structures and lesions contained in the medical images. The medical images and the analysis results by CAD are transmitted to the terminals of medical staff such as radiologists who read the medical images. The medical staff such as radiologists refer to the medical images and the analysis results using their own terminals to read the medical images and create a reading report.

[0003] That is, the analysis results by CAD are described in the reading report after being confirmed by a radiologist. For example, Patent Document 1 discloses a technique for assisting in reviewing the reading findings by a doctor by comparing the CAD findings obtained by analyzing the examination data of a subject with the reading findings by a doctor for the examination data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, it is known that regions of interest in medical images have significant physical characteristics. For example, in CT images of the brain, areas that appear relatively white compared to the surrounding area are suspected of being cerebral hemorrhage, and areas that appear relatively black compared to the surrounding area are suspected of being cerebral infarction. Therefore, if the visibility of regions of interest in medical images can be improved by performing a findings extraction process according to the type of region of interest when radiologists interpret medical images, it will be easier to interpret the images.

[0006] On the other hand, with the recent advancements in imaging equipment and CAD (Computer-Aided Scan) technology, the types of regions of interest detectable by CAD, and the corresponding types of findings extraction processes, are increasing. For radiologists, this has sometimes required trial and error in performing findings extraction processes to identify desired regions of interest in medical images.

[0007] This disclosure provides an information processing device, an information processing method, and an information processing program that can assist in the interpretation of images. [Means for solving the problem]

[0008] A first aspect of the present disclosure is an information processing device comprising at least one processor, the processor acquires a document describing a subject, extracts document findings information indicating findings of the subject contained in the document, and identifies a finding extraction process from a first image that indicates findings indicated by the document findings information, among a plurality of finding extraction processes for extracting image findings information indicating a plurality of different types of findings that may be contained in a first image obtained by photographing the subject.

[0009] In the first embodiment described above, the processor may acquire a first image and extract image finding information indicating at least one type of finding contained in the first image by performing multiple types of finding extraction processes on the first image.

[0010] In the first embodiment described above, the processor may associate the results of extracting document findings information with the results of extracting image findings information for the same region of interest.

[0011] In the first embodiment described above, the processor may associate the results of extracting document findings information with the results of extracting image findings information obtained by performing the same type of findings extraction process as that specified for the document findings information on the first image.

[0012] In the first embodiment described above, the processor may present, in an identifiable manner, the findings included in both the document findings information extraction result and the image findings information extraction result, as well as the findings included in either one of them.

[0013] In the first embodiment described above, the processor may provide a suggestion indicating the possibility of omissions in the extraction process for findings that are included in the document findings information extraction results but not in the image findings information extraction results.

[0014] In the first embodiment described above, the document is a document describing a second image obtained by photographing the subject at a time prior to the time the first image was taken, and the processor may provide a presentation indicating that the findings included in the document findings information extraction results and the image findings information extraction results are being followed up on.

[0015] In the first embodiment described above, the document is a document describing a second image obtained by photographing the subject at a time prior to the time the first image was taken, and the processor may provide a presentation indicating that the findings not included in the document findings information extraction results but included in the image findings information extraction results are new.

[0016] In the first embodiment described above, the document may be a document describing a second image obtained by photographing the subject at a time prior to the time the first image was taken.

[0017] In the first embodiment described above, the first image is a medical image, and the document findings information and image findings information are information indicating at least one of the name, characteristics, measured values, location and estimated disease name relating to the region of interest included in the medical image, and the region of interest may be at least one of the region of a structure included in the medical image and the region of an abnormal shadow included in the medical image.

[0018] A second aspect of this disclosure is an information processing method, which includes a process for obtaining a document describing a subject, extracting document findings information indicating findings of the subject contained in the document, and identifying a findings extraction process among several types of findings extraction processes for extracting image findings information indicating findings indicated by document findings information from a first image obtained by photographing the subject, where image findings information indicating findings indicated by document findings information is extracted from the first image.

[0019] A third aspect of this disclosure is an information processing program that causes a computer to execute a process to identify a finding extraction process, among several types of finding extraction processes for which a document describing a subject is obtained, a document finding information indicating findings of the subject contained in the document is extracted, and a picture of the subject is taken, the image finding information indicating findings of multiple different types that may be contained in the first image is extracted, the image finding information indicating findings of the document finding information is extracted from the first image. [Effects of the Invention]

[0020] According to the above embodiments, the information processing device, information processing method, and information processing program of the present disclosure can support the interpretation of images. [Brief explanation of the drawing]

[0021] [Figure 1] This figure shows an example of the schematic configuration of an information processing system. [Figure 2] This figure shows an example of a medical image. [Figure 3] This figure shows an example of a medical image. [Figure 4] This is a block diagram showing an example of the hardware configuration of an information processing device. [Figure 5] It is a block diagram showing an example of the functional configuration of an information processing apparatus. [Figure 6] It is a diagram showing an example of a radiographic report. [Figure 7] It is a diagram showing an example of document finding information. [Figure 8] It is a diagram showing an example of finding extraction processing. [Figure 9] It is a diagram showing an example of a screen displayed on a display. [Figure 10] It is a flowchart showing an example of first information processing. [Figure 11] It is a diagram showing an example of image finding information. [Figure 12] It is a diagram showing an example of the result of associating document finding information and image finding information. [Figure 13] It is a diagram showing an example of a pattern based on document finding information and image finding information. [Figure 14] It is a diagram showing an example of a screen displayed on a display. [Figure 15] It is a flowchart showing an example of second information processing. [Figure 16] It is a diagram showing an example of a radiographic report. [Figure 17] It is a diagram showing an example of document finding information. [Figure 18] It is a diagram showing an example of image finding information. [Figure 19] It is a diagram showing an example of the result of associating document finding information and image finding information.

Embodiments for Carrying Out the Invention

[0022] Hereinafter, each embodiment of the present disclosure will be described with reference to the drawings.

[0023] [First Embodiment] First, the configuration of the information processing system 1 to which the information processing device of this disclosure is applied will be described. Figure 1 is a diagram showing the schematic configuration of the information processing system 1. The information processing system 1 shown in Figure 1 performs imaging of the area to be examined of a subject and stores the medical images obtained from imaging, based on examination orders from physicians in clinical departments using a known ordering system. It also performs medical image interpretation work by radiologists and the creation of interpretation reports, and allows physicians in the requesting clinical departments to view the interpretation reports.

[0024] As shown in Figure 1, the information processing system 1 includes an imaging device 2, an image interpretation terminal (Image Interpretation WS (WorkStation) 3, a medical examination WS 4, an image server 5, an image database (DB) 6, a report server 7, and a report database 8. The imaging device 2, image interpretation WS 3, medical examination WS 4, image server 5, image database 6, report server 7, and report database 8 are connected to each other via a wired or wireless network 9, enabling them to communicate with one another.

[0025] Each device is a computer on which an application program is installed to function as a component of the information processing system 1. The application program may be recorded and distributed on recording media such as DVDs (Digital Versatile Discs) and CD-ROMs (Compact Disc Read Only Memory), and then installed on the computer from those recording media. Alternatively, it may be stored in a storage device or network storage of a server computer connected to the network 9 in an externally accessible state, and downloaded and installed on the computer upon request.

[0026] The imaging device 2 is a modality that generates a medical image T representing the area to be diagnosed by imaging the area of ​​the subject to be diagnosed. Specifically, this includes plain X-ray imaging devices, CT scanners, MRI scanners, and PET (Positron Emission Tomography) scanners. The medical image generated by imaging device 2 is transmitted to the image server 5 and stored in the image database 6.

[0027] The Image Interpretation WS3 is a computer used by medical professionals, such as radiologists in the radiology department, for interpreting medical images and creating interpretation reports, and it incorporates the information processing device 10 according to this embodiment. The Image Interpretation WS3 handles requests to view medical images from the image server 5, various image processing on medical images received from the image server 5, display of medical images, and acceptance of input of text related to medical images. The Image Interpretation WS3 also performs analysis processing on medical images, assists in creating interpretation reports based on the analysis results, requests registration and viewing of interpretation reports from the report server 7, and displays interpretation reports received from the report server 7. These processes are performed by the Image Interpretation WS3 executing software programs for each process.

[0028] The Clinical WS4 is a computer used by medical professionals, such as physicians in a clinical department, for tasks such as detailed observation of medical images, viewing of image interpretation reports, and creation of electronic medical records. It consists of a processing unit, display devices such as a display, and input devices such as a keyboard and mouse. The Clinical WS4 performs tasks such as requesting access to medical images from the image server 5, displaying medical images received from the image server 5, requesting access to image interpretation reports from the report server 7, and displaying image interpretation reports received from the report server 7. These processes are carried out by the Clinical WS4 executing software programs for each process.

[0029] Image Server 5 is a general-purpose computer with a software program installed that provides the functionality of a Database Management System (DBMS). Image Server 5 is connected to Image DB 6. The connection method between Image Server 5 and Image DB 6 is not particularly limited; it may be connected via a data bus, or via a network such as NAS (Network Attached Storage) or SAN (Storage Area Network).

[0030] The image database 6 is implemented using storage media such as an HDD (Hard Disk Drive), SSD (Solid State Drive), and flash memory. The image database 6 stores medical images acquired by the imaging device 2, along with associated information attached to those medical images.

[0031] The supplementary information may include identification information such as an image ID (identification) for identifying medical images, a tomographic ID assigned to each tomographic image contained in the medical image, a subject ID for identifying the subject, and an examination ID for identifying the examination. The supplementary information may also include information related to the acquisition of medical images, such as the acquisition method, acquisition conditions, and acquisition date and time. "Acquisition method" and "acquisition conditions" refer to, for example, the type of imaging device 2, the acquisition site, acquisition protocol, acquisition sequence, imaging technique, whether or not contrast agent was used, and the slice thickness in tomography. The supplementary information may also include information related to the subject, such as the subject's name, age, and sex.

[0032] Furthermore, when the image server 5 receives a request to register a medical image from the imaging device 2, it formats the medical image into a database format and registers it in the image DB 6. Also, when the image server 5 receives a viewing request from the image interpretation WS3 and the medical treatment WS4, it searches for the medical image registered in the image DB 6 and sends the retrieved medical image to the image interpretation WS3 and the medical treatment WS4 that made the viewing request.

[0033] Report Server 7 is a general-purpose computer with software programs installed that provide the functionality of a database management system. Report Server 7 is connected to Report DB 8. The connection method between Report Server 7 and Report DB 8 is not particularly limited; it may be connected via a data bus, or via a network such as a NAS or SAN.

[0034] The report database (DB8) is implemented using storage media such as HDDs, SSDs, and flash memory. The report database (DB8) stores the image interpretation reports created in the image interpretation workstation (WS3).

[0035] Furthermore, when the report server 7 receives a request to register an image interpretation report from the image interpretation WS3, it formats the image interpretation report into a database format and registers it in the report DB8. Also, when the report server 7 receives a request to view an image interpretation report from the image interpretation WS3 and the medical WS4, it searches the report DB8 for the image interpretation report registered therein and sends the retrieved image interpretation report to the image interpretation WS3 and medical WS4 that made the viewing request.

[0036] Network 9 is, for example, a LAN (Local Area Network) and a WAN (Wide Area Network). The imaging device 2, image interpretation WS3, medical examination WS4, image server 5, image DB6, report server 7, and report DB8 included in the information processing system 1 may be located in the same medical institution or in different medical institutions. Furthermore, the number of each device, imaging device 2, image interpretation WS3, medical examination WS4, image server 5, image DB6, report server 7, and report DB8, is not limited to the number shown in Figure 1, and each device may consist of multiple devices with similar functions.

[0037] Figure 2 is a schematic diagram showing an example of a medical image acquired by the imaging device 2. The medical image T shown in Figure 2 is a CT image consisting of multiple tomographic images T1 to Tm (where m is 2 or more), each representing a cross-sectional plane from the chest to the waist of a single subject (human body). Medical image T and tomographic images T1 to Tm are examples of the first and second images of this disclosure.

[0038] Figure 3 schematically shows an example of a tomographic image Tx from among multiple tomographic images T1 to Tm. The tomographic image Tx shown in Figure 3 represents a tomographic plane including the lung. Each tomographic image T1 to Tm may include regions SA of structures showing various organs of the human body (e.g., lungs and liver, etc.) and various tissues that make up these organs (e.g., blood vessels, nerves and muscles, etc.). In addition, each tomographic image may include regions AA of abnormal shadows showing lesions (e.g., nodules, tumors, injuries, defects and inflammation, etc.) and areas that are unclear in the image. In the tomographic image Tx shown in Figure 3, the lung region is the region SA of structures, and the nodule region is the region AA of abnormal shadows. Hereinafter, at least one of the region SA of structures and the region AA of abnormal shadows will be referred to as the "region of interest". Note that a single tomographic image may contain multiple regions of interest.

[0039] Next, the information processing device 10 will be described. The information processing device 10 according to this embodiment has a function to support the user in interpreting medical images. As described above, the information processing device 10 is included in the image interpretation WS3.

[0040] First, an example of the hardware configuration of the information processing device 10 according to this embodiment will be described with reference to Figure 4. As shown in Figure 4, the information processing device 10 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 as a temporary storage area. The information processing device 10 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a keyboard and mouse, and a network interface 26. The network interface 26 is connected to a network 9 and performs wired or wireless communication. The CPU 21, storage unit 22, memory 23, display 24, input unit 25, and network interface 26 are connected to each other via a bus 28 such as a system bus and a control bus, enabling the exchange of various types of information.

[0041] The storage unit 22 is implemented by a storage medium such as an HDD, SSD, or flash memory. The information processing program 27 of the information processing device 10 is stored in the storage unit 22. The CPU 21 reads the information processing program 27 from the storage unit 22, expands it into memory 23, and executes the expanded information processing program 27. The CPU 21 is an example of the processor of this disclosure. The information processing device 10 can be appropriately applied to, for example, a personal computer, a server computer, a smartphone, a tablet terminal, or a wearable terminal.

[0042] Next, with reference to Figure 5, an example of the functional configuration of the information processing device 10 according to this embodiment will be described. As shown in Figure 5, the information processing device 10 includes an acquisition unit 30, an extraction unit 32, a specification unit 34, and a control unit 36. The CPU 21 executes the information processing program 27, thereby enabling the CPU 21 to function as the acquisition unit 30, the specification unit 34, and the control unit 36.

[0043] The acquisition unit 30 acquires an image interpretation report describing the subject from the report server 7. Figure 6 shows an example of an image interpretation report. The image interpretation report shown in Figure 6 includes a description of the findings regarding the lungs, a description of the findings regarding the liver, and a description indicating that there were no particular findings (np: not particular) regarding the kidneys. The image interpretation report is an example of the document of this disclosure.

[0044] The extraction unit 32 extracts documentary findings information that indicates the findings of the subject included in the image interpretation report acquired by the acquisition unit 30. Figure 7 shows the documentary findings information extracted from the image interpretation report in Figure 6. As shown in Figure 7, documentary findings information is information that indicates, for example, the name (type), characteristics, measured value, location, and at least one of the estimated disease name related to the region of interest included in the medical image.

[0045] Examples of names (types) include the names of structures such as "lungs" and "liver," as well as the names of abnormal shadows such as "nodules." Characteristics mainly refer to the features of the abnormal shadow. For example, in the case of lung nodules, findings include attenuation values ​​such as "solid" and "ground-glass opacity," marginal shapes such as "clear / indistinct," "smooth / irregular," "spicules," "lobulated" and "serrated," and overall shapes such as "round" and "irregular." Other findings include relationships with surrounding tissues such as "pleural contact" and "pleural invagination," as well as findings regarding the presence or absence of contrast enhancement and washout.

[0046] Measured values ​​are values ​​that can be quantitatively measured from medical images, such as size (major axis, minor axis, and volume, etc.), CT values ​​in HU units, and the number of regions of interest and the distance between them if there are multiple regions of interest. Measured values ​​may also be replaced with qualitative expressions such as "large / small" and "many / few." Location refers to anatomical location, location in the medical image, and relative positional relationships with other regions of interest such as "internal," "periphery," and "surroundings." Anatomical location may be indicated by organ names such as "lung" and "liver," or by subdivisions of the lung such as "right lung," "upper lobe," and apical segment ("S1"). Estimated disease name is the evaluation result estimated by the extraction unit 32 based on abnormal shadows, such as disease names such as "cirrhosis," "cancer," and "inflammation," as well as evaluation results such as "negative / positive," "benign / malignant," and "mild / severe" regarding the disease name and characteristics.

[0047] Specifically, the extraction unit 32 may structure each sentence in the image interpretation report using known natural language processing. For example, it may extract words from the image interpretation report and compare them with a dictionary in which the words are pre-associated with the various document observation information mentioned above, thereby extracting the document observation information contained in the image interpretation report. The dictionary may be pre-stored in, for example, the storage unit 22.

[0048] Furthermore, the extraction unit 32 preferably identifies the factual accuracy of words corresponding to document findings information based on the word sequence. "Factual accuracy" refers to information indicating whether a finding is observed or not, as well as the degree of certainty. This is because image interpretation reports may include not only findings that are clearly visible from medical images, but also findings that are suspected but have a low degree of certainty, or findings that are not visible from medical images. For example, in the case of pulmonary nodules, the presence or absence of "calcification" may be used to diagnose the severity, and the image interpretation report may deliberately state that "calcification is not observed."

[0049] Here, we will describe multiple types of finding extraction processes for extracting image finding information that shows multiple different types of findings that may be contained in medical images. Image finding information refers to information that shows, for example, the name (type), characteristics, measured value, location, and at least one of the estimated disease name related to the region of interest contained in the medical image. The details of the various types of information shown by the image finding information are the same as the details of the various types of information shown by the document finding information described above, so we will omit the explanation.

[0050] Figure 8 shows a list of multiple types of finding extraction processes M1 to M6 for extracting various image finding information from medical images. As shown in Figure 8, each of the finding extraction processes M1 to M6 targets different organs, lesions, and / or disease names. By applying the finding extraction processes M1 to M6 to a medical image, image finding information related to the target organs, lesions, and / or disease names is extracted. The correspondence between the finding extraction processes M1 to M6 and the target organs, lesions, and / or disease names is pre-stored in the storage unit 22, for example, as a table.

[0051] The findings extraction processes M1, M2, M4-M6 ("pixel value filters 1-5") are pixel value filters with different thresholds, such as high-pass and low-pass filters. For example, in a CT image of the brain, areas that appear relatively white compared to the surrounding area are suspected of being cerebral hemorrhage, and areas that appear relatively black compared to the surrounding area are suspected of being cerebral infarction. For example, if findings extraction process M5 ("pixel value filter 4") is applied to a medical image of the brain and an area that appears relatively white compared to the surrounding area is detected, image findings information indicating the presence of cerebral hemorrhage can be extracted. On the other hand, if findings extraction process M6 ("pixel value filter 5") is applied to a medical image of the brain and an area that appears relatively black compared to the surrounding area is detected, image findings information indicating the presence of cerebral infarction can be extracted.

[0052] The findings extraction process M3 ("shape enhancement filter") is a shape enhancement filter such as a known edge detection filter. For example, in CT images of the liver, cirrhosis is suspected when the edges of the liver have an irregular, uneven shape. Therefore, if, for example, the findings extraction process M3 ("shape enhancement filter") is applied to a medical image of the liver and an area with an irregular, uneven edge is detected, image findings information indicating cirrhosis can be extracted.

[0053] Furthermore, for the findings extraction process, a pre-trained model such as a CNN (Convolutional Neural Network) may be used, which is pre-trained to take a medical image as input and output image findings information extracted from the medical image. This pre-trained model is, for example, a model trained by machine learning using a combination of a medical image in which the region of interest (i.e., a region with predetermined physical characteristics) is known, and the image findings information indicated by the region of interest contained in the medical image, as training data. "Regions with physical characteristics" include, for example, regions within a range of pixel values ​​that are predetermined (for example, regions where the pixel values ​​are relatively white / black compared to the surrounding area), and regions with predetermined shapes.

[0054] For example, instead of the finding extraction process M1 ("pixel value filter 1"), a pre-trained model may be used that takes a medical image of the lung as input and outputs image finding information showing the characteristics, measured values, location, and estimated disease name of a pulmonary nodule extracted from the medical image. Alternatively, for example, instead of the finding extraction process M3 ("shape enhancement filter"), a pre-trained model may be used that takes a medical image of the liver as input and outputs image finding information showing the characteristics, measured values, location, and estimated disease name (e.g., cirrhosis) of the liver extracted from the medical image. Alternatively, for example, instead of the finding extraction process M5 ("pixel value filter 4"), a pre-trained model may be used that takes a medical image of the brain as input and outputs image finding information showing the characteristics, measured values, location, and estimated disease name of a cerebral hemorrhage extracted from the medical image.

[0055] Furthermore, multiple pre-trained models may be combined to extract image findings from medical images. For example, instead of the findings extraction process M1 ("pixel value filter 1"), a combination may be used of a first pre-trained model that takes a medical image of the lung as input and outputs the region of a pulmonary nodule extracted from the medical image, and a second pre-trained model that takes the region of a pulmonary nodule extracted from the medical image as input and outputs the image findings information of the pulmonary nodule region.

[0056] The identification unit 34 identifies a finding extraction process from among the predetermined multiple types of finding extraction processes described above, for extracting image finding information from medical images that indicates the findings indicated by the document finding information extracted from the image interpretation report by the extraction unit 32. Specifically, the identification unit 34 identifies the corresponding finding extraction process by comparing the document finding information extracted from the image interpretation report (see Figure 7) with a table (see Figure 8) that defines the correspondence between the finding extraction process and the target organ, lesion, and / or disease name.

[0057] In the examples shown in Figures 6 to 8, the identification unit 34 identifies "pixel value filter 1" (finding extraction process M1) as a finding extraction process for extracting image finding information indicating "pulmonary nodules" that correspond to document finding information indicating "nodules" in the "lungs". The identification unit 34 also identifies "shape enhancement filter" (finding extraction process M3) as a finding extraction process for extracting image finding information indicating "cirrhosis" that corresponds to document finding information indicating "cirrhosis" in the "liver".

[0058] The control unit 36 ​​presents the document findings information extracted from the image interpretation report by the extraction unit 32, and the findings extraction process identified by the identification unit 34. Figure 9 shows an example of screen D1 displayed on the display 24 by the control unit 36. Screen D1 includes the image interpretation report acquired by the acquisition unit 30. The document findings information identified by the identification unit 34 and the findings extraction process are presented in association with each other.

[0059] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Figure 10. In the information processing device 10, the CPU 21 executes the information processing program 27, thereby executing the first information processing shown in Figure 10. The first information processing is executed, for example, when the user gives an instruction to start execution via the input unit 25.

[0060] In step S10, the acquisition unit 30 acquires the image interpretation report from the report server 7. In step S12, the extraction unit 32 extracts the document findings information contained in the image interpretation report acquired in step S10. In step S14, the identification unit 34 identifies a findings extraction process from among a predetermined number of findings extraction processes to extract image findings information that indicates the findings indicated by the document findings information extracted in step S12. In step S16, the control unit 36 ​​presents the findings extraction process identified in step S14 and terminates this first information processing.

[0061] As described above, an information processing device 10 according to one aspect of the present disclosure comprises at least one processor, the processor acquires a document describing a subject, extracts document findings information indicating findings of the subject contained in the document, and identifies a findings extraction process from the first image that indicates findings indicated by the document findings information, among a plurality of findings extraction processes for extracting image findings information indicating a plurality of different types of findings that may be contained in a first image obtained by photographing the subject.

[0062] In other words, the information processing device 10 according to this embodiment can identify the appropriate finding extraction process when interpreting findings described in the image interpretation report from medical images. Therefore, for example, when a reader of an image interpretation report checks a medical image, or when a radiologist re-interprets an image, the appropriate finding extraction process for the organ and lesion to be interpreted can be identified, thereby supporting the interpretation of medical images.

[0063] Furthermore, for example, when conducting follow-up observations on the same subject, it is sometimes necessary to refer to interpretation reports created in the past while interpreting the current medical images. In other words, when interpreting current medical images, it is sometimes necessary to search for findings described in interpretation reports created in the past. According to the information processing device 10 of this embodiment, it is possible to identify an appropriate finding extraction process when confirming findings described in interpretation reports created in the past with respect to the current medical images, thereby supporting the interpretation of medical images. That is, the "document" to be processed by the information processing device 10 of this embodiment may be a document describing past medical images obtained by photographing the subject at a time earlier than the time the medical images to be processed for finding extraction were taken.

[0064] [Second Embodiment] The information processing device 10 according to this embodiment is specifically designed to support the user in interpreting medical images by referring to interpretation reports created in the past. The information processing device 10 according to the second embodiment will be described below, but the same configuration and functions as in the first embodiment will be omitted from the explanation as appropriate.

[0065] The acquisition unit 30 acquires a medical image of the current time (hereinafter referred to as "current image") from the image server 5. The acquisition unit 30 also acquires an interpretation report describing past medical images (hereinafter referred to as "past images") from the report server 7. The current image and past images are images of the same subject. The current image is an example of the first image of this disclosure, and the past image is an example of the second image of this disclosure.

[0066] The extraction unit 32 extracts image finding information indicating at least one type of finding contained in the current image by performing multiple types of finding extraction processes (see Figure 8) on the current image. Specifically, the extraction unit 32 performs finding extraction processes M1 to M6 on each of the multiple tomographic images acquired as the current image, and extracts image finding information from each tomographic image. In the following description, it will be assumed that the results shown in Figure 11 are obtained as the result of extracting image finding information by the extraction unit 32.

[0067] Furthermore, the extraction unit 32 extracts document findings information contained in the image interpretation report describing the past images acquired by the acquisition unit 30. In the following explanation, the image interpretation report describing the past images is the image interpretation report shown in Figure 6, and the results of the document findings information extracted by the extraction unit 32 are as shown in Figure 7.

[0068] The identification unit 34 identifies a findings extraction process from among several predetermined types of findings extraction processes, for extracting image findings information from the current image that indicates the findings indicated by the document findings information extracted from the image interpretation report described by the extraction unit 32 for past images.

[0069] Furthermore, the identification unit 34 associates the extraction results of document findings information extracted by the extraction unit 32 with the image findings information extracted by the extraction unit 32 for the same region of interest. Specifically, the identification unit 34 may associate the extraction results of document findings information extracted by the extraction unit 32 with the extraction results of image findings information obtained by performing the same type of findings extraction process as the one specified for the document findings information on the current image. As described above, the findings extraction processes M1 to M6 each target different organs and / or lesions. Therefore, it can be said that the document findings information and image findings information related to the same type of findings extraction process each concern the same region of interest (i.e., organs and / or lesions).

[0070] For example, for document findings indicating a "nodule" in the "lung," the "pixel value filter 1" (finding extraction process M1) is specified. In this case, the identification unit 34 associates the extracted document findings indicating a "nodule" in the "lung" with the extracted image findings obtained by applying the "pixel value filter 1" to the current image. Also, for example, for document findings indicating "cirrhosis" in the "liver," the "shape enhancement filter" (finding extraction process M3) is specified. In this case, the identification unit 34 associates the extracted document findings indicating "cirrhosis" in the "liver" with the extracted image findings obtained by applying the "shape enhancement filter" to the current image. Figure 12 shows the result of associating the extracted document findings shown in Figure 7 with the extracted image findings shown in Figure 11.

[0071] Furthermore, the identification unit 34 determines patterns based on the findings included in both the document findings information extraction results and the image findings information extraction results, and the findings included in either one. Figure 13 shows each pattern. As shown in Figure 13, the identification unit 34 determines that findings included in both the document findings information extraction results and the image findings information extraction results are follow-up findings. The identification unit 34 also determines that findings not included in the document findings information extraction results but included in the image findings information extraction results are new findings. The identification unit 34 also determines that findings included in the document findings information extraction results but not in the image findings information extraction results are findings that may have been missed during the findings extraction process performed by the extraction unit 32.

[0072] The control unit 36 ​​presents the findings extraction process identified by the identification unit 34 for the document findings information extracted by the extraction unit 32 from the image interpretation report describing past images. The control unit 36 ​​also presents findings that are included in both the document findings information extraction result and the image findings information extraction result, and findings that are included in either one, in an identifiable manner. "Identifiable presentation" can be achieved, for example, as shown in Figure 14, by displaying strings such as "Follow-up lesion" for follow-up findings, "New lesion" for new findings, and "Requires confirmation" for findings that may have been missed, depending on the pattern determined by the identification unit 34. Alternatively, this can be achieved by varying the display format, such as the type of text (color, font, bold and italics, etc.), background color, and border type, when presenting findings, according to each pattern. Alternatively, it can be achieved by displaying icons that represent each pattern.

[0073] Figure 14 shows an example of screen D2 displayed on the display 24 by the control unit 36. Screen D2 includes an image interpretation report describing past images acquired by the acquisition unit 30, and the current image. In addition, the extraction results of document findings information and image findings information extracted by the extraction unit 32, and the findings extraction process identified for the document findings information by the identification unit 34 are presented in association with each other. The pattern determination results by the identification unit 34 are also presented.

[0074] Furthermore, the control unit 36 ​​may add a hyperlink 80 to the medical image from which image finding information is extracted to a string indicating the finding (for example, "pulmonary nodule," "liver cirrhosis," and "renal tumor"). The user operates a cursor (not shown) on screen D2 via the input unit 25 and requests to view a medical image by selecting the string to which the hyperlink 80 is attached. For example, if the hyperlink 80 attached to the string "pulmonary nodule" is selected on screen D2 in Figure 14, the control unit 36 ​​may control the display 24 to display the medical image from which the image finding information indicating a pulmonary nodule was extracted by the extraction unit 32.

[0075] However, if a finding is not included in the image findings information and there is a possibility that it has been missed during extraction, the medical image from which the image findings information was extracted cannot be identified. In this case, the control unit 36 ​​may use a medical image containing the organ from which the finding can be extracted as the link destination of the hyperlink 80. For example, if the hyperlink 80 attached to the string "cirrhosis" is selected on screen D2 in Figure 14, the control unit 36 ​​may control the display 24 to display a medical image containing the liver, regardless of whether cirrhosis has been extracted or not.

[0076] Furthermore, the control unit 36 ​​may automatically perform the corresponding findings extraction process on the medical image linked to by the hyperlink 80. For example, when the hyperlink 80 attached to the string "pulmonary nodule" is selected on screen D2 in Figure 14, the control unit 36 ​​may perform "pulmonary nodule extraction" (finding extraction process M1) on the original medical image from which the extraction unit 32 extracted image findings indicating a pulmonary nodule, and then control the display to show it on the display 24.

[0077] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Figure 15. In the information processing device 10, the CPU 21 executes the information processing program 27, thereby executing the second information processing shown in Figure 15. The second information processing is executed, for example, when the user gives an instruction to start execution via the input unit 25.

[0078] In step S20, the acquisition unit 30 acquires an image interpretation report describing past images from the report server 7. In step S22, the extraction unit 32 extracts document findings information contained in the image interpretation report acquired in step S20. In step S24, the identification unit 34 identifies a findings extraction process from among several predetermined types of findings extraction processes to extract image findings information that indicates the findings indicated by the document findings information extracted in step S22.

[0079] In step S26, the acquisition unit 30 acquires the current image from the image server. In step S28, the extraction unit 32 extracts image observation information indicating at least one type of observation contained in the current image by performing multiple types of observation extraction processing on the current image acquired in step S26. In step S30, the identification unit 34 associates the extraction result of document observation information extracted in step S22 with the extraction result of image observation information extracted in step S28.

[0080] In steps S32 to S42, the identification unit 34 determines patterns based on the document and image information extraction results associated in step S30, corresponding to findings included in both the document and image information extraction results and findings included in either one. Specifically, the identification unit 34 determines that findings included in the document information extraction results (step S32 is Y) and included in the image information extraction results (step S34 is Y) are follow-up findings, as shown in step S36. The identification unit 34 also determines that findings included in the document information extraction results (step S32 is Y) and not included in the image information extraction results (step S34 is N) are findings that may have been missed, as shown in step S38. Furthermore, the identification unit 34 determines that any findings that are not included in the document findings information extraction results (step S32 is N) but are included in the image findings information extraction results (step S40 is Y) are new findings, as shown in step S42.

[0081] In step S44, the control unit 36 ​​presents the determination results from steps S36, S38, and S42 in an identifiable manner, and terminates this second information processing. On the other hand, the control unit 36 ​​does not present findings that are not included in the document findings information (step S32 is N) and are not included in the image findings information (step S40 is N), and terminates this second information processing as is.

[0082] As described above, the information processing device 10 according to one aspect of the present disclosure comprises at least one processor, which acquires a document describing a subject and extracts document findings information indicating findings of the subject contained in the document. The processor also extracts image findings information indicating at least one type of finding contained in a first image obtained by photographing the subject, and associates the results of extracting document findings information with the results of extracting image findings information for the same region of interest. Furthermore, the processor presents in an identifiable manner the findings contained in both the results of extracting document findings information and the results of extracting image findings information, and the findings contained in either one of them.

[0083] In other words, the information processing device 10 according to this embodiment can present, in an identifiable manner, whether or not each finding is described in the image interpretation report for past images, and whether or not it has been extracted from the current image using CAD. This allows the radiologist to perform the image interpretation work while understanding whether each finding is a follow-up, new, or potentially a missed extraction by CAD. Therefore, the information processing device 10 according to this embodiment can support the interpretation of medical images.

[0084] Furthermore, in the information processing device 10 according to this embodiment, whether or not findings are included in past images is determined based on the image interpretation report, and CAD analysis is not performed on past images. In other words, according to the information processing device 10 according to this embodiment, even without performing CAD analysis on past images, it is possible to compare findings between past images and current images, thereby supporting the interpretation of medical images.

[0085] In the second embodiment described above, an example of applying a reading report that describes past images was explained, but it is also possible to apply a reading report that describes current images. Even in this case, the identification unit 34 can identify findings that are included in the document findings information but not in the image findings information as findings that may have been missed during the findings extraction process.

[0086] Furthermore, in the second embodiment described above, a configuration was described in which the results of extracting document findings information are associated with the results of extracting image findings information obtained by performing the same type of findings extraction process as the one specified for the document findings information on the current image. However, the method of association is not limited to this. For example, the identification unit 34 may associate the results of extracting document findings information and the results of extracting image findings information for each lesion based on measured values ​​such as the characteristics and size of the lesion, and information indicating its location.

[0087] Referring to Figures 16 to 19, a specific example of the process of associating the extracted document findings with the extracted image findings for each lesion will be explained. Figure 16 is an example of a reading report describing past images acquired by the acquisition unit 30, and includes descriptions of multiple pulmonary nodules. Figure 17 shows the document findings extracted by the extraction unit 32 from the reading report in Figure 16. As shown in Figure 17, the extraction unit 32 may distinguish lesions by using words that indicate different characteristics for each lesion, such as nature ("solid type", etc.), size ("3 cm", etc.), and location ("right lung S3", etc.).

[0088] Figure 18 shows the image findings information extracted from the current image by the extraction unit 32. As shown in Figure 18, the extraction unit 32 may distinguish lesions by extracting different characteristics from the current image, such as the nature, size, and location of the lesions included in the current image.

[0089] Figure 19 shows the results of associating the extracted document findings information shown in Figure 17 with the extracted image findings information shown in Figure 18 for each lesion. As shown in Figure 19, the identification unit 34 may associate the extracted document findings information with the extracted image findings information if at least one of the findings indicating the characteristics, size, and location of the lesion matches. The identification unit 34 may also perform pattern determination for each lesion between findings included in both the extracted document findings information and the extracted image findings information, and findings included in either one.

[0090] Furthermore, the specific unit 34 may identify the trend of change in characteristics, size, and location of lesions determined to be lesions requiring follow-up. "Trend of change" refers to, for example, improvement or deterioration of characteristics, enlargement or reduction in the size of the lesion, primary and metastatic status, and the degree of these changes (large / small / no change). In the example in Figure 19, for a lesion whose size has increased from "2 cm" to "3 cm", information indicating a trend of change, "increase," has been added to the "size" column. The control unit 36 ​​may present this information indicating a trend of change.

[0091] In the embodiments described above, medical images were used as examples of the first and second images. However, the technology of this disclosure can also be used with images other than medical images. For example, the technology of this disclosure can be applied to images taken in non-destructive testing of civil engineering structures, industrial products, and piping (e.g., CT images, visible light images, and infrared images) and reports describing such images.

[0092] Furthermore, in each of the above embodiments, the hardware structure of the processing unit that executes various processes, such as the acquisition unit 30, extraction unit 32, identification unit 34, and control unit 36, can be the various processors shown below. As mentioned above, these various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to execute specific processes.

[0093] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0094] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, which then functions as multiple processing units, as exemplified by client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System on Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0095] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits, which are combinations of circuit elements such as semiconductor devices.

[0096] Furthermore, although the above embodiment describes an embodiment in which the information processing program 27 is pre-stored (installed) in the storage unit 22, the invention is not limited to this. The information processing program 27 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), and USB (Universal Serial Bus) memory. Alternatively, the information processing program 27 may be provided in the form of a download from an external device via a network. Moreover, the technology of this disclosure extends not only to the information processing program but also to storage media for non-temporarily storing the information processing program.

[0097] The technology of this disclosure can also be appropriately combined with the above-described embodiments. The descriptions and illustrations shown above are detailed explanations of the parts relating to the technology of this disclosure and are merely examples of the technology of this disclosure. For example, the above-described explanation of the configuration, function, operation, and effect is an explanation of an example of the configuration, function, operation, and effect of the parts relating to the technology of this disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements added, or replaced from the descriptions and illustrations shown above, as long as they do not deviate from the spirit of the technology of this disclosure. [Explanation of Symbols]

[0098] 1. Information Processing System 2. Imaging device 3 Image Interpretation Workshop 4. Medical Workshop 5 Image Server 6 Image Database 7. Report Server 8 Report Database 9 Network 10 Information Processing Devices 21 CPU 22 Memory section 23 memory 24 displays 25 Input section 26 Network Interface 27 Information Processing Programs 28 buses 30 Acquisition Department 32 Extraction part 34 Specific part 36 Control Unit AA Area of ​​abnormal shadows D1, D2 screen Area of ​​SA structures T Medical Images T1-Tm, Tx tomographic images

Claims

1. Equipped with at least one processor, The aforementioned processor, Obtain a document describing the subject, Extract document findings information that shows the findings of the subject contained in the aforementioned document, Among multiple types of finding extraction processes for extracting image finding information that shows multiple different types of findings that may be included in the first image obtained by photographing the subject, a finding extraction process for extracting image finding information that shows the findings indicated by the document finding information from the first image is specified. The first image is obtained, By performing the multiple types of finding extraction processes on the first image, the image finding information indicating at least one type of finding contained in the first image is extracted. The results of extracting document findings information and the results of extracting image findings information for the same area of ​​interest are associated with each other. The findings included in both the extracted document findings information and the extracted image findings information, and the findings included in either one, are presented in a way that allows for identification. Information processing device.

2. Equipped with at least one processor, The aforementioned processor, Obtain a document describing the subject, Extract document findings information that shows the findings of the subject contained in the aforementioned document, Among multiple types of finding extraction processes for extracting image finding information that shows multiple different types of findings that may be included in the first image obtained by photographing the subject, a finding extraction process for extracting image finding information that shows the findings indicated by the document finding information from the first image is specified. The first image is obtained, By performing the multiple types of finding extraction processes on the first image, the image finding information indicating at least one type of finding contained in the first image is extracted. The results of extracting the document findings information are associated with the results of extracting the image findings information obtained by performing the same type of findings extraction process on the first image as the findings extraction process identified for the document findings information. The findings included in both the extracted document findings information and the extracted image findings information, and the findings included in either one, are presented in a way that allows for identification. Information processing device.

3. The aforementioned processor, For findings included in the document findings information extraction results but not included in the image findings information extraction results, the possibility of omissions due to the findings extraction process will be presented. The information processing apparatus according to claim 1 or claim 2.

4. The aforementioned document describes a second image obtained by photographing the subject at a time prior to the time the first image was taken. The aforementioned processor, The document findings included in the extraction results of the document findings information, and the image findings included in the extraction results of the image findings information, will be presented to indicate that they have been followed up on. An information processing apparatus according to any one of claims 1 to 3.

5. The aforementioned document describes a second image obtained by photographing the subject at a time prior to the time the first image was taken. The aforementioned processor, For findings that are not included in the extraction results of the aforementioned document findings information, but are included in the extraction results of the aforementioned image findings information, a presentation will be made indicating that they are new. An information processing apparatus according to any one of claims 1 to 4.

6. The aforementioned document describes a second image obtained by photographing the subject at a time prior to the time the first image was taken. An information processing apparatus according to any one of claims 1 to 5.

7. The first image mentioned above is a medical image, The aforementioned document findings information and the aforementioned image findings information are information indicating at least one of the name, characteristics, measured values, location, and estimated disease name related to the region of interest included in the medical image. The region of interest is at least one of the regions of structures included in the medical image and the regions of abnormal shadows included in the medical image. An information processing apparatus according to any one of claims 1 to 6.

8. Obtain a document describing the subject, Extract document findings information that shows the findings of the subject contained in the aforementioned document, Among multiple types of finding extraction processes for extracting image finding information that shows multiple different types of findings that may be included in the first image obtained by photographing the subject, a finding extraction process for extracting image finding information that shows the findings indicated by the document finding information from the first image is specified. The first image is obtained, By performing the multiple types of finding extraction processes on the first image, the image finding information indicating at least one type of finding contained in the first image is extracted. The results of extracting document findings information and the results of extracting image findings information for the same area of ​​interest are associated with each other. The findings included in both the extracted document findings information and the extracted image findings information, and the findings included in either one, are presented in a way that allows for identification. An information processing method in which a computer performs the processing.

9. Obtain a document describing the subject, Extract document findings information that shows the findings of the subject contained in the aforementioned document, Among multiple types of finding extraction processes for extracting image finding information that shows multiple different types of findings that may be included in the first image obtained by photographing the subject, a finding extraction process for extracting image finding information that shows the findings indicated by the document finding information from the first image is specified. The first image is obtained, By performing the multiple types of finding extraction processes on the first image, the image finding information indicating at least one type of finding contained in the first image is extracted. The results of extracting document findings information and the results of extracting image findings information for the same area of ​​interest are associated with each other. The findings included in both the extracted document findings information and the extracted image findings information, and the findings included in either one, are presented in a way that allows for identification. An information processing program that causes a computer to perform a task.

Images