Information processing device, information processing method, and information processing program

Abstract

This information processing device is provided with at least one processor. The processor: acquires a character string containing a description of one or more first images obtained by photographing a specimen at a first point in time; identifies a first region of interest described in the character string; from among the first images, identifies a first image of interest that contains the first region of interest; from among one or more second images obtained by photographing the specimen at a second point in time, identifies a second image of interest corresponding to the first image of interest; and displays, in mutual association, the first image of interest and the second image of interest on a display.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] Conventionally, image diagnosis has been performed using medical images obtained by imaging devices such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices. Furthermore, medical images have been analyzed by CAD (Computer Aided Detection / Diagnosis) using a classifier trained by deep learning or the like to detect and / or diagnose regions of interest, including structures and lesions, contained in the medical images. The medical images and the results of the CAD analysis are transmitted to a terminal of a medical professional, such as a radiologist, who interprets the medical images. The medical professional, such as a radiologist, uses his or her own terminal to refer to the medical images and the analysis results, interprets the medical images, and creates a radiology report.

[0003] Furthermore, various methods for supporting the creation of radiology reports have been proposed to reduce the burden of radiology work. For example, Japanese Patent Application Laid-Open No. 2019-153250 discloses a technology for creating radiology reports based on keywords entered by a radiologist and the results of analysis of medical images. The technology described in Japanese Patent Application Laid-Open No. 2019-153250 uses a recurrent neural network trained to generate sentences from input characters to create sentences to be included in the radiology report.

[0004] Furthermore, for example, in periodic health checkups and follow-up observations after treatment, the same subject may be examined multiple times, and the changes in the condition over time may be confirmed by comparative interpretation of the medical images taken at each time point. Therefore, various methods for comparative interpretation have been proposed. For example, Japanese Patent Laid-Open Publication No. 2005-012248 discloses a method for aligning images by calculating an index value representing the degree of agreement between multiple past images and multiple current images for all combinations of the two images and extracting the combination with the highest degree of agreement.

[0005] When creating a current interpretation report by comparatively interpreting past and current medical images, it is sometimes necessary to refer to not only the medical images but also the interpretation reports created in the past. Therefore, there is a need for a technology that enables comparative interpretation of past and current medical images for a region of interest described in a past interpretation report.

[0006] The present disclosure provides an information processing device, an information processing method, and an information processing program that can support the creation of an interpretation report.

[0007] A first aspect of the present disclosure is an information processing device comprising at least one processor, wherein the processor acquires a string including a description of at least one first image obtained by photographing a subject at a first time point, identifies a first region of interest described in the string, identifies a first image of interest among the first images that includes the first region of interest, identifies a second image of interest corresponding to the first image of interest among at least one second image obtained by photographing the subject at a second time point, and displays the first image of interest and the second image of interest on a display in association with each other.

[0008] In a second aspect of the present disclosure, in the first aspect, the processor may identify, as the second image of interest, a second image obtained by capturing an image at the same position as the first image of interest.

[0009] In a third aspect of the present disclosure, in the first or second aspect, the processor may accept a selection of a part of the character string to be used to identify the first region of interest.

[0010] A fourth aspect of the present disclosure is any one of the first to third aspects, wherein when a plurality of first regions of interest described in the character string are identified, the processor may identify a first image of interest and a second image of interest for each of the plurality of first regions of interest.

[0011] A fifth aspect of the present disclosure is the fourth aspect, wherein the processor may cause the display to display the first image of interest and the second image of interest identified for each of the plurality of first regions of interest in turn.

[0012] A sixth aspect of the present disclosure is that in the above-mentioned fifth aspect, the processor may cause the display to display the first image of interest and the second image of interest in sequence in an order according to a predetermined priority for each of the multiple first regions of interest.

[0013] A seventh aspect of the present disclosure is that in the above fifth or sixth aspect, the processor may display an input field on the display, associated with the second image of interest, for accepting a string of characters including a description related to the second image of interest, and after accepting the string of characters including a description related to the second image of interest in the input field, cause the next first image of interest and the second image of interest to be displayed on the display.

[0014] An eighth aspect of the present disclosure is the fourth aspect, wherein the processor may display, on a display, a list of first images of interest and second images of interest identified for each of the plurality of first regions of interest.

[0015] A ninth aspect of the present disclosure is that, in any one of the first to eighth aspects, the processor may notify the user to check an area in the second target image corresponding to the first area of ​​interest.

[0016] A tenth aspect of the present disclosure is the above-mentioned ninth aspect, wherein the processor may cause the display to display, as the notification, at least one of a character string indicating the first region of interest, a symbol, and a graphic.

[0017] An eleventh aspect of the present disclosure is any one of the first to tenth aspects, wherein the processor generates comparison information indicating the result of comparing a first region of interest in the first image of interest with a region corresponding to the first region of interest in the second image of interest, and displays the comparison information on a display.

[0018] A twelfth aspect of the present disclosure is any one of the first to eleventh aspects, wherein the processor may highlight a region in the second image of interest that corresponds to the first region of interest.

[0019] A thirteenth aspect of the present disclosure is that, in any one of the first to twelfth aspects above, the processor may cause a character string including at least a description regarding the first region of interest to be displayed on the display in association with the first target image.

[0020] A fourteenth aspect of the present disclosure is that in any one of the first to thirteenth aspects, the processor may cause an input field to be displayed on the display, associated with a second image of interest, for accepting a string of characters including a description related to the second image of interest.

[0021] A fifteenth aspect of the present disclosure is any one of the first to fourteenth aspects, wherein the processor may display the first image of interest and the second image of interest on the display using the same display settings.

[0022] A sixteenth aspect of the present disclosure is the fifteenth aspect, wherein the display settings may be settings relating to at least one of the resolution, gradation, brightness, contrast, window level, window width, and color of the first target image and the second target image.

[0023] A seventeenth aspect of the present disclosure is that in any one of the first to sixteenth aspects above, the processor may, if the second image of interest does not include an area corresponding to the first area of ​​interest, provide a notification indicating that the second image of interest does not include an area corresponding to the first area of ​​interest.

[0024] An eighteenth aspect of the present disclosure is any one of the first to seventeenth aspects, wherein the first image and the second image are medical images, and the first region of interest may be at least one of a region of a structure included in the medical image and a region of an abnormal shadow included in the medical image.

[0025] A 19th aspect of the present disclosure is an information processing method, which includes a process of acquiring a character string including a description of at least one first image obtained by photographing a subject at a first time point, identifying a first region of interest described in the character string, identifying a first image of interest among the first images that includes the first region of interest, identifying a second image of interest corresponding to the first image of interest among at least one second image obtained by photographing the subject at a second time point, and displaying the first image of interest and the second image of interest in association with each other on a display.

[0026] A twentieth aspect of the present disclosure is an information processing program that causes a computer to execute a process of acquiring a string of characters including a description of at least one first image obtained by photographing a subject at a first time point, identifying a first region of interest described in the string of characters, identifying a first image of interest among the first images that includes the first region of interest, identifying a second image of interest corresponding to the first image of interest among at least one second image obtained by photographing the subject at a second time point, and displaying the first image of interest and the second image of interest in association with each other on a display.

[0027] According to the above aspects, the information processing device, the information processing method, and the information processing program of the present disclosure can assist in creating an interpretation report.

[0028] 1 is a diagram illustrating an example of a schematic configuration of an information processing system. 2 is a diagram illustrating an example of a medical image. 3 is a diagram illustrating an example of a medical image. 4 is a block diagram illustrating an example of a hardware configuration of an information processing device. 5 is a block diagram illustrating an example of a functional configuration of an information processing device. 6 is a diagram illustrating an example of a written observation. 7 is a diagram illustrating an example of a screen displayed on a display. 8 is a diagram illustrating an example of a screen displayed on a display. 9 is a flowchart illustrating an example of information processing. 10 is a diagram illustrating an example of a screen displayed on a display. 11 is a diagram illustrating an example of a screen displayed on a display. 12 is a diagram illustrating an example of a screen displayed on a display.

[0029] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. First, the configuration of an information processing system 1 to which an information processing device of the present disclosure is applied will be described. FIG. 1 is a diagram showing a schematic configuration of the information processing system 1. The information processing system 1 shown in FIG. 1 captures an examination target region of a subject and stores the medical images acquired by the capture, based on an examination order from a doctor of a medical department using a known ordering system. The information processing system 1 also allows a radiologist to interpret the medical images and create an interpretation report, and allows the doctor of the requesting medical department to view the interpretation report.

[0030] 1, the information processing system 1 includes an imaging device 2, an interpretation WS (Work Station) 3 which is an interpretation terminal, a medical treatment WS 4, an image server 5, an image DB (Data Base) 6, a report server 7, and a report DB 8. The imaging device 2, the interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8 are connected via a wired or wireless network 9 in a state in which they can communicate with each other.

[0031] Each device is a computer installed with an application program that causes the device to function as a component of the information processing system 1. The application program may be recorded on a recording medium such as a DVD-ROM (Digital Versatile Disc Read Only Memory) or a CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on the computer from the recording medium. Alternatively, the application program may be stored in a state accessible from the outside in a storage device or network storage of a server computer connected to the network 9, and downloaded to the computer upon request and installed.

[0032] The imaging device 2 is a device (modality) that captures an image of a diagnostic target region of a subject, thereby generating a medical image T representing the diagnostic target region. Examples of the imaging device 2 include a plain X-ray imaging device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, an ultrasound diagnostic device, an endoscope, and a fundus camera. The medical image generated by the imaging device 2 is transmitted to an image server 5 and stored in an image DB 6.

[0033] Fig. 2 is a diagram schematically illustrating an example of a medical image acquired by the imaging device 2. The medical image T shown in Fig. 2 is a CT image consisting of a plurality of tomographic images T1 to Tm (m is 2 or more), each representing a cross-sectional plane from the head to the waist of a single subject (human body), for example.

[0034] FIG. 3 is a schematic diagram illustrating an example of one tomographic image Tx among the multiple tomographic images T1-Tm. The tomographic image Tx illustrated in FIG. 3 represents a tomographic plane including the lungs. Each tomographic image T1-Tm may include a structure region SA representing various organs and tissues of the human body (e.g., lungs and liver) and various tissues constituting the organs and tissues (e.g., blood vessels, nerves, muscles, etc.). Each tomographic image may also include an abnormal shadow region AA representing a lesion such as a nodule, tumor, injury, defect, or inflammation. In the tomographic image Tx illustrated in FIG. 3, the lung region is the structure region SA, and the nodule region is the abnormal shadow region AA. Note that a single tomographic image may include multiple structure regions SA and / or abnormal shadow regions AA. Hereinafter, at least one of the structure region SA included in a medical image and the abnormal shadow region AA included in a medical image will be referred to as a "region of interest."

[0035] The image interpretation WS 3 is a computer used by medical professionals, such as radiologists, to interpret medical images and prepare image interpretation reports, and includes the information processing device 10 according to this embodiment. The image interpretation WS 3 issues requests for viewing medical images to the image server 5, performs various image processing on medical images received from the image server 5, displays the medical images, and accepts input of text related to the medical images. The image interpretation WS 3 also performs analysis processing on medical images, supports the preparation of image interpretation reports based on the analysis results, requests the report server 7 to register and view image interpretation reports, and displays image interpretation reports received from the report server 7. These processes are performed by the image interpretation WS 3 by executing software programs for each process.

[0036] The clinical WS 4 is a computer used by medical personnel such as doctors in the clinical department to perform detailed observations of medical images, view interpretation reports, and create electronic medical records, and is composed of a processing device, a display device such as a monitor, and input devices such as a keyboard and a mouse. The clinical WS 4 sends requests to the image server 5 to view medical images, displays medical images received from the image server 5, requests to the report server 7 to view interpretation reports, and displays interpretation reports received from the report server 7. These processes are performed by the clinical WS 4 executing software programs for each process.

[0037] The image server 5 is a general-purpose computer installed with a software program that provides the functions of a database management system (DBMS). The image server 5 is connected to an image DB 6. The connection between the image server 5 and the image DB 6 is not particularly limited, and may be via a data bus or a network such as a NAS (Network Attached Storage) or a SAN (Storage Area Network).

[0038] The image DB 6 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. Medical images acquired by the imaging device 2 and accompanying information attached to the medical images are registered in the image DB 6 in association with each other.

[0039] The incidental information may include, for example, identification information such as an image ID (identification) for identifying the medical image, a cross-sectional ID assigned to each cross-sectional image included in the medical image, a subject ID for identifying the subject, and an examination ID for identifying the examination. The incidental information may also include, for example, information related to the medical image capture, such as the imaging method, imaging conditions, and imaging date and time. Examples of the "imaging method" and "imaging conditions" include the type of imaging device 2, the imaging site, the imaging protocol, the imaging sequence, the imaging technique, whether or not a contrast agent was used, and the slice thickness in the tomographic imaging. The incidental information may also include information related to the subject, such as the subject's name, date of birth, age, and gender. The incidental information may also include information related to the purpose of capturing the medical image.

[0040] Furthermore, when the image server 5 receives a request to register a medical image from the imaging device 2, it converts the medical image into a database format and registers it in the image DB 6. Furthermore, when the image server 5 receives a request to view from the image interpretation WS 3 and the medical treatment WS 4, it searches for medical images registered in the image DB 6 and transmits the searched medical images to the image interpretation WS 3 and the medical treatment WS 4 that have issued the request to view.

[0041] The report server 7 is a general-purpose computer installed with a software program that provides the functions of a database management system. The report server 7 is connected to a report DB 8. The connection between the report server 7 and the report DB 8 is not particularly limited, and may be via a data bus or a network such as a NAS or SAN.

[0042] The report DB 8 is realized by a storage medium such as a HDD, SSD, or flash memory. The report DB 8 stores interpretation reports created by the interpretation WS 3. The report DB 8 may also store finding information related to medical images. The finding information may be, for example, information obtained by the interpretation WS 3 through image analysis of medical images using CAD (Computer Aided Detection / Diagnosis) technology and AI (Artificial Intelligence) technology, or information entered by a user after interpreting a medical image.

[0043] For example, the finding information includes information indicating various findings, such as the name (type), characteristics, location, measurements, and suspected disease name of the region of interest included in the medical image. Examples of names (types) include the names of structures such as "lung" and "liver" and the name of abnormal shadows such as "nodule." Characteristics primarily refer to the characteristics of abnormal shadows. For example, in the case of a pulmonary nodule, findings include absorption values ​​such as "solid" and "ground-glass," margin shapes such as "clear / indistinct," "smooth / irregular," "spicule," "lobulated," and "serrated," and overall shapes such as "near-circular" and "irregular." Other examples include the relationship with surrounding tissues such as "pleural contact" and "pleural indentation," and findings regarding the presence or absence of contrast enhancement and washout.

[0044] The term "location" refers to the anatomical location, the location in the medical image, and the relative positional relationship with other regions of interest, such as "inside," "periphery," and "surroundings." The anatomical location may be indicated by organ names, such as "lung" and "liver," or by subdivisions of the lung, such as "right lung," "upper lobe," and "apical segment (S1)." The measurement value is a value that can be quantitatively measured from the medical image, such as at least one of the size and signal value of the region of interest. The size is expressed, for example, as the major axis, minor axis, area, and volume of the region of interest. The signal value is expressed, for example, as the pixel value of the region of interest and the CT value in units of HU. The estimated disease name is an evaluation result estimated based on the abnormal shadow, such as the disease name, "cancer," "inflammation," and the like, as well as evaluation results regarding the disease name and characteristics, such as "negative / positive," "benign / malignant," and "mild / severe."

[0045] Furthermore, when the report server 7 receives a request to register an interpretation report from the image interpretation WS 3, it converts the interpretation report into a database format and registers it in the report DB 8. Furthermore, when the report server 7 receives a request to view an interpretation report from the image interpretation WS 3 and the medical treatment WS 4, it searches for the interpretation report registered in the report DB 8 and transmits the retrieved interpretation report to the image interpretation WS 3 and the medical treatment WS 4 that have made the view request.

[0046] The network 9 is, for example, a network such as a LAN (Local Area Network) or a WAN (Wide Area Network). The imaging device 2, the image interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8 included in the information processing system 1 may be located in the same medical institution or in different medical institutions. The number of devices, namely the imaging device 2, the image interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8, is not limited to the number shown in FIG. 1 , and each device may be composed of multiple devices having similar functions.

[0047] Incidentally, for example, in periodic health checkups and follow-up observations after treatment, the same subject may be examined multiple times, and changes in the condition over time may be confirmed by comparative interpretation of medical images taken at each time point. Furthermore, when creating a current interpretation report, not only medical images but also interpretation reports created at previous times may be referenced. Therefore, the information processing device 10 according to this embodiment has a function that enables comparative interpretation of a medical image from a previous time point with a medical image from a current time point for a region of interest described in a previous interpretation report. The information processing device 10 will be described below. As described above, the information processing device 10 is included in the interpretation WS 3.

[0048] First, an example of the hardware configuration of an information processing device 10 according to this embodiment will be described with reference to Fig. 4. As shown in Fig. 4, the information processing device 10 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 serving as a temporary storage area. The information processing device 10 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a keyboard and a mouse, and a network I / F (Interface) 26. The network I / F 26 is connected to a network 9 for wired or wireless communication. The CPU 21, the storage unit 22, the memory 23, the display 24, the input unit 25, and the network I / F 26 are connected via a bus 28 such as a system bus and a control bus so as to be able to exchange various information with one another.

[0049] The storage unit 22 is realized by a storage medium such as an HDD, an SSD, or a flash memory. An information processing program 27 for the information processing device 10 is stored in the storage unit 22. The CPU 21 reads the information processing program 27 from the storage unit 22, loads it into the memory 23, and executes the loaded information processing program 27. The CPU 21 is an example of a processor of the present disclosure. For example, a personal computer, a server computer, a smartphone, a tablet terminal, a wearable terminal, or the like can be appropriately applied as the information processing device 10.

[0050] Next, an example of the functional configuration of the information processing device 10 according to this embodiment will be described with reference to Figures 5 to 8. As shown in Figure 5, the information processing device 10 includes an acquisition unit 30, a generation unit 32, an identification unit 34, and a control unit 36. The CPU 21 executes the information processing program 27, causing the CPU 21 to function as each of the functional units of the acquisition unit 30, the generation unit 32, the identification unit 34, and the control unit 36.

[0051] The acquisition unit 30 acquires, from the image server 5, at least one medical image (hereinafter referred to as a "first image") obtained by imaging the subject at a past time point. The acquisition unit 30 also acquires, from the image server 5, at least one medical image (hereinafter referred to as a "second image") obtained by imaging the subject at a current time point. The subject that is the subject of the first image and the second image is the same subject. Hereinafter, an example will be described in which the acquisition unit 30 acquires, as a plurality of first images, a plurality of tomographic images included in a CT image captured at a past time point, and acquires, as a plurality of second images, a plurality of tomographic images included in a CT image captured at a current time point (see FIG. 2 ). The past time point is an example of the first time point in the present disclosure, and the current time point is an example of the second time point in the present disclosure.

[0052] The acquiring unit 30 also acquires, from the report server 7, a character string created in the past that includes a description of the first image. FIG. 6 shows a finding sentence L1 as an example of a character string. The finding sentence L1 includes multiple sentences, including a finding sentence L11 related to a nodule in the lung field, a finding sentence L12 related to mediastinal lymph node enlargement, and a finding sentence L13 related to a hemangioma in the liver. In this manner, the character string acquired by the acquiring unit 30 may include descriptions of multiple regions of interest (e.g., lesions and structures). The character string may be, for example, a document such as an image interpretation report, a sentence such as a finding contained in a document such as an image interpretation report, a sentence including multiple sentences, or a document, a sentence, and words contained in the sentence. Alternatively, the character string may be, for example, a character string indicating finding information stored in the report DB 8.

[0053] The identification unit 34 identifies a first region of interest described in a character string such as a finding statement acquired by the acquisition unit 30. The identification unit 34 may also identify multiple first regions of interest described in a character string such as a finding statement. For example, the identification unit 34 may extract words representing names (types) of lesions and structures, such as "left lower lobe of lung," "nodule," "mediastinal lymphadenopathy," "liver," and "hemangioma," from the finding statement L1, and identify these as the first region of interest. Note that, as a method for extracting words from a character string such as a finding statement, a known named entity extraction method using a natural language processing model, such as BERT (Bidirectional Encoder Representations from Transformers), can be appropriately applied.

[0054] Furthermore, the identification unit 34 may identify a first image of interest that includes a first region of interest identified from a character string such as a finding statement, from among the first images acquired by the acquisition unit 30. For example, the identification unit 34 may extract a region of interest included in each first image by performing image analysis on each of a plurality of first images (tomographic images), and identify a first image that includes a region of interest that substantially matches the first region of interest identified from a character string such as a finding statement, as the first image of interest T11. For example, the identification unit 34 may identify a first image that represents a tomographic plane that includes a "nodule" in the "lower lobe of the left lung" identified from the finding statement L11, as the first image of interest T11.

[0055] The method for extracting the region of interest included in the first image may be, for example, a method using known CAD technology or AI technology. For example, the identification unit 34 may extract the region of interest included in the first image using a learning model such as a convolutional neural network (CNN) that receives a medical image as input and is trained to extract and output the region of interest included in the medical image. By extracting the region of interest included in the first image, the position of the first region of interest in the first target image is also identified.

[0056] Furthermore, the identification unit 34 identifies a second image of interest that corresponds to the first image of interest from among the second images acquired by the acquisition unit 30. Specifically, the identification unit 34 identifies, as the second image of interest, a second image obtained by photographing the same position as the identified first image of interest. Note that, as a method for identifying a second image obtained by photographing the same position as the first image of interest, a known alignment method such as the technique described in Japanese Patent Laid-Open No. 2005-012248 can be appropriately applied.

[0057] Furthermore, when the identification unit 34 identifies multiple first regions of interest from a character string such as a finding sentence, the identification unit 34 may identify a first image of interest and a second image of interest for each of the multiple identified first regions of interest. This is because the multiple first regions of interest may each include different first and second images. For example, in addition to the first image of interest T11 including a "nodule" in the "lower lobe of the left lung," the identification unit 34 may identify a first image representing a cross section including "mediastinal lymph node enlargement" identified from the finding sentence L12 as another first image of interest T12. Furthermore, the identification unit 34 may identify a first image representing a cross section including "liver" and "hemangioma" identified from the finding sentence L13 as another first image of interest T13.

[0058] The generating unit 32 generates a character string such as a finding statement related to the second image of interest identified by the identifying unit 34. Specifically, the generating unit 32 first extracts a region corresponding to the first region of interest in the second image of interest (hereinafter referred to as the "second region of interest"). For example, the generating unit 32 may extract the second region of interest included in the second image of interest using a learning model such as a CNN that receives a medical image as input and is trained to extract and output a region of interest included in the medical image. Furthermore, for example, the generating unit 32 may extract, as the second region of interest, a region in the second image of interest that is located at the same position as the first region of interest in the first image of interest identified by the identifying unit 34.

[0059] The generator 32 then performs image analysis on the extracted second region of interest to generate finding information for the second region of interest. Methods using known CAD and AI technologies can be appropriately applied as methods for acquiring finding information through image analysis. For example, the generator 32 may input the region of interest extracted from the medical image and generate the finding information for the second region of interest using a learning model such as a CNN that has been trained in advance to output finding information for the region of interest.

[0060] The generator 32 then generates a character string such as a finding statement including the generated finding information of the second region of interest. For example, the generator 32 may generate the finding statement using a machine learning technique such as a recurrent neural network described in Japanese Patent Application Laid-Open No. 2019-153250. Alternatively, the generator 32 may generate the finding statement by embedding the finding information in a predetermined template. Alternatively, the generator 32 may generate the finding statement of the second region of interest by reusing a character string such as a finding statement including a description of the first image acquired by the acquisition unit 30 and modifying a portion corresponding to the changed finding information.

[0061] The generator 32 may also generate comparison information indicating the results of comparing a first region of interest in the first image of interest with a second region of interest in the second image of interest. For example, the generator 32 may generate comparison information indicating fluctuations in measurement values ​​such as size and signal values ​​of each region of interest, as well as changes over time such as improvement or deterioration of characteristics, based on the finding information of the first and second regions of interest. For example, when the size of the second region of interest is larger than the first region of interest, the generator 32 may generate comparison information indicating an increasing trend in size. The generator 32 may generate a character string such as a finding statement including the comparison information, or may generate a graph indicating fluctuations in measurement values ​​such as size and signal values.

[0062] The control unit 36 ​​controls the display 24 to display the first image of interest and the second image of interest identified by the identification unit 34 in association with each other. Fig. 7 shows an example of a screen D1 displayed on the display 24 by the control unit 36. The screen D1 displays a first image of interest T11 including a nodule A11 (an example of a first region of interest) in the left lower lobe identified from the finding statement L11 in Fig. 6, and a second image of interest T21 corresponding to the first image of interest T11. As shown in the screen D1, the control unit 36 ​​may display the first image of interest T11 and the second image of interest T21 identified by the identification unit 34 side by side to facilitate comparative interpretation.

[0063] The control unit 36 ​​may also highlight at least one of the first region of interest in the first image of interest and the second region of interest in the second image of interest. For example, as shown in screen D1, the control unit 36 ​​may surround the nodule A11 (first region of interest) in the first image of interest T11 and the nodule A21 (second region of interest) in the second image of interest T21 with a bounding box 90. For example, the control unit 36 ​​may also add a marker such as an arrow near the first region of interest and / or the second region of interest, color-code the first region of interest and / or the second region of interest from other regions, or enlarge and display the first region of interest and / or the second region of interest.

[0064] The control unit 36 ​​may also provide a notification to the user to check the second region of interest in the second image of interest. For example, the control unit 36 ​​may cause the display 24 to display at least one of a character string, a symbol, and a graphic indicating the first region of interest near the nodule A21 (second region of interest) in the second image of interest T21 as a notification. As an example, the screen D1 shows an icon 96 near the nodule A21. For example, the control unit 36 ​​may also provide the notification by means of sound output from a speaker, blinking of a light source such as a light bulb or an LED (Light Emitting Diode), or the like.

[0065] The control unit 36 ​​may also control the display 24 to display the first and second images of interest using the same display settings. The display settings may be, for example, settings related to at least one of the resolution, gradation, brightness, contrast, window level (WL), window width (WW), and color of the first and second images of interest. The window level is a parameter related to the gradation of the CT image and is the median value of the CT number displayed on the display 24. The window width is a parameter related to the gradation of the CT image and is the width between the lower and upper limits of the CT number displayed on the display 24. For example, even for CT images of the same position, display settings suitable for observing the lung field and display settings suitable for observing the mediastinum are different. It is preferable that the control unit 36 ​​use the same display settings for the first and second images of interest to be displayed on the display 24, thereby facilitating comparative interpretation.

[0066] The control unit 36 ​​may also perform control to cause a character string such as a finding statement including a description related to at least the first region of interest acquired by the acquisition unit 30 to be displayed on the display 24 in association with the first image of interest T11. On the screen D1, a finding statement L11 related to the nodule A11 (first region of interest) is displayed below the first image of interest T11.

[0067] The control unit 36 ​​may also perform control to cause a character string such as a finding statement including finding information of the second region of interest generated by the generation unit 32 to be displayed on the display 24 in association with the second image of interest. On the screen D1, a finding statement L21 regarding the nodule A21 (second region of interest) is displayed below the second image of interest T21.

[0068] Furthermore, the control unit 36 ​​may control the display 24 to display comparison information between the first region of interest and the second region of interest generated by the generation unit 32. The finding text L21 on the screen D1 includes a character string indicating a change in the size of the nodule ("It has grown compared to the previous time."), which is underlined 95. In this way, when the control unit 36 ​​displays the character string indicating the comparison information on the display 24, the control unit 36 ​​may highlight the character string by, for example, underlining it or changing the font, bolding, italics, character color, etc.

[0069] The control unit 36 ​​may also accept additions and corrections by the user to the finding statement including the finding information of the second region of interest generated by the generation unit 32. Specifically, the control unit 36 ​​may perform control to display an input field on the display 24 in association with the second image of interest, for accepting a character string such as a finding statement including a description regarding the second image of interest. For example, when the "Edit" button 97 or icon 96 is selected by operating the mouse pointer 92 on the screen D1, the control unit 36 ​​may display an input field (not shown) in the display area 93 of the finding statement L21 for accepting additions and corrections to the finding statement L21.

[0070] Furthermore, when the identification unit 34 identifies multiple first regions of interest from character strings such as findings, the control unit 36 ​​may perform control to display the first image of interest and the second image of interest identified for each of the multiple first regions of interest in turn on the display 24. For example, when the "Next" button 98 is selected on the screen D1, the control unit 36 ​​may transition to a screen D2 that displays the first image of interest and the second image of interest identified for a first region of interest other than the nodule A11.

[0071] 8 shows an example of a screen D2 displayed on the display 24 by the control unit 36. The screen D2 displays a first image of interest T12 including a mediastinal lymph node enlargement A12 (an example of a first region of interest) identified from the finding statement L12 in FIG. 6, and a second image of interest T22 corresponding to the first image of interest T12. As with the screen D1, the screen D2 displays the mediastinal lymph node enlargement A12 in the first image of interest T12 surrounded by a bounding box 90, and a finding statement L12 related to the mediastinal lymph node enlargement A12 is displayed below the first image of interest T12.

[0072] However, a second region of interest corresponding to a first region of interest included in a first image of interest may not necessarily be included in the second image of interest. For example, if a lesion included in a first image of interest captured at a past time point has healed by the present time point, the second region of interest will not be extracted from the second image of interest captured at the present time point.

[0073] Therefore, if the second image of interest does not include the second region of interest, i.e., if the generation unit 32 fails to extract the second region of interest from the second image of interest, the control unit 36 ​​may issue a notification indicating that the second image of interest does not include the second region of interest. As an example, on screen D2, a notification 99 indicates that the second region of interest corresponding to the mediastinal lymph node enlargement A12 in the first image of interest T12 was not extracted from the second image of interest T22. In this case, the generation unit 32 may omit generating a finding statement regarding the second region of interest that could not be extracted. Furthermore, in this case, the control unit 36 ​​may omit displaying the second image of interest T22.

[0074] Furthermore, similar to screen D1, when a "Modify" button 97 is selected on screen D2, the control unit 36 ​​may accept a user input regarding a finding statement related to the second image of interest T22. Furthermore, similar to screen D1, when a "Next" button 98 is selected on screen D2, the control unit 36 ​​may transition to a screen displaying a first image of interest and a second image of interest identified for a first region of interest other than the nodule A11 and the mediastinal lymph node enlargement A12. For example, the control unit 36 ​​may control the display 24 to display a screen including a first image of interest including a hemangioma of the liver (an example of a first region of interest) identified from the finding statement L13 in FIG. 6 and a second image of interest corresponding to the first image of interest (not shown).

[0075] When the first and second focus images are displayed in sequence as described above, the control unit 36 ​​may control the display 24 to display the first and second focus images in sequence in an order according to a predetermined priority for each of the multiple first regions of interest.

[0076] The priority may be determined based on the position of the first image of interest. For example, the priority may be set lower from the head side toward the lumbar region (i.e., the priority may be higher closer to the head). Furthermore, the priority may be determined according to guidelines, manuals, etc. that define the order in which structures and / or lesions included in medical images are to be interpreted.

[0077] Furthermore, for example, the priority may be determined according to at least one of the findings of the first region of interest and the second region of interest diagnosed based on at least one of the first image of interest and the second image of interest. For example, the priority may be higher as the condition of the disease estimated based on at least one of the finding information of the first region of interest acquired by the acquisition unit 30 and the finding information of the second region of interest generated by the generation unit 32 becomes worse.

[0078] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Fig. 9. In the information processing device 10, the CPU 21 executes the information processing program 27, thereby executing the information processing shown in Fig. 9. The information processing is executed, for example, when a user issues an instruction to start execution via the input unit 25.

[0079] In step S10, the acquisition unit 30 acquires at least one medical image (first image) obtained by imaging the subject at a past time point and at least one medical image (second image) obtained by imaging the subject at a current time point. In step S12, the acquisition unit 30 acquires a character string including a description of the first image acquired in step S10.

[0080] In step S14, the identification unit 34 identifies a first region of interest described in the character string acquired in step S12. In step S16, the identification unit 34 identifies a first image of interest from the first images acquired in step S10, the first image of interest including the first region of interest identified in step S14. In step S18, the identification unit 34 identifies a second image of interest from the second images acquired in step S10, the second image of interest corresponding to the first image of interest identified in step S16.

[0081] In step S20, the control unit 36 ​​controls the display 24 to display the first target image identified in step S16 and the first target image identified in step S18 in association with each other, and then ends this information processing.

[0082] As described above, an information processing device 10 according to one aspect of the present disclosure includes at least one processor, and the processor acquires a string including a description of at least one first image obtained by photographing a subject at a first time point, identifies a first region of interest described in the string, identifies a first image of interest among the first images that includes the first region of interest, identifies a second image of interest corresponding to the first image of interest among at least one second image obtained by photographing the subject at a second time point, and displays the first image of interest and the second image of interest on a display in association with each other.

[0083] That is, the information processing device 10 according to the present embodiment enables comparative interpretation of a medical image from the past (first image of interest) and a medical image from the present time (second image of interest) for a first region of interest described in a radiology report from the past, thereby supporting the creation of a radiology report from the present time.

[0084] In the above embodiment, one first image of interest is identified as including the first region of interest, and if there are multiple first regions of interest, the first images of interest that include each of the first regions of interest are different from each other. However, this is not limiting. For example, the identification unit 34 may identify multiple first images of interest that include a certain first region of interest (e.g., a nodule in a lung field). Furthermore, for example, the identification unit 34 may identify the same image as the first image of interest that includes each of multiple first regions of interest (e.g., a nodule in a lung field and enlarged mediastinal lymph nodes).

[0085] In the above embodiment, the generator 32 generates finding information for the second region of interest by performing image analysis on the second image and generates a character string such as a finding statement including the finding information. However, the present invention is not limited to this. For example, the generator 32 may acquire finding information pre-stored in the storage unit 22, the image server 5, the image DB 6, the report server 7, the report DB 8, or other external devices. Furthermore, for example, the generator 32 may acquire finding information manually input by a user via the input unit 25.

[0086] For example, the generator 32 may acquire a character string such as a statement of findings that is stored in advance in the storage unit 22, the report server 7, the report DB 8, or another external device. For example, the generator 32 may accept a character string such as a statement of findings that is manually input by a user. For example, the generator 32 may generate multiple candidate character strings such as a statement of findings that include finding information about the second region of interest, and allow the user to select one of the candidate character strings to be adopted.

[0087] In the above embodiment, the first and second images of interest are identified and displayed for all first regions of interest included in the character string of the findings or the like acquired by the acquisition unit 30. However, the present invention is not limited to this. For example, the control unit 36 ​​may accept a selection of a portion of the character string of the findings or the like acquired by the acquisition unit 30 to be used by the identification unit 34 to identify the first region of interest.

[0088] 10 shows an example of a screen D3 for selecting a portion of the finding L1. As shown in screen D3, the control unit 36 ​​may display, on the display 24, finding statements L11 to L13 obtained by dividing the finding L1 into regions of interest (lesions, structures, etc.), and may accept the selection of at least one of the findings L11 to L13. The user operates the mouse pointer 92 to select at least one of the findings L11 to L13 displayed on screen D3.

[0089] 11 shows another example of a screen D4 for selecting a portion of the finding L1. As shown in screen D4, the control unit 36 ​​may display the finding L1 on the display 24 and accept the selection of any portion of the finding L1. The user operates the mouse pointer 92 to select any portion of the finding L1 displayed on screen D4.

[0090] In the above embodiment, the first and second images of interest identified for each of the plurality of first regions of interest are displayed in sequence, but this is not limiting. For example, the control unit 36 ​​may perform control to display the first and second images of interest identified for each of the plurality of first regions of interest on the display 24 as a list.

[0091] 12 shows a screen D5 in which first images of interest T11-T13 and second images of interest T21-T23 identified for each of a plurality of first regions of interest are displayed in a list format. As shown in screen D5, the control unit 36 ​​may control the display 24 to display a plurality of written findings L11-L13 in association with each of the plurality of first images of interest T11-T13. The control unit 36 ​​may also control the display 24 to display the written findings L21 and L23 in association with the plurality of second images of interest T21-T23, as well as a notification 99 indicating that the second region of interest is not included.

[0092] 13 shows screen D6 as a modified example of screen D5. In screen D6, first images of interest T11 to T13 and second images of interest T21 to T23 are grouped together at the top, and findings L11 to L13, L21, and L23, as well as a notification 99 indicating that the second region of interest is not included, are grouped together at the bottom.

[0093] Furthermore, when displaying the first and second images of interest as a list, the control unit 36 ​​may list the first and second images of interest in an order according to a predetermined priority for each of the multiple first regions of interest. For example, the control unit 36 ​​may rearrange the first and second images of interest so that the upper side of the screen D5 is toward the head and the lower side is toward the lumbar region. Also, for example, the control unit 36 ​​may rearrange the first and second images of interest in an order in which the condition of the first and / or second regions of interest is estimated to be worse.

[0094] In the above embodiment, when the "Next" button 98 is selected on the screen D1 or the screen D2, a first image of interest and a second image of interest specified for a first region of interest other than the currently displayed first region of interest are displayed. However, this is not limiting. For example, the control unit 36 ​​may receive a character string (findings L21) such as a finding statement including a description of the second image of interest in the input field displayed on the screen D1, and then control the display 24 to display the first image of interest and the second image of interest specified for the next first region of interest. In other words, once the addition and correction of the finding statement L21 is completed, the screen may automatically transition to the screen D2 displaying the first image of interest and the second image of interest specified for the first region of interest other than the nodule A11.

[0095] In the above embodiment, a lesion not included in the first image may be included in the second image. For example, a lesion that did not exist in the past may have newly appeared by the present time and may be extractable from the second image captured at the present time. Therefore, the identification unit 34 may identify a region of interest not included in the first image by performing image analysis on the second image. Furthermore, the control unit 36 ​​may notify the user that a region of interest not included in the first image has been detected in the second image.

[0096] In the above embodiment, a radiology report on a medical image has been described, but the present disclosure is not limited thereto. The information processing device 10 of the present disclosure can be applied to various documents that include descriptions of images obtained by photographing a subject. For example, the information processing device 10 may be applied to documents that include descriptions of images obtained using equipment, buildings, pipes, welds, and the like as subjects in non-destructive testing such as radiographic testing and ultrasonic flaw detection testing.

[0097] Furthermore, in the above embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition unit 30, the generation unit 32, the identification unit 34, and the control unit 36. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0098] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0099] Examples of configuring multiple processing units with a single processor include: first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units; second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs); and thus, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0100] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0101] In the above embodiment, the information processing program 27 is pre-stored (installed) in the storage unit 22, but this is not limiting. The information processing program 27 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 27 may also be downloaded from an external device via a network. Furthermore, the technology disclosed herein extends to not only information processing programs but also storage media that non-temporarily store information processing programs.

[0102] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other parts from the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure.

[0103] The disclosure of Japanese Patent Application No. 2022-065907, filed on April 12, 2022, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. At least one processor; The processor, obtaining a character string including a description of at least one first image obtained by imaging the subject at a first time point; Identifying a first region of interest described in the character string; Identifying a first image of interest that includes the first region of interest from among the first images; Identifying a second image of interest corresponding to the first image of interest from at least one second image obtained by imaging the subject at a second time point; The first target image and the second target image are displayed on a display in association with each other. Information processing device.

2. The processor, The second image obtained by photographing the same position as the first image of interest is specified as the second image of interest. The information processing device according to claim 1 .

3. The processor, Accepting a selection of a portion of the character string to be used to identify the first region of interest.

3. The information processing device according to claim 1 or 2.

4. The processor, When a plurality of the first regions of interest described in the character string are identified, the first image of interest and the second image of interest are identified for each of the plurality of first regions of interest.

3. The information processing device according to claim 1 or 2.

5. The processor, The first image of interest and the second image of interest identified for each of the plurality of first regions of interest are displayed in sequence on the display. The information processing device according to claim 4.

6. The processor, The first image of interest and the second image of interest are displayed on the display in sequence in accordance with a priority level that is determined in advance for each of the first regions of interest. The information processing device according to claim 5 .

7. The processor, displaying, on the display, an input field for receiving a character string including a description related to the second image of interest in association with the second image of interest; After receiving a character string including a description related to the second image of interest in the input field, the next first image of interest and the second image of interest are displayed on the display. The information processing device according to claim 5 .

8. The processor, The first image of interest and the second image of interest identified for each of the plurality of first regions of interest are displayed on the display as a list. The information processing device according to claim 4.

9. The processor, A notification is given to a user to check an area in the second image of interest that corresponds to the first area of ​​interest.

3. The information processing device according to claim 1 or 2.

10. The processor, As the notification, at least one of a character string, a symbol, and a graphic representing the first region of interest is displayed on a display. The information processing device according to claim 9.

11. The processor, generating comparison information indicating a result of comparing the first region of interest in the first image of interest with a region corresponding to the first region of interest in the second image of interest; The comparison information is displayed on a display.

3. The information processing device according to claim 1 or 2.

12. The processor, A region corresponding to the first region of interest in the second image of interest is highlighted.

3. The information processing device according to claim 1 or 2.

13. The processor, A character string including at least a description regarding the first region of interest is displayed on the display in association with the first image of interest.

3. The information processing device according to claim 1 or 2.

14. The processor, An input field for receiving a character string including a description regarding the second image of interest is displayed on the display in association with the second image of interest.

3. The information processing device according to claim 1 or 2.

15. The processor, The first target image and the second target image are displayed on the display with the same display setting.

3. The information processing device according to claim 1 or 2.

16. The display settings are settings related to at least one of the resolution, gradation, brightness, contrast, window level, window width, and color of the first image of interest and the second image of interest. The information processing device according to claim 15.

17. The processor, When the second image of interest does not include a region corresponding to the first region of interest, a notification is given to the effect that the second image of interest does not include a region corresponding to the first region of interest.

3. The information processing device according to claim 1 or 2.

18. the first image and the second image are medical images, The first region of interest is at least one of a region of a structure included in the medical image and a region of an abnormal shadow included in the medical image.

3. The information processing device according to claim 1 or 2.

19. obtaining a character string including a description of at least one first image obtained by imaging the subject at a first time point; Identifying a first region of interest described in the character string; Identifying a first image of interest that includes the first region of interest from among the first images; Identifying a second image of interest corresponding to the first image of interest from at least one second image obtained by imaging the subject at a second time point; The first target image and the second target image are displayed on a display in association with each other. An information processing method in which processing is performed by a computer.

20. obtaining a character string including a description of at least one first image obtained by imaging the subject at a first time point; Identifying a first region of interest described in the character string; Identifying a first image of interest that includes the first region of interest from among the first images; Identifying a second image of interest corresponding to the first image of interest from at least one second image obtained by imaging the subject at a second time point; The first target image and the second target image are displayed on a display in association with each other. An information processing program for causing a computer to execute processing.