Information processing device, information processing method, and information processing program

Abstract

Provided is an information processing device comprising at least one processor, the processor acquiring a character string containing a description of a first region of interest, identifying a second region of interest that is not described in the character string but that is associated with the first region of interest, and issuing a notice to a user to confirm whether to display an image that could include the second region of interest.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] Conventionally, image diagnosis has been performed using medical images obtained by imaging devices such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices. Furthermore, medical images have been analyzed by CAD (Computer Aided Detection / Diagnosis) using a classifier trained by deep learning or the like to detect and / or diagnose regions of interest, including structures and lesions, contained in the medical images. The medical images and the results of the CAD analysis are transmitted to a terminal of a medical professional, such as a radiologist, who interprets the medical images. The medical professional, such as a radiologist, uses his or her own terminal to refer to the medical images and the analysis results, interprets the medical images, and creates a radiology report.

[0003] Furthermore, various methods for supporting the creation of radiology reports have been proposed to reduce the burden of radiology work. For example, Japanese Patent Application Laid-Open No. 2019-153250 discloses a technology for creating radiology reports based on keywords entered by a radiologist and the results of analysis of medical images. The technology described in Japanese Patent Application Laid-Open No. 2019-153250 uses a recurrent neural network trained to generate sentences from input characters to create sentences to be included in the radiology report.

[0004] For example, Japanese Patent Application Laid-Open No. 2017-021648 discloses a method for accepting a selection of a sentence from an inputter, searching a report database based on the selected sentence, and extracting the sentence following the selected sentence. For example, Japanese Patent Application Laid-Open No. 2016-038726 discloses a method for analyzing an image interpretation report being input and creating candidates for correction information to be used to correct the image interpretation report.

[0005] In medical image interpretation, after interpreting a region of interest, interpretation of another related region of interest may also be performed. For example, if a lesion is discovered in the lung field during interpretation, it is common to check whether there are any related lesions in other areas, such as the mediastinal lymph nodes and the liver. Therefore, there is a need for a technology that can support interpretation of another region of interest related to the region of interest that has already been interpreted.

[0006] The present disclosure provides an information processing device, an information processing method, and an information processing program that can support the creation of an interpretation report.

[0007] A first aspect of the present disclosure is an information processing device in which a processor acquires a character string including a description related to a first region of interest, identifies a second region of interest that is not included in the character string and is related to the first region of interest, and notifies a user to confirm whether or not to display an image that may include the second region of interest.

[0008] A second aspect of the present disclosure is that, in the above-mentioned first aspect, the processor may identify a second region of interest related to the first region of interest based on correlation data in which the degree of relevance to other regions of interest is predetermined for each type of region of interest.

[0009] A third aspect of the present disclosure is that in the second aspect, the correlation data may be determined based on a co-occurrence indicating the probability that two different types of regions of interest appear simultaneously in a string describing an image.

[0010] A fourth aspect of the present disclosure may be such that, in any one of the first to third aspects, the processor displays, as the notification, on the display, a character string indicating the second region of interest, and at least one of a symbol and a graphic.

[0011] A fifth aspect of the present disclosure is any one of the first to fourth aspects, wherein the processor may cause a display to display an image that may include the second region of interest.

[0012] According to a sixth aspect of the present disclosure, in the fifth aspect, when a second region of interest is included in the image, the processor may highlight the second region of interest.

[0013] According to a seventh aspect of the present disclosure, in the fifth or sixth aspect, the processor may, when instructed, cause the display to display an image that may include the second region of interest.

[0014] In an eighth aspect of the present disclosure, in any one of the first to seventh aspects, the processor may generate a character string including a description of the second region of interest, and cause the character string to be displayed on the display.

[0015] A ninth aspect of the present disclosure is that, in the eighth aspect, the processor may acquire an image that may include the second region of interest, and generate a string of characters including a description of the second region of interest based on the acquired image.

[0016] A tenth aspect of the present disclosure is the eighth or ninth aspect, wherein the processor generates a plurality of candidate character strings including a description related to the second region of interest, displays the plurality of candidate character strings on a display, and accepts selection of at least one of the plurality of candidate character strings.

[0017] An eleventh aspect of the present disclosure is any one of the first to tenth aspects, wherein when the processor identifies multiple second areas of interest related to the first area of ​​interest, the processor may notify the second areas of interest in an order according to the priority of the second areas of interest.

[0018] In a twelfth aspect of the present disclosure, in the eleventh aspect, the priority of the second region of interest may be determined according to a degree of association with the first region of interest.

[0019] A thirteenth aspect of the present disclosure is the eleventh or twelfth aspect, wherein the priority of the second region of interest may be determined according to findings of the second region of interest diagnosed based on the image.

[0020] A fourteenth aspect of the present disclosure is any one of the first to thirteenth aspects, wherein the processor may identify a finding of a first region of interest described in the character string, and identify a second region of interest related to the finding of the first region of interest.

[0021] A fifteenth aspect of the present disclosure is any one of the first to fourteenth aspects, wherein the image is a medical image, and the first region of interest and the second region of interest may each be at least one of a region of a structure that may be included in the medical image and a region of an abnormal shadow that may be included in the medical image.

[0022] A sixteenth aspect of the present disclosure is an information processing method that includes a process of obtaining a character string including a description related to a first region of interest, identifying a second region of interest that is not included in the character string and is related to the first region of interest, and notifying a user whether or not to display an image that may include the second region of interest.

[0023] A seventeenth aspect of the present disclosure is an information processing program that causes a computer to execute a process of acquiring a character string including a description related to a first region of interest, identifying a second region of interest that is not included in the character string and is related to the first region of interest, and notifying a user whether or not to display an image that may include the second region of interest.

[0024] According to the above aspects, the information processing device, the information processing method, and the information processing program of the present disclosure can assist in creating an interpretation report.

[0025] 1 is a diagram illustrating an example of a schematic configuration of an information processing system. 2 is a diagram illustrating an example of a medical image. 3 is a diagram illustrating an example of a medical image. 4 is a block diagram illustrating an example of a hardware configuration of an information processing device. 5 is a block diagram illustrating an example of a functional configuration of an information processing device. 6 is a diagram illustrating an example of a screen displayed on a display. 7 is a diagram illustrating an example of a screen displayed on a display. 8 is a diagram illustrating an example of a screen displayed on a display. 9 is a flowchart illustrating an example of information processing. 10 is a diagram illustrating an example of a screen displayed on a display.

[0026] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. First, the configuration of an information processing system 1 to which an information processing device of the present disclosure is applied will be described. FIG. 1 is a diagram showing a schematic configuration of the information processing system 1. The information processing system 1 shown in FIG. 1 captures an examination target region of a subject and stores the medical images acquired by the capture, based on an examination order from a doctor of a medical department using a known ordering system. The information processing system 1 also allows a radiologist to interpret the medical images and create an interpretation report, and allows the doctor of the requesting medical department to view the interpretation report.

[0027] 1, the information processing system 1 includes an imaging device 2, an interpretation WS (Work Station) 3 which is an interpretation terminal, a medical treatment WS 4, an image server 5, an image DB (Data Base) 6, a report server 7, and a report DB 8. The imaging device 2, the interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8 are connected via a wired or wireless network 9 in a state in which they can communicate with each other.

[0028] Each device is a computer installed with an application program that causes the device to function as a component of the information processing system 1. The application program may be recorded on a recording medium such as a DVD-ROM (Digital Versatile Disc Read Only Memory) or a CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on the computer from the recording medium. Alternatively, the application program may be stored in a state accessible from the outside in a storage device or network storage of a server computer connected to the network 9, and downloaded to the computer upon request and installed.

[0029] The imaging device 2 is a device (modality) that captures an image of a diagnostic target region of a subject, thereby generating a medical image T representing the diagnostic target region. Examples of the imaging device 2 include a plain X-ray imaging device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, an ultrasound diagnostic device, an endoscope, and a fundus camera. The medical image generated by the imaging device 2 is transmitted to an image server 5 and stored in an image DB 6.

[0030] The image interpretation WS 3 is a computer used by medical professionals, such as radiologists, to interpret medical images and prepare image interpretation reports, and includes the information processing device 10 according to this embodiment. The image interpretation WS 3 issues requests for viewing medical images to the image server 5, performs various image processing on medical images received from the image server 5, displays the medical images, and accepts input of text related to the medical images. The image interpretation WS 3 also performs analysis processing on medical images, supports the preparation of image interpretation reports based on the analysis results, requests the report server 7 to register and view image interpretation reports, and displays image interpretation reports received from the report server 7. These processes are performed by the image interpretation WS 3 by executing software programs for each process.

[0031] The clinical WS 4 is a computer used by medical personnel such as doctors in the clinical department to perform detailed observations of medical images, view interpretation reports, and create electronic medical records, and is composed of a processing device, a display device such as a monitor, and input devices such as a keyboard and a mouse. The clinical WS 4 sends requests to the image server 5 to view medical images, displays medical images received from the image server 5, requests to the report server 7 to view interpretation reports, and displays interpretation reports received from the report server 7. These processes are performed by the clinical WS 4 executing software programs for each process.

[0032] The image server 5 is a general-purpose computer installed with a software program that provides the functions of a database management system (DBMS). The image server 5 is connected to an image DB 6. The connection between the image server 5 and the image DB 6 is not particularly limited, and may be via a data bus or a network such as a NAS (Network Attached Storage) or a SAN (Storage Area Network).

[0033] The image DB 6 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. Medical images acquired by the imaging device 2 and accompanying information attached to the medical images are registered in the image DB 6 in association with each other.

[0034] The incidental information may include, for example, identification information such as an image ID (identification) for identifying the medical image, a cross-sectional ID assigned to each cross-sectional image included in the medical image, a subject ID for identifying the subject, and an examination ID for identifying the examination. The incidental information may also include, for example, information related to the medical image capture, such as the imaging method, imaging conditions, and imaging date and time. Examples of the "imaging method" and "imaging conditions" include the type of imaging device 2, the imaging site, the imaging protocol, the imaging sequence, the imaging technique, whether or not a contrast agent was used, and the slice thickness in the tomographic imaging. The incidental information may also include information related to the subject, such as the subject's name, date of birth, age, and gender. The incidental information may also include information related to the purpose of capturing the medical image.

[0035] Furthermore, when the image server 5 receives a request to register a medical image from the imaging device 2, it converts the medical image into a database format and registers it in the image DB 6. Furthermore, when the image server 5 receives a request to view from the image interpretation WS 3 and the medical treatment WS 4, it searches for medical images registered in the image DB 6 and transmits the searched medical images to the image interpretation WS 3 and the medical treatment WS 4 that have issued the request to view.

[0036] The report server 7 is a general-purpose computer installed with a software program that provides the functions of a database management system. The report server 7 is connected to a report DB 8. The connection between the report server 7 and the report DB 8 is not particularly limited, and may be via a data bus or a network such as a NAS or SAN.

[0037] The report DB 8 is realized by a storage medium such as a HDD, SSD, or flash memory. The report DB 8 stores the interpretation reports created by the interpretation WS 3. The report DB 8 may also store information on findings (described in detail below) related to medical images acquired by the interpretation WS 3.

[0038] Furthermore, when the report server 7 receives a request to register an interpretation report from the image interpretation WS 3, it converts the interpretation report into a database format and registers it in the report DB 8. Furthermore, when the report server 7 receives a request to view an interpretation report from the image interpretation WS 3 and the medical treatment WS 4, it searches for the interpretation report registered in the report DB 8 and transmits the retrieved interpretation report to the image interpretation WS 3 and the medical treatment WS 4 that have made the view request.

[0039] The network 9 is, for example, a network such as a LAN (Local Area Network) or a WAN (Wide Area Network). The imaging device 2, the image interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8 included in the information processing system 1 may be located in the same medical institution or in different medical institutions. The number of devices, namely the imaging device 2, the image interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8, is not limited to the number shown in FIG. 1 , and each device may be composed of multiple devices having similar functions.

[0040] Fig. 2 is a diagram schematically illustrating an example of a medical image acquired by the imaging device 2. The medical image T shown in Fig. 2 is, for example, a CT image consisting of multiple tomographic images T1 to Tm (m is 2 or more), each representing a cross-sectional plane from the head to the waist of a single subject (human body). The medical image T is an example of an image disclosed herein.

[0041] FIG. 3 is a schematic diagram illustrating an example of one tomographic image Tx among the multiple tomographic images T1 to Tm. The tomographic image Tx illustrated in FIG. 3 represents a tomographic plane including the lungs. Each tomographic image T1 to Tm may include a structure region SA representing various organs and tissues of the human body (e.g., lungs and liver) and various tissues constituting the various organs and tissues (e.g., blood vessels, nerves, muscles, etc.). Each tomographic image may also include an abnormal shadow region AA representing a lesion such as a nodule, tumor, injury, defect, or inflammation. In the tomographic image Tx illustrated in FIG. 3, the lung region is the structure region SA, and the nodule region is the abnormal shadow region AA. Note that a single tomographic image may include multiple structure regions SA and / or abnormal shadow regions AA. Hereinafter, at least one of the structure region SA and the abnormal shadow region AA will be referred to as a "region of interest."

[0042] In interpreting medical images, after interpreting a region of interest, interpretation of another related region of interest may also be performed. For example, if a lesion is discovered in the lung field during interpretation, it is common to check for related lesions in other areas, such as the mediastinal lymph nodes and the liver. Therefore, the information processing device 10 according to this embodiment has a function to support interpretation of another region of interest related to a region of interest that has already been interpreted (i.e., a region of interest that has already been described in the findings). The information processing device 10 will be described below. As described above, the information processing device 10 is included in the interpretation WS 3.

[0043] First, an example of the hardware configuration of an information processing device 10 according to this embodiment will be described with reference to Fig. 4. As shown in Fig. 4, the information processing device 10 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 serving as a temporary storage area. The information processing device 10 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a keyboard and a mouse, and a network I / F (Interface) 26. The network I / F 26 is connected to a network 9 for wired or wireless communication. The CPU 21, the storage unit 22, the memory 23, the display 24, the input unit 25, and the network I / F 26 are connected via a bus 28 such as a system bus and a control bus so as to be able to exchange various information with one another.

[0044] The storage unit 22 is realized by a storage medium such as an HDD, an SSD, or a flash memory. An information processing program 27 for the information processing device 10 is stored in the storage unit 22. The CPU 21 reads the information processing program 27 from the storage unit 22, loads it into the memory 23, and executes the loaded information processing program 27. The CPU 21 is an example of a processor of the present disclosure. For example, a personal computer, a server computer, a smartphone, a tablet terminal, a wearable terminal, or the like can be appropriately applied as the information processing device 10.

[0045] Next, an example of the functional configuration of the information processing device 10 according to this embodiment will be described with reference to Figures 5 to 9. As shown in Figure 5, the information processing device 10 includes an acquisition unit 30, a generation unit 32, an identification unit 34, and a control unit 36. The CPU 21 executes the information processing program 27, causing the CPU 21 to function as each of the functional units of the acquisition unit 30, the generation unit 32, the identification unit 34, and the control unit 36.

[0046] 6 to 9 are diagrams showing examples of screens D1 to D4 that are displayed on the display 24 by the control unit 36. The functions of the acquisition unit 30, generation unit 32, identification unit 34, and control unit 36 ​​will be described below with reference to FIGS.

[0047] (Interpretation of the First Region of Interest) First, the interpretation of the first region of interest A1 will be described with reference to screen D1 in Fig. 6. The acquisition unit 30 acquires a medical image (hereinafter referred to as "first image TF") including the first region of interest A1 from the image server 5. The first image TF is displayed on screen D1 under conditions suitable for interpretation of the lung field. The first region of interest A1 is at least one of a region of a structure that may be included in the first image TF and a region of an abnormal shadow that may be included in the first image TF.

[0048] The acquiring unit 30 also acquires finding information for the first region of interest A1. Screen D1 shows, as an example, finding information 62 for the first region of interest A1 being a nodule. The finding information includes information indicating various findings, such as the name (type), characteristics, position, measurement values, and suspected disease name.

[0049] Examples of names (types) include the names of structures such as "lung" and "liver," and the names of abnormal shadows such as "nodule." Characteristics primarily refer to the characteristics of abnormal shadows. For example, in the case of pulmonary nodules, these include absorption values ​​such as "solid" and "ground-glass," margin shapes such as "clear / indistinct," "smooth / irregular," "spicule," "lobulated," and "serrated," and findings indicating the overall shape such as "near-circular" and "irregular." Other examples include the relationship with surrounding tissues such as "pleural contact" and "pleural indentation," and findings regarding the presence or absence of contrast enhancement and washout.

[0050] The term "location" refers to the anatomical location, the location in the medical image, and the relative positional relationship with other regions of interest, such as "inside," "periphery," and "surroundings." The anatomical location may be indicated by organ names, such as "lung" and "liver," or by subdivisions of the lung, such as "right lung," "upper lobe," and "apical segment (S1)." The measurement value is a value that can be quantitatively measured from the medical image, such as at least one of the size and signal value of the region of interest. The size is expressed, for example, as the major axis, minor axis, area, and volume of the region of interest. The signal value is expressed, for example, as the pixel value of the region of interest and the CT value in units of HU. The estimated disease name is an evaluation result estimated based on the abnormal shadow, such as the disease name, "cancer," "inflammation," and the like, as well as evaluation results regarding the disease name and characteristics, such as "negative / positive," "benign / malignant," and "mild / severe."

[0051] Specifically, the acquisition unit 30 may extract a first region of interest A1 from the acquired first image TF and acquire the finding information by performing image analysis on the first region of interest A1. Note that methods using known CAD technology and AI (Artificial Intelligence) technology, etc., may be appropriately applied as a method for extracting the first region of interest A1 from the first image TF. For example, the acquisition unit 30 may extract the first region of interest A1 from the first image TF using a learning model, such as a convolutional neural network (CNN), that receives a medical image as input and is trained to extract and output a region of interest included in the medical image. Also, methods using known CAD technology and AI technology may be appropriately applied as a method for acquiring the finding information by image analysis. For example, the acquisition unit 30 may receive a region of interest extracted from a medical image as input and acquire the finding information of the first region of interest A1 using a learning model, such as a CNN, that is trained in advance to output finding information about the region of interest.

[0052] The acquiring unit 30 also queries the report server 7 to determine whether or not an interpretation report (hereinafter referred to as a "past report") created for the first region of interest A1 at a previous time point is registered in the report DB 8. For example, there is a case where medical images of the same lesion in the same subject are captured and interpreted multiple times for follow-up observation. In this case, since a past report has already been registered in the report DB 8, the acquiring unit 30 acquires the past report from the report server 7. In this case, the acquiring unit 30 also acquires from the image server 5 a medical image (hereinafter referred to as a "past image") that includes the first region of interest A1 captured at a previous time point.

[0053] As shown in screen D1, the control unit 36 ​​controls the display 24 to display the first image TF acquired by the acquisition unit 30 and its finding information 62. The control unit 36 ​​may also highlight the first region of interest A1 in the first image TF. For example, as shown in screen D1, the control unit 36 ​​may surround the first region of interest A1 in the first image TF with a bounding box 90. For example, the control unit 36 ​​may also add a marker such as an arrow near the first region of interest A1 in the first image TF, color-code the first region of interest A1 from other regions, or enlarge and display the first region of interest A1.

[0054] Furthermore, when a mouse pointer 92 operated by the user via the input unit 25 is placed over the first region of interest A1 on the screen D1 (so-called mouse hover / mouse over), the control unit 36 ​​may perform control to display a past report on the display 24. In the example of Fig. 6, the mouse pointer 92 is placed over the first region of interest A1, and a past report for the first region of interest A1 is displayed on a pop-up screen D1A.

[0055] Furthermore, when the first region of interest A1 is selected on screen D1 by mouse pointer 92 (for example, when an operation such as a click, double-click, or drag is received), control unit 36 ​​may accept various operations regarding first region of interest A1. Fig. 7 shows an example of screen D2, which is transitioned to when first region of interest A1 is selected on screen D1. Screen D2 displays menu D1B for accepting various operations regarding first region of interest A1. When "Display Previous Images" is selected on menu D1B, control unit 36 ​​performs control to display previous images acquired by acquisition unit 30 on display 24 (not shown).

[0056] When "Create a finding" is selected in the menu D1B, the generator 32 generates a finding regarding the first region of interest A1. Fig. 8 shows an example of a screen D3 to which the screen transitions when "Create a finding" is selected in the menu D1B of Fig. 7. The screen D3 displays a finding 64 regarding the first region of interest A1 generated by the generator 32.

[0057] Specifically, the generator 32 generates a finding statement including finding information 62 related to the first region of interest A1 acquired by the acquirer 30. For example, the generator 32 may generate the finding statement using a machine learning technique such as a recurrent neural network described in Japanese Patent Application Laid-Open No. 2019-153250. Furthermore, for example, the generator 32 may generate the finding statement by embedding the finding information 62 in a predetermined template. Furthermore, the generator 32 may accept user corrections to the generated finding statement.

[0058] If the acquisition unit 30 makes an inquiry to the report server 7 and finds that the past report is not registered in the report DB 8 , the control unit 36 ​​omits displaying the past report and past image on the display 24 .

[0059] (Confirming Whether or Not to Display the Second Region of Interest) After the interpretation of the first region of interest A1 is completed as described above, each functional unit confirms with the user whether or not to display another second region of interest A2 related to the first region of interest A1, i.e., whether or not the user also intends to interpret the second region of interest A2. The second region of interest A2 is at least one of a region of a structure that may be included in the medical image and a region of an abnormal shadow that may be included in the medical image. Note that the medical image that may include the second region of interest A2 may be an image obtained by capturing the same subject as the subject captured in the first image TF that may include the first region of interest A1, and may be the same image as the first image TF or a different image. The following description will be given of an example in which the second region of interest A2 is included in a second image TS that is different from the first image TF.

[0060] Specifically, the acquiring unit 30 acquires a finding statement including a description related to the first region of interest A1 generated by the generating unit 32. The identifying unit 34 identifies a second region of interest A2 that is not described in the finding statement acquired by the acquiring unit 30 and is related to the first region of interest A1. For example, the identifying unit 34 identifies mediastinal lymph node enlargement as the second region of interest A2 that is not described in the finding statement 64 in FIG. 8 and is related to the first region of interest A1 (nodule) described in the finding statement 64.

[0061] Specifically, the identification unit 34 identifies the second region of interest A2 related to the first region of interest A1 based on correlation data in which the degree of association with other regions of interest is predetermined for each type of region of interest. For example, the correlation data may be determined based on a co-occurrence rate indicating the probability that two different types of regions of interest simultaneously appear in a character string (e.g., a finding sentence) described in a medical image. For example, the identification unit 34 may create correlation data indicating that the degree of association between "nodule" and "mediastinal lymph node enlargement" is relatively high when the number and / or proportion of finding sentences containing "mediastinal lymph node enlargement" among multiple finding sentences containing "nodule" registered in the report DB 8 is equal to or greater than a threshold. The correlation data may be created in advance and stored in the storage unit 22 or the like, or may be created each time the second region of interest A2 is identified. The correlation data may be created not only by the identification unit 34 but also by an external device or the like.

[0062] Furthermore, for example, the correlation data may be determined based on guidelines, manuals, etc. that define structures and / or lesions that should be simultaneously confirmed. In this case, the correlation data may be manually created by a user.

[0063] Furthermore, the identification unit 34 identifies and acquires a second image TS that may include the second region of interest A2 from among the medical images registered in the image server 5. For example, when the identification unit 34 identifies mediastinal lymph node enlargement as the second region of interest A2, the identification unit 34 identifies a medical image representing a tomographic plane that includes the mediastinal lymph node enlargement as the second image TS (see FIG. 9 ).

[0064] The second image TS may include the second region of interest A2, but does not necessarily include the second region of interest A2. For example, even if a nodule is found in a lung field, swelling of the mediastinal lymph nodes does not necessarily occur. In this case, the identification unit 34 may identify, as the second image TS, a medical image representing a tomographic plane including mediastinal lymph nodes that are not swollen.

[0065] The control unit 36 ​​notifies the user to confirm whether or not to display the second image TS, which may include the second region of interest A2 identified by the identification unit 34. This notification allows the user to recognize the existence of the second region of interest A2 and to decide whether or not to interpret the second image TS.

[0066] 8 displays a notification 94 for confirming whether the user wants to check the mediastinal lymph node enlargement identified as the second region of interest A2 (i.e., whether the control unit 36 ​​wants to display the second image TS that may include mediastinal lymph node enlargement on the display 24). Screen D3 also displays an icon 96 for making the notification 94 stand out. In this manner, the control unit 36 ​​may perform control to display, as the notification, at least one of a character string (notification 94) indicating the second region of interest A2, and a symbol and / or graphic (icon 96). For example, the control unit 36 ​​may also perform the notification by means of sound output from a speaker, blinking of a light source such as a light bulb or an LED (Light Emitting Diode), or the like.

[0067] (Interpretation of Second Region of Interest) After providing the user with the notification to confirm whether or not the second image TS needs to be displayed as described above, the control unit 36 ​​may control the display 24 to display the second image TS, which may include the second region of interest A2. Specifically, the control unit 36 ​​may control the display 24 to display the second image TS when instructed by the user. For example, the control unit 36 ​​may cause the display 24 to display the second image TS when the notification 94 is selected with the mouse pointer 92 on the screen D3 (e.g., when an operation such as a click or double-click is received). Figure 9 shows an example of a screen D4 to which the screen transitions when the notification 94 is selected on the screen D3 of Figure 8. The second image TS is displayed on the screen D4.

[0068] Similarly to the interpretation of the first region of interest A1, each functional unit may interpret the second region of interest A2. Below, the functions of each functional unit related to the interpretation of the second region of interest A2 will be described, but some of the functions similar to those for the interpretation of the first region of interest A1 will not be described.

[0069] The acquiring unit 30 acquires the finding information for the second region of interest A2. Specifically, the acquiring unit 30 may acquire the finding information by extracting the second region of interest A2 from the second image TS and performing image analysis on the second region of interest A2. As an example, the screen D4 shows finding information 62 when the second region of interest A2 is lymph node enlargement.

[0070] The acquisition unit 30 also queries the report server 7 to determine whether a radiology report created in the past for the second region of interest A2 has been registered in the report DB 8, and acquires the report if it has already been registered. In this case, the acquisition unit 30 also acquires from the image server 5 a medical image captured in the past that includes the second region of interest A2.

[0071] As shown on screen D4, the control unit 36 ​​controls the display 24 to display finding information 62 about the second region of interest A2 acquired by the acquisition unit 30. Furthermore, when the acquisition unit 30 analyzes that the second image TS includes the second region of interest A2, the control unit 36 ​​may highlight the second region of interest A2 in the second image TS. For example, as shown on screen D4, the control unit 36 ​​may surround the second region of interest A2 in the second image TS with a bounding box 90.

[0072] The control unit 36 ​​may also cause the display 24 to display an interpretation report (not shown) created for the second region of interest A2 at a previous time point, which was acquired by the acquisition unit 30. The control unit 36 ​​may also perform control to cause the display 24 to display a medical image (not shown) acquired by the acquisition unit 30 that includes the second region of interest A2 and was captured at a previous time point.

[0073] The generator 32 may generate a finding statement including a description related to the second region of interest A2. Specifically, the generator 32 may generate a finding statement including finding information related to the second region of interest A2 acquired by the acquirer 30 based on the second image TS. That is, the generator 32 may generate a finding statement including a description related to the second region of interest A2 based on the acquired second image TS.

[0074] The control unit 36 ​​controls the display 24 to display the finding statement including a description related to the second region of interest A2 generated by the generation unit 32. Screen D4 in Fig. 9 shows, as an example of a finding statement generated by the generation unit 32, a finding statement 64 in which a finding statement related to the second region of interest A2 (mediastinal lymph node enlargement) is added to the finding statement related to the first region of interest A1 (nodule) in Fig. 8.

[0075] (Interpretation of Multiple Second Regions of Interest) The number of second regions of interest A2 associated with the first region of interest A1 is not limited to one. For example, in addition to mediastinal lymph node enlargement, liver metastasis may also be confirmed as another lesion associated with the lung nodule. Therefore, the identification unit 34 may identify multiple second regions of interest A2 that are not described in the findings acquired by the acquisition unit 30 and are associated with the first region of interest A1. In this case, each functional unit may perform an interpretation of the respective second regions of interest A2.

[0076] After completing the interpretation of the second image TS that may include a certain second region of interest A2, the control unit 36 ​​may issue a notification to the user to confirm whether or not to display a medical image that may include another second region of interest A2. On the screen D4 in Fig. 9, a notification 94 is displayed on the screen D4 for interpreting mediastinal lymph node enlargement, prompting the user to confirm whether or not to display a medical image that may include liver metastasis.

[0077] Furthermore, the control unit 36 ​​may issue notifications in an order according to the priorities of the multiple second regions of interest A2 identified by the identification unit 34. That is, the control unit 36 ​​may issue notifications to the user to confirm whether or not to display medical images that may include each second region of interest A2 in an order according to the priorities of each second region of interest A2. For example, suppose that mediastinal lymph node enlargement and liver metastasis are identified as second regions of interest A2 related to lung nodules, and that mediastinal lymph node enlargement has a higher priority. In this case, the control unit 36 ​​may issue a notification to the user to confirm whether or not to display medical images that may include mediastinal lymph node enlargement before issuing a notification to the user to confirm whether or not to display medical images that may include liver metastasis (i.e., when interpretation of the nodule is completed).

[0078] The priority of each second region of interest A2 may be determined, for example, according to the degree of association with the first region of interest A1. The degree of association between the first region of interest A1 and the second region of interest A2 may be determined, for example, using correlation data in which the degree of association with other regions of interest is predetermined for each type of region of interest.

[0079] Furthermore, for example, the priority of each second region of interest A2 may be determined according to the findings of the second region of interest A2 diagnosed based on the medical image. For example, the control unit 36 ​​may estimate the severity of the condition of each second region of interest A2 based on the finding information for each second region of interest A2 acquired by the acquisition unit 30, and may notify the patients in order of severity.

[0080] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Fig. 10. In the information processing device 10, the CPU 21 executes the information processing program 27, thereby executing the information processing shown in Fig. 10. The information processing is executed, for example, when a user issues an instruction to start execution via the input unit 25.

[0081] In step S10, the acquiring unit 30 acquires a statement of findings that includes a description of the first region of interest A1. In step S12, the identifying unit 34 identifies a second region of interest A2 that is not included in the statement of findings acquired in step S10 and is related to the first region of interest A1. In step S14, the control unit 36 ​​notifies the user to confirm whether or not to display a second image TS that may include the second region of interest A2 identified in step S12.

[0082] In step S16, the control unit 36 ​​receives an instruction (display instruction) to display the second image TS on the display 24. That is, the user who confirmed the notification in step S14 inputs an instruction to display the second image TS as necessary. If the display instruction is received (Y in step S16), the process proceeds to step S18, where the control unit 36 ​​performs control to display the second image TS on the display 24.

[0083] In step S20, the control unit 36 ​​receives an instruction to generate a finding statement including a description related to the second region of interest A2 (a finding statement generation instruction). That is, after checking the second image TS displayed on the display 24 in step S18, the user inputs an instruction to generate a finding statement related to the second region of interest A2 as needed. If the instruction to generate a finding statement is received (Y in step S20), the process proceeds to step S22, where the generation unit 32 generates a finding statement including a description related to the second region of interest A2. In step S24, the control unit 36 ​​displays the finding statement related to the second region of interest A2 generated in step S22 on the display 24, and the information processing ends.

[0084] On the other hand, if the display instruction is not received (N in step S16), the second image TS is not displayed and the information processing ends. Also, if the instruction to generate a finding statement is not received (N in step S20), the finding statement is not generated and the information processing ends.

[0085] As described above, an information processing device 10 according to one aspect of the present disclosure includes at least one processor, which acquires a character string including a description related to a first region of interest, identifies a second region of interest that does not include the description in the character string and is related to the first region of interest, and notifies the user to confirm whether or not to display an image that may include the second region of interest.

[0086] That is, the information processing device 10 according to this embodiment allows the user to confirm, based on the findings obtained by interpreting the first region of interest A1, whether or not to display the second image TS, which may include a second region of interest A2 related to the first region of interest A1. This allows for smooth interpretation of both the first region of interest A1 described in the findings and the second region of interest A2 not described in the findings. Furthermore, the notification allows the user to be made aware of the existence of the second region of interest A2, thereby preventing the second region of interest A2 from being overlooked. This can therefore assist in the creation of an interpretation report.

[0087] In the above embodiment, the acquiring unit 30 acquires the finding information of the first region of interest A1 and the second region of interest A2 by performing image analysis on the medical image. However, the present invention is not limited to this. For example, the acquiring unit 30 may acquire the finding information stored in advance in the storage unit 22, the image server 5, the image DB 6, the report server 7, the report DB 8, or other external devices. Alternatively, for example, the acquiring unit 30 may acquire the finding information manually input by the user via the input unit 25.

[0088] In the above embodiment, the generator 32 generates one finding statement including a description about the second region of interest A2 based on the second image TS. However, the present invention is not limited to this. For example, the generator 32 may acquire a finding statement including a description about the second region of interest A2 that is stored in advance in the report DB 8, the storage unit 22, or another external device, without relying on the second image TS. For another example, the generator 32 may receive a finding statement manually input by the user.

[0089] Furthermore, for example, the generator 32 may generate a plurality of candidate finding sentences including descriptions related to the second region of interest A2. Fig. 11 shows an example of a screen D5 displayed on the display 24 by the controller 36, in which a plurality of candidate finding sentences 641 to 643 related to the second region of interest A2 (mediastinal lymph node enlargement) are displayed. As shown in Fig. 11, the controller 36 may cause the display 24 to display the plurality of candidate finding sentences 641 to 643 generated by the generator 32. Furthermore, the controller 36 may accept selection of at least one of the plurality of candidate finding sentences 641 to 643.

[0090] In the above embodiment, the second region of interest A2 related to the first region of interest A1 is identified. However, this is not limiting. For example, even for lung nodules, if the characteristics are different, such as "solid type" and "ground glass type," the associated second region of interest A2 may be different. Therefore, the identification unit 34 may identify the findings of the first region of interest A1 described in the finding statement including a description of the first region of interest A1, and identify the second region of interest A2 related to the findings of the first region of interest A1.

[0091] In the above embodiment, various processes are performed using the finding sentences, but the present invention is not limited to this. For example, instead of finding sentences, various processes may be performed using documents such as radiology reports, sentences including multiple finding sentences, and various character strings such as finding information and words included in finding sentences.

[0092] In the above embodiment, a nodule (i.e., a region of abnormal shadow) is used as an example of the first region of interest A1, and mediastinal lymph node enlargement (i.e., a region of abnormal shadow) is used as an example of the second region of interest A2, but this is not limiting. As described above, the first region of interest A1 and the second region of interest A2 may each be at least one of a region of a structure that may be included in a medical image and a region of abnormal shadow that may be included in a medical image, and the combination thereof is free.

[0093] For example, the first region of interest A1 may be a lung (i.e., a region of a structure) and the second region of interest A2 may be a mediastinal lymph node (i.e., a region of a structure). Alternatively, the first region of interest A1 may be a lung (i.e., a region of a structure) and the second region of interest A2 may be an enlarged mediastinal lymph node (i.e., a region of an abnormal shadow). Alternatively, the first region of interest A1 may be a nodule (i.e., a region of an abnormal shadow) and the second region of interest A2 may be a mediastinal lymph node (i.e., a region of a structure).

[0094] In the above embodiment, a radiology report on a medical image has been described, but the present disclosure is not limited thereto. The information processing device 10 of the present disclosure can be applied to various documents that include descriptions of images obtained by photographing a subject. For example, the information processing device 10 may be applied to documents that include descriptions of images obtained using equipment, buildings, pipes, welds, and the like as subjects in non-destructive testing such as radiographic testing and ultrasonic flaw detection testing.

[0095] Furthermore, in the above embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition unit 30, the generation unit 32, the identification unit 34, and the control unit 36. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0096] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0097] Examples of configuring multiple processing units with a single processor include: first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units; second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs); and thus, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0098] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0099] In the above embodiment, the information processing program 27 is pre-stored (installed) in the storage unit 22, but this is not limiting. The information processing program 27 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 27 may also be downloaded from an external device via a network. Furthermore, the technology disclosed herein extends to not only information processing programs but also storage media that non-temporarily store information processing programs.

[0100] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other parts from the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure.

[0101] The disclosure of Japanese Patent Application No. 2022-065906, filed on April 12, 2022, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. At least one processor; The processor, Obtaining a string that includes a description of the first region of interest; identifying a second region of interest that is not included in the character string and is related to the first region of interest; A notification is given to a user to confirm whether or not an image that may include the second region of interest needs to be displayed. Information processing device.

2. The processor, The second region of interest related to the first region of interest is identified based on correlation data in which the degree of association with other regions of interest is predetermined for each type of region of interest. The information processing device according to claim 1 .

3. The correlation data is determined based on a co-occurrence degree indicating a probability that two different types of the regions of interest appear simultaneously in a character string describing an image. The information processing device according to claim 2 .

4. The processor, As the notification, at least one of a character string, a symbol, and a graphic representing the second region of interest is displayed on a display.

3. The information processing device according to claim 1 or 2.

5. The processor, Displaying the image on a display, the image including the second region of interest.

3. The information processing device according to claim 1 or 2.

6. The processor, If the image includes the second region of interest, the second region of interest is highlighted. The information processing device according to claim 5 .

7. The processor, When instructed, displaying on the display the image that may include the second region of interest. The information processing device according to claim 5 .

8. The processor, generating a string including a description of the second region of interest; Display the character string on the display 3. The information processing device according to claim 1 or 2.

9. The processor, acquiring the image, which may include the second region of interest; generating a character string including a description of the second region of interest based on the acquired image; The information processing device according to claim 8.

10. The processor, generating a plurality of character string candidates including a description of the second region of interest; Displaying a plurality of candidates for the character string on a display; Accepting a selection of at least one of the plurality of character string candidates The information processing device according to claim 8.

11. The processor, When a plurality of the second regions of interest related to the first region of interest are identified, the notification is performed in an order according to the priority of the second regions of interest.

3. The information processing device according to claim 1 or 2.

12. The priority of the second region of interest is determined according to the degree of association with the first region of interest. The information processing device according to claim 11.

13. The priority of the second region of interest is determined according to a finding of the second region of interest diagnosed based on the image. The information processing device according to claim 11.

14. The processor, Identifying a finding of the first region of interest described in the character string; Identifying the second region of interest related to the findings of the first region of interest.

3. The information processing device according to claim 1 or 2.

15. the image is a medical image, The first region of interest and the second region of interest are at least one of a region of a structure that may be included in the medical image and a region of an abnormal shadow that may be included in the medical image.

3. The information processing device according to claim 1 or 2.

16. Obtaining a string that includes a description of the first region of interest; identifying a second region of interest that is not included in the character string and is related to the first region of interest; A notification is given to a user to confirm whether or not an image that may include the second region of interest needs to be displayed. An information processing method in which processing is performed by a computer.

17. Obtaining a string that includes a description of the first region of interest; identifying a second region of interest that is not included in the character string and is related to the first region of interest; A notification is given to a user to confirm whether or not an image that may include the second region of interest needs to be displayed. An information processing program for causing a computer to execute processing.