Information processing device, method for operating information processing device and program

Abstract

To provide an information processing device for achieving visualization of each disease in a display of a plurality of disease regions detected from one medical image, a method for operating the information processing device and a program.SOLUTION: An information processing device acquires a certainty factor representing at least one of the presence / absence of a disease and the degree of the disease about each pixel of a medical image by using a multi-disease detection model for detecting a plurality of disease regions from the medical image, creates certainty factor distribution information for visualizing a distribution of certainty factors of each disease, acquires a threshold to be applied to the a certainty factor, and superimposes at least one of a first color and a first opacity on the medical image for pixels of a first image having the certainty factor being the same value as the threshold about any disease to display the medical image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an operating method for an information processing device, and a program. [Background technology]

[0002] In recent years, AI technology has been used to assist in diagnosis of medical images. Plain X-rays can detect chest diseases such as nodules, pneumothorax, and pleural effusion. A heat map showing the distribution of confidence levels for disease regions detected as diseases from the The information, which uses a rectangle that encloses the disease area, is generated and displayed superimposed on the medical image. AI is an abbreviation for Artificial Intelligence.

[0003] In Patent Document 1, when multiple types of detection areas are displayed at once, the visibility of each detection area is low. The task of prioritizing disease detection areas based on predefined conditions is to The medical image processing device determines the priority of the detected area and changes the display form of the detected area according to the priority. will be done. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-029387 Summary of the Invention [Problem to be solved by the invention]

[0005] The heatmap output of a disease detection model for a medical image is grasped as a distribution of confidence levels indicating the presence or absence of the target disease for each pixel of the input image. Generally, when there is a single target disease, a heatmap to which a pseudo-color corresponding to the confidence value is applied is superimposed and displayed on the medical image. Thereby, the suspicion of the disease is presented to the radiologist in an easy-to-understand manner.

[0006] However, when the heatmap output of the disease detection model is extended to a multi-disease detection model in which multiple diseases are detected, the following problems exist. When heatmaps are displayed for each disease, the number of heatmaps to be read increases according to the number of target diseases, and there is concern about a decrease in the efficiency of reading.

[0007] When multiple diseases are integrated and displayed as a single heatmap, there is concern about a decrease in the readability of individual diseases when multiple diseases exist at the same position and when multiple diseases exist at adjacent positions.

[0008] The device described in Patent Document 1 has a high dependence on the performance of disease priority determination, and the priority of important diseases may be relatively lowered. Further, the device described in Patent Document 1 does not consider the problem that the visibility of each disease decreases when diseases overlap in the depth direction in a two-dimensional medical image.

[0009] The present invention has been made in view of such circumstances, and an object thereof is to provide an information processing device, an operation method of the information processing device, and a program that realize visualization for each detected disease in the display of a plurality of disease regions detected from a single medical image.

Means for Solving the Problems

[0010] The information processing apparatus according to the first aspect of the present disclosure includes one or more processors and one or more memories in which instructions to be executed by the one or more processors are stored. The one or more processors use a multi-disease detection model that acquires a medical image and detects a plurality of disease regions from the medical image, and for each pixel of the medical image, obtains, for each disease, a confidence level representing at least one of the presence or absence of a disease and the degree of the disease, creates confidence distribution information for visualizing the distribution of the confidence levels for each detected disease, obtains a threshold value applied to the confidence level, and performs control to superimpose at least one of a first color and a first opacity on the medical image and display it on a display device for pixels of a first image having a confidence level equal to the threshold value for any of the diseases.

[0011] According to the information processing apparatus according to the first aspect of the present disclosure, the pixels of the first image on which at least one of the first color and the first opacity is superimposed function as the contour of the disease region. Thereby, in the display of a plurality of disease regions detected from a single medical image, visualization for each detected disease is realized.

[0012] The disease region may include a state different from the region in a normal state. Examples of the disease region include regions including color, texture, unevenness, and the like.

[0013] As an example of the confidence distribution information, a heat map in which differences in confidence levels are expressed using mutually different colors can be mentioned. The difference in color may be a difference in hue or a difference in brightness.

[0014] The information processing apparatus according to the second aspect is the information processing apparatus according to the first aspect, wherein the one or more processors obtain a threshold range that is an upper limit value exceeding the threshold value, a pre-defined upper limit value, and a lower limit value less than the threshold value, a pre-defined lower limit value, and may classify pixels having a confidence level within the threshold range as pixels of the first image.

[0015] According to such an aspect, the pixels of the first image that function as the contour of the disease region are relatively increased. Thereby, the contour of the disease region is emphasized.

[0016] In the information processing apparatus according to the third aspect, in the information processing apparatus of the first aspect, one or more processors may classify pixels whose distance from pixels classified as the first image is within a specified range as pixels of the first image.

[0017] According to such an aspect, the number of pixels of the first image that function as the contour of the disease region is relatively increased. Thereby, the contour of the disease region is emphasized.

[0018] In the information processing apparatus according to the fourth aspect, in the information processing apparatus of any one of the first aspect to the third aspect, for any one of the diseases, one or more processors may perform control to superimpose and display at least one of a second color corresponding to the maximum value of the confidence level and a second opacity corresponding to the maximum value of the confidence level on the medical image for pixels of a second image whose confidence level exceeds the confidence level of the pixels of the first image.

[0019] According to such an aspect, at least one of the color and the opacity of the pixels of the second image is distinguished from the pixels of the first image. Thereby, the disease region for each disease is emphasized.

[0020] In the information processing apparatus according to the fifth aspect, in the information processing apparatus of the fourth aspect, when a pixel adjacent to a pixel of the second image is classified as a pixel of the first image, one or more processors may set a third opacity lower than the first opacity for the pixel of the second image.

[0021] According to such an aspect, the transmittance at the contour of the disease region is relatively decreased. Thereby, the contour of the disease region is emphasized.

[0022] In the information processing apparatus according to the sixth aspect, in the information processing apparatus of any one of the first aspect to the fifth aspect, for any one of the diseases, one or more processors may perform control to display the medical image with the color and opacity for pixels of a third image whose confidence level is less than the confidence level of the pixels of the first image as non-overlapped.

[0023] According to such an aspect, it does not interfere with the visual recognition of the medical image in the non-disease region.

[0024] The information processing apparatus according to the seventh aspect is the information processing apparatus according to any one of the first aspect to the sixth aspect. One or more processors acquire the maximum value of the confidence level for each disease, determine whether the maximum value of the confidence level for each disease exceeds a threshold value, and when the maximum value of the confidence level exceeds the threshold value, perform control to display text representing the disease name and the maximum value of the confidence level.

[0025] According to such an aspect, the user can grasp the maximum value of the confidence level for each disease.

[0026] The information processing apparatus according to the eighth aspect is the information image processing apparatus according to any one of the first aspect to the seventh aspect. One or more processors perform labeling processing on one or more connected regions including a plurality of consecutive pixels whose confidence level values for each disease exceed a threshold value, determine whether the maximum value of the confidence level for each connected region for each disease exceeds the threshold value, receive an input from the user designating a connected region, and perform control to display the maximum value of the confidence level for each disease in the designated connected region.

[0027] According to such an aspect, for each connected region designated by the user, the user can grasp the maximum value of the confidence level for each disease.

[0028] The information processing apparatus according to the ninth aspect is the information processing apparatus according to any one of the first aspect to the eighth aspect. One or more processors receive an input from the user designating a pixel in the medical image, and when the confidence level for each disease in the designated pixel exceeds the threshold value, perform control to display the confidence level value.

[0029] According to such an aspect, for each pixel designated due to the user's input, the user can grasp the maximum value of the confidence level for each disease.

[0030] The information processing apparatus according to the tenth aspect is the information processing apparatus according to any one of the first to ninth aspects, wherein one or more processors perform a labeling process on one or more connected regions including a plurality of consecutive pixels whose confidence value for each disease exceeds a threshold value, and for each connected region for each disease, control may be performed to display at least one of the disease name and the confidence value of the connected region.

[0031] According to such an aspect, for each connected region, the user can grasp the disease name and the confidence value for each disease.

[0032] The information processing apparatus according to the eleventh aspect is the information processing apparatus according to the tenth aspect, wherein one or more processors may perform control to display at least one of the disease name for each connected region and the confidence value of the connected region outside the detection target region in the medical image.

[0033] According to such an aspect, the character information does not interfere with the visual recognition of the medical image.

[0034] The information processing apparatus according to the twelfth aspect is the information processing apparatus according to any one of the first to eleventh aspects, wherein one or more processors perform a labeling process on one or more connected regions including a plurality of consecutive pixels whose confidence value for each disease exceeds a threshold value, and control may be performed to display all the confidence values for each disease for each connected region.

[0035] According to such an aspect, for each connected region, the user can grasp all the confidence values for each disease.

[0036] The information processing apparatus according to the thirteenth aspect is the information processing apparatus according to any one of the first to twelfth aspects, wherein one or more processors perform a labeling process on one or more connected regions including a plurality of consecutive pixels whose confidence value for each disease exceeds a threshold value, and for the connected regions for each disease, when the overlapping portion for each disease exceeds a specified value, control may be performed to display character information obtained by integrating a plurality of disease names in the overlapping portion.

[0037] According to such an aspect, for a plurality of disease regions overlapping each other, the visibility of character information can be relatively improved. Also, the character information does not interfere with the viewing of the medical image.

[0038] The information processing apparatus according to the 14th aspect is the information processing apparatus according to any one of the 1st to 13th aspects, wherein one or more processors may replace the value of the confidence of a pixel constituting the confidence distribution information with the maximum value of the confidence in a local region located within a specified range from a specified pixel.

[0039] According to such an aspect, the pixels in the foreground region of the medical image on which the confidence distribution information is superimposed do not become isolated, and the visibility of the confidence distribution information can be improved.

[0040] The information processing apparatus according to the 15th aspect is the information processing apparatus according to any one of the 1st to 14th aspects, wherein one or more processors may receive an input from a user to change the control to display the confidence distribution information.

[0041] According to such an aspect, selective switching between the visualization display of the disease region and the display of the confidence distribution information is realized.

[0042] The operation method of the information processing apparatus according to the 16th aspect of the present disclosure is a method for operating an information processing apparatus, in which a computer functioning as the information processing apparatus performs steps of acquiring a medical image, using a multi-disease detection model for detecting a plurality of disease regions from the medical image, and obtaining, for each pixel of the medical image, a confidence representing at least either the presence or absence of a disease and the degree of the disease for each disease, creating confidence distribution information for visualizing the distribution of the confidence for each detected disease, obtaining a threshold value applied to the confidence, and performing control to superimpose at least either a first color and a first opacity on the medical image and display it on a display device for pixels of a first image having a confidence equal to the same value as the threshold value for any one of the diseases.

[0043] According to the operation method of the information processing apparatus according to the 16th aspect of the present disclosure, it is possible to obtain the same operational effects as the information processing apparatus according to the 1st aspect of the present disclosure.

[0044] In the operation method of the information processing apparatus according to the 16th aspect, matters similar to those specified in the 2nd to 15th aspects can be appropriately combined. In that case, the components that perform the processes and functions specified in the information processing apparatus can be grasped as the components of the operation method of the information processing apparatus that perform the corresponding processes and functions.

[0045] The program according to the 17th aspect of the present disclosure is a program that causes a computer functioning as an information processing apparatus to realize functions of acquiring a medical image, using a multi-disease detection model that detects a plurality of disease regions from the medical image, and for each pixel of the medical image, acquiring, for each disease, a confidence level indicating at least either the presence or absence of the disease and the degree of the disease, creating confidence distribution information for visualizing the distribution of the confidence levels for each detected disease, acquiring a threshold value applied to the confidence level, and for a pixel of a first image having a confidence level equal to the threshold value for any one of the diseases, performing control to superimpose at least either a first color and a first opacity on the medical image and display it on a display device.

[0046] According to the program according to the 17th aspect of the present disclosure, it is possible to obtain the same operational effects as the information processing apparatus according to the 1st aspect of the present disclosure.

[0047] In the program according to the 17th aspect, matters similar to those specified in the 2nd to 15th aspects can be appropriately combined. In that case, the components that perform the processes and functions specified in the information processing apparatus can be grasped as the components of the program that perform the corresponding processes and functions.

[0048] The information processing apparatus according to the 18th aspect of the present disclosure includes one or more processors and one or more memories that store instructions to be executed by the one or more processors. The one or more processors use a multi-disease detection model that acquires a medical image and detects a plurality of disease regions from the medical image, and for each pixel of the medical image, acquires, for each disease, a confidence level indicating at least one of the presence or absence of a disease and the degree of the disease, acquires a threshold value applied to the confidence level, acquires information on regions for each anatomical structure from the medical image, acquires the maximum value of the confidence level for each disease in the region for each anatomical structure, and stores a combination of the anatomical structure name, the disease name, and the maximum value of the confidence level when the maximum value of the confidence level for each disease in the region for each anatomical structure exceeds the threshold value.

[0049] According to the image processing apparatus according to the 18th aspect of the present disclosure, support for creating a finding report is realized.

Effects of the Invention

[0050] According to the present invention, a pixel of the first image on which at least one of the first color and the first opacity is superimposed functions as a contour of a disease region. Thereby, in the display of a plurality of disease regions detected from a single medical image, visualization for each detected disease is realized.

Brief Description of the Drawings

[0051]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

[0052] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, the same reference numerals are assigned to the same components, and redundant descriptions will be omitted as appropriate. Further, when a plurality of components are exemplified in the following embodiments, it can be interpreted as including at least one of the plurality of components.

[0053] [Configuration Example of Medical Information Processing System] FIG. 1 is an overall configuration diagram of a medical information processing system. The medical information processing system 10 shown in the figure includes a medical imaging device 12, a medical image database 14, a user terminal device 16, a diagnostic report database 18, and a medical information processing device 20.

[0054] The medical imaging device 12, the medical image database 14, the user terminal device 16, the diagnostic report database 18, and the medical information processing device 20 are electrically connected to each other via a network 22 so that data can be freely transmitted and received. The communication form of the network 22 may be wired or wireless.

[0055] As an example of the network 22, a LAN that connects various devices within a medical institution can be mentioned. The network 22 may be a WAN that connects LANs of a plurality of medical institutions. Note that LAN is an abbreviation for Local Area Network. WAN is an abbreviation for Wide Area Network.

[0056] The medical imaging device 12 photographs an examination target part of a subject and generates a medical image. In FIG. 1, an X-ray imaging device is shown as the medical imaging device 12. Other examples of the medical imaging device 12 include a CT device, an MRI device, a PET device, an ultrasonic device, a CR device using a flat panel X-ray detector, and an endoscope device. Note that CT is an abbreviation for Computed Tomography. MRI is an abbreviation for Magnetic Resonance Imaging. PET is an abbreviation for Positron Emission Tomography. CR is an abbreviation for Computed Radiography.

[0057] The medical image database 14 is a database that manages medical images captured and generated using the medical imaging device 12. The medical image database 14 includes a large-capacity storage device in which medical images are stored. The medical image database 14 includes a computer in which software that provides the functions of a database management system is incorporated. Note that the term software is synonymous with the term program.

[0058] The medical image may be a two-dimensional image generated using an X-ray imaging device or the like, or may be a three-dimensional reconstructed image generated using a CT device or the like.

[0059] The format of the medical image may be subject to the DICOM standard. The medical image may be added with additional information such as DICOM tag information defined in the DICOM standard. Note that DICOM is an abbreviation for Digital Imaging and Communications in Medicine. Here, the term image in this specification may include, in addition to the meaning of an image such as a photograph, the meaning of image data that is a signal representing the image.

[0060] The user terminal device 16 is a terminal device for a user such as a doctor to create a reading report and view the reading report. The user terminal device 16 is installed with viewer software used when the user views a medical image. The user terminal device 16 applies a computer. The computer applied to the user terminal device 16 may be a workstation or a tablet terminal.

[0061] An input device 16A and a display device 16B are connected to the user terminal device 16. In FIG. 1, a keyboard and a mouse are illustrated as the input device 16A. As the display device 16B, a touch panel type display is provided, and the input device 16A and the display device 16B may be integrally configured.

[0062] The display device 16B can apply a liquid crystal display, an organic EL display, a projector, etc. The display device 16B can apply any combination of a plurality of devices. Note that EL of the organic EL display is an abbreviation of Electro-Luminescence.

[0063] A doctor as a user can instruct the display of a medical image using the input device 16A. The user terminal device 16 receives the user's instruction and causes the medical image to be displayed on the display device 16B according to the user's instruction. The doctor creates a key image determined to be important in the reading from the content of the findings text among the medical images displayed on the display device 16B. Further, the doctor inputs the findings text representing the reading result of the medical image using the input device 16A. In this way, the doctor creates a reading report including the key image and the findings text using the user terminal device 16.

[0064] The reading report database 18 is a database that manages the reading reports created by doctors. The reading report database 18 includes a large-capacity storage device in which the reading reports are stored. The reading report database 18 includes a computer in which software providing the functions of a database management system is incorporated. The medical image database 14 and the reading report database 18 may be configured using one computer.

[0065] The medical information processing device 20 functions as a medical image processing device that performs various processes on medical images. For example, the medical information processing device 20 generates various information to be superimposed on a medical image and causes the display device to display the medical image with the various information superimposed thereon. The display device for displaying the medical image may be the display device 16B provided in the user terminal device 16.

[0066] The medical information processing apparatus 20 is applied to a computer including one or more processors and one or more memories storing a program including one or more instructions executed by the one or more processors. The computer applied to the medical information processing apparatus 20 may be a workstation or a workstation or the like. The medical information processing apparatus 20 may be a virtual machine.

[0067] [Configuration example of medical information processing apparatus according to the first embodiment] FIG. 2 is a functional block diagram showing the electrical configuration of the medical information processing apparatus according to the first embodiment. The medical information processing apparatus 20 has a function of visualizing the heat map output of a multi-disease detection model that detects a plurality of disease regions from a medical image.

[0068] The medical information processing apparatus 20 includes a medical image acquisition unit 30, a disease region detection unit 32, a heat map image generation unit 34, a user threshold acquisition unit 36, a classification unit 38, a superimposed information generation unit 40, and a synthesis unit 42.

[0069] The medical image acquisition unit 30 acquires a medical image to be processed. The medical image acquisition unit 30 may acquire a medical image from the medical image database 14 or may acquire a medical image from the medical image imaging apparatus 12.

[0070] The medical image acquired using the medical image acquisition unit 30 may be a reconstructed image or may be raw data. In this embodiment, an example is shown in which the medical image acquisition unit 30 acquires a two-dimensional X-ray image as a medical image. The pixels of the two-dimensional medical image are specified using the coordinate values of the two-dimensional coordinate system applied to the medical image. As an example of the two-dimensional coordinate system, a two-dimensional orthogonal coordinate system can be cited. The origin of the two-dimensional orthogonal coordinate system is defined as appropriate. The coordinate values of the two-dimensional coordinate system can be grasped as the position of each pixel.

[0071] The medical image acquisition unit 30 may include the function of a processing unit that performs processing on the acquired medical image. For example, the medical image acquisition unit 30 may acquire raw data as a medical image and generate a reconstructed image from the raw data.

[0072] The disease area detection unit 32 detects a disease area from a medical image acquired using the medical image acquisition unit 30, and acquires a confidence level representing at least one of the presence or absence of a disease and the degree of the disease for each pixel constituting the medical image. The disease area detection unit 32 acquires a confidence level for each disease.

[0073] For example, when the confidence level is represented as a value between 0 and 100, a confidence level value of 0 means that there is no disease, and a confidence level value of 100 means that there is a disease. When the confidence level is a value exceeding 0 and less than 100, a relatively large confidence level value indicates a relatively high probability of having a disease, and a relatively small confidence level value indicates a relatively low probability of having a disease. The confidence level may represent the degree of a disease such as the severity of the disease.

[0074] The disease area detection unit 32 may be a multi-disease detection model that detects a plurality of different diseases from a single medical image. A pre-trained model may be applied to the multi-disease detection model. A deep learning model such as a CNN may be applied to the pre-trained model. Note that CNN is an abbreviation for Convolutional Neural Network.

[0075] The disease area detection unit 32 may detect a plurality of diseases from a single medical image using one multi-disease detection model. The disease area detection unit 32 may be equipped with a plurality of disease detection models for different diseases to be detected, and may detect a plurality of diseases.

[0076] The heatmap image generation unit 34 generates a heatmap image representing the distribution of the confidence level for each pixel. When a plurality of diseases are detected in the disease area detection unit 32, a heatmap image is generated for each disease. The plurality of heatmap images are integrated for all findings.

[0077] Pixels constituting the heatmap image are represented using a two-dimensional coordinate system applied to medical images. That is, pixels constituting the medical image are associated with a confidence level for each disease, and pixels constituting the heatmap image are associated with and stored. Note that the heatmap image described in the embodiment is an example of confidence distribution information representing the distribution of confidence levels.

[0078] The user threshold acquisition unit 36 acquires a user threshold used when classifying the confidence level for each disease acquired using the disease region detection unit 32. The user threshold acquisition unit 36 may acquire the user threshold input by the user, or may read and acquire the user threshold stored in advance for each specified condition according to the condition.

[0079] The classification unit 38 classifies the confidence level for each disease for each pixel of the medical image using the user threshold. The classification unit 38 classifies pixels whose confidence level matches the user threshold into the first class. The classification unit 38 classifies pixels whose confidence level exceeds the user threshold into the second class. The classification unit 38 classifies pixels whose confidence level is less than the user threshold into the third class.

[0080] That is, the classification unit 38 classifies pixels having a confidence level exceeding the confidence level in the pixels classified into the first class into the second class. The classification unit 38 classifies pixels having a confidence level less than the confidence level in the pixels classified into the first class into the third class.

[0081] The classification unit 38 classifies pixels whose confidence level is within the user threshold range into the first class, pixels whose confidence level exceeds the upper limit value of the user threshold range into the second class, and pixels whose confidence level is less than the lower limit value of the user threshold range into the third class. The classification unit 38 stores the classification result for each pixel.

[0082] For example, when the confidence level is represented by a value between 0 and 100, and the user threshold is 50, pixels with a confidence level of 50 are classified into the first class, pixels with a confidence level exceeding 50 are classified into the second class, and pixels with a confidence level less than 50 are classified into the third class. Note that the pixels classified into the first class described in the embodiment are an example of the pixels of the first image. The pixels classified into the second class described in the embodiment are an example of the pixels of the second image. The pixels classified into the third class described in the embodiment are an example of the pixels of the third image.

[0083] The user threshold acquisition unit 36 may acquire a user threshold range including values in the vicinity of the user threshold for the user threshold. That is, the user threshold acquisition unit 36 may acquire the upper limit value of the user threshold range and the lower limit value of the user threshold range.

[0084] The classification unit 38 may classify each pixel constituting the medical image based on the confidence level for each disease for each pixel using the user threshold range. For example, when the range of plus or minus 1.0 percent of the user threshold is defined as the vicinity of the user threshold, a range of 49.5 or more and 50.5 or less is acquired as the user threshold range.

[0085] The classification unit 38 may classify pixels with a confidence level of 49.5 or more and 50.5 or less into the first class. The classification unit 38 may classify pixels with a confidence level exceeding 50.5 into the second class and pixels with a confidence level less than 49.5 into the second class.

[0086] The classification unit 38 may classify pixels within a specified range from the pixels classified into the first class into the first class. For example, the classification unit 38 may classify all pixels within 1.0 millimeter from the pixels classified into the first class into the first class regardless of the confidence level.

[0087] The superimposed information generation unit 40 generates superimposed information corresponding to each class for each pixel. The superimposed information generation unit 40 generates first superimposed information in which the values of a fixed first color and first opacity are superimposed on the input medical image for the pixels classified into the first class. The fixed first color is preferably a color that can be visually distinguished from the color used for the heat map image.

[0088] The superimposed information generation unit 40 generates second superimposed information in which the values of a second color and second opacity are superimposed on the input medical image according to the maximum value of the confidence level among the diseases classified into the second class for the pixels classified into the second class.

[0089] The superimposed information generation unit 40 generates third superimposed information in which the input medical image is displayed as it is for the pixels classified into the third class. For example, the superimposed information generation unit 40 may perform a process of not generating superimposed information to be superimposed on the input medical image for the pixels classified into the third class. Note that the third superimposed information in which the input medical image described in the embodiment is displayed as it is is an example of a process of setting the color and opacity applied to the pixels of the third class to non-superimposed.

[0090] The synthesizing unit 42 generates a synthesized image in which a heat map image is superimposed on the input medical image, and further, at least any one of the first superimposed information, the second superimposed information, and the third superimposed information is superimposed. For example, the synthesizing unit 42 may generate a synthesized image in which the first superimposed information is superimposed on the heat map image. The synthesizing unit 42 may generate a synthesized image in which the first superimposed information and the second superimposed information are superimposed on the input medical image. The synthesizing unit 42 causes the display device 44 to display the synthesized image.

[0091] The medical information processing apparatus 20 may include a display control unit that generates a display signal representing the synthesized image generated using the synthesizing unit 42. The display device 44 may acquire a signal representing the synthesized image and convert it into a display signal representing the synthesized image.

[0092] The display device 44 shown in FIG. 2 may be a display device connected to a computer connected via the network 22 shown in FIG. 1. For example, the display device 44 shown in FIG. 2 may be the display device 16B connected to the user terminal device 16 shown in FIG. 1.

[0093] FIG. 3 is a block diagram showing an example of the hardware configuration of the medical information processing device shown in FIG. 2. The medical information processing device 20 includes a processor 102, a computer-readable medium 104, a communication interface 106, an input / output interface 108, and a bus 110.

[0094] The processor 102 includes a CPU. The processor 102 may include a GPU. Note that CPU is an abbreviation for Central Processing Unit. GPU is an abbreviation for Graphics Processing Unit.

[0095] The processor 102 is connected to the computer-readable medium 104, the communication interface 106, and the input / output interface 108 via the bus 110.

[0096] The medical information processing device 20 causes the processor 102 to execute a program stored in the computer-readable medium 104 to realize various functions. Note that the term "program" is synonymous with the term "software".

[0097] The computer-readable medium 104 includes a memory 112 which is a main storage device and a storage 114 which is an auxiliary storage device. The computer-readable medium 104 can apply a semiconductor memory, a hard disk device, a solid state drive device, etc. The computer-readable medium 104 can apply any combination of a plurality of devices.

[0098] Note that a hard disk drive can be referred to as an HDD, which is an abbreviation of Hard Disk Drive in English. A solid state drive device can be referred to as an SSD, which is an abbreviation of Solid State Drive in English.

[0099] The medical information processing device 20 performs data communication with an external device via the communication interface 106. The communication interface 106 can apply various standards such as USB. The communication mode of the communication interface 106 may apply either wired communication or wireless communication. Note that USB is an abbreviation of Universal Serial Bus and is a registered trademark.

[0100] The memory 112 of the computer-readable medium 104 stores a medical image acquisition program 120, a disease region detection program 122, a heat map image generation program 124, a user threshold acquisition program 126, a classification program 128, a superimposed information generation program 130, and a synthesis program 132 that are executed by the processor 102. The disease region detection program 122 may include a confidence level acquisition program 134 that calculates the confidence level for each disease for each pixel.

[0101] The medical image acquisition program 120 is applied to the medical image acquisition unit 30 illustrated in FIG. 2 and realizes the function of acquiring a medical image. The disease region detection program 122 and the confidence level acquisition program 134 detect a disease region from the medical image and realize the function of acquiring the confidence level for each disease for each pixel as a detection result. The disease region detection program 122 and the confidence level acquisition program 134 are a learned multi-disease detection model applied to the disease region detection unit 32.

[0102] The heat map image generation program 124 is applied to the heat map image generation unit 34 and realizes the function of generating a heat map. The user threshold acquisition program 126 is applied to the user threshold acquisition unit 36 and realizes the function of acquiring a user threshold. The classification program 128 is applied to the classification unit 38 and realizes the function of classifying pixels based on the confidence level for each disease for each pixel.

[0103] The superimposed information generation program 130 is applied to the superimposed information generation unit 40, and the synthesis program 132 that realizes the function of generating superimposed information is applied to the synthesis unit 42, and superimposes a heat map image on the input medical image, and further superimposes the superimposed information for each class on the heat map image to generate a synthesized image.

[0104] The various programs stored in the computer-readable medium 104 include one or more instructions. The computer-readable medium 104 stores various data and various parameters used when the various programs are executed.

[0105] Here, examples of the hardware structure of the processor 102 include a CPU, a GPU, a PLD (Programmable Logic Device), and an ASIC (Application Specific Integrated Circuit). The CPU is a general-purpose processor that executes a program and acts as various functional units. The GPU is a processor specialized for image processing.

[0106] The PLD is a processor whose electrical circuit configuration can be changed after the device is manufactured. An example of the PLD is an FPGA (Field Programmable Gate Array). The ASIC is a processor having a dedicated electric circuit designed specifically to execute a specific process.

[0107] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. Examples of combinations of various processors include a combination of one or more FPGAs and one or more CPUs, and a combination of one or more FPGAs and one or more GPUs. Other examples of combinations of various processors include a combination of one or more CPUs and one or more GPUs.

[0108] A single processor may be used to form a plurality of functional units. As an example of forming a plurality of functional units using a single processor, a combination of one or more CPUs such as a SoC (System On a Chip) and software, represented by a computer such as a client or a server, is applied to configure a single processor, and this processor is made to act as a plurality of functional units.

[0109] As another example of forming a plurality of functional units using a single processor, there is an aspect of using a processor that realizes the functions of an entire system including a plurality of functional units using a single IC chip. Note that IC is an abbreviation for Integrated Circuit.

[0110] Thus, various processing units are configured as a hardware structure using one or more of the various processors described above. Further, the hardware structure of the various processors described above is more specifically an electrical circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0111] [Procedure of the medical information processing method according to the first embodiment] FIG. 4 is a flowchart showing the procedure of the medical information processing method according to the first embodiment. In the medical image acquisition step S10, the medical image acquisition unit 30 shown in FIG. 2 acquires a medical image. After the medical image acquisition step S10, the process proceeds to the disease region detection step S12.

[0112] In the disease region detection step S12, the disease region detection unit 32 detects one or more disease regions from the acquired medical image. In the disease region detection step S12, as the detection result of the disease region, a confidence level for each pixel and each disease is acquired. After the disease region detection step S12, the process proceeds to the heatmap image generation step S14.

[0113] In the heatmap image generation step S14, the heatmap image generation unit 34 generates a heatmap image representing the distribution of confidence levels in the medical image for each disease. After the heatmap image generation step S14, the process proceeds to the user threshold acquisition step S16.

[0114] In the user threshold acquisition step S16, the user threshold acquisition unit 36 acquires the user threshold. In the user threshold acquisition step S16, the user threshold acquisition unit 36 may acquire the upper limit value and the lower limit value of the user threshold range. After the user threshold acquisition step S16, the process proceeds to the first classification step S18.

[0115] The user threshold acquisition step S16 may be executed before the heatmap image generation step S14 is executed, and may be executed in parallel with the heatmap image generation step S14. That is, it is sufficient that the user threshold is acquired before the first classification step S18 is started.

[0116] In the first classification step S18, the classification unit 38 determines for each pixel and each disease whether the confidence level is different from the user threshold. In the first classification step S18, the classification unit 38 may determine for each pixel and each disease whether the confidence level is the same as the user threshold.

[0117] Pixels with a confidence level equal to the user threshold are determined as a No decision and the process proceeds to the first class determination step S20. When the user threshold range is applied, pixels with a confidence level within the user threshold range may be determined as a No decision and the process may proceed to the first class determination step S20.

[0118] In the first class determination step S20, the classification unit 38 determines the classification of pixels with a confidence level equal to the user threshold as the first class and stores the classification result for each pixel and each disease. When the user threshold range is applied, the classification of pixels with a confidence level within the user threshold range may be determined as the first class. After the first class determination step S20, the process proceeds to the first overlay information generation step S22.

[0119] In the first overlay information generation step S22, the overlay information generation unit 40 generates the first overlay information for the pixels classified into the first class. After the first overlay information generation step S22, the process proceeds to the composite image generation step S34.

[0120] On the one hand, in the first classification step S18, pixels with a confidence level exceeding the user threshold and pixels with a confidence level less than the user threshold are determined as Yes, and the process proceeds to the second classification step S24. When the user threshold range is applied, pixels with a confidence level less than the lower limit of the user threshold range and pixels with a confidence level exceeding the upper limit of the user threshold range are determined as Yes, and the process may proceed to the second classification step S24.

[0121] In the second classification step S24, the classification unit 38 determines whether the confidence level is less than the user threshold. In the second classification step S24, pixels with a confidence level exceeding the user threshold are determined as No, and the process proceeds to the second class determination step S26. When the user threshold range is applied, pixels with a confidence level exceeding the upper limit of the user threshold range are determined as No, and the process may proceed to the second class determination step S26.

[0122] In the second class determination step S26, the classification unit 38 determines the classification of pixels with a confidence level exceeding the user threshold as the second class, and stores the classification results for each pixel and each disease. When the user threshold range is applied, the classification of pixels with a confidence level exceeding the upper limit of the user threshold range may be determined as the second class. After the second class determination step S26, the process proceeds to the second superimposed information generation step S28.

[0123] In the second superimposed information generation step S28, the superimposed information generation unit 40 generates second superimposed information for the pixels classified into the second class. After the second superimposed information generation step S28, the process proceeds to the composite image generation step S34.

[0124] On the other hand, in the second classification step S24, pixels with a confidence level less than the user threshold are determined as Yes, and the process proceeds to the third class determination step S30. When the user threshold range is applied, pixels with a confidence level less than the lower limit of the user threshold range are determined as Yes, and the process may proceed to the third class determination step S30.

[0125] In the third-class determination step S30, the classification unit 38 determines the classification of pixels with a confidence level lower than the user threshold as the third class, and stores the classification results for each pixel and each disease. When the user threshold range is applied, the classification of pixels with a confidence level lower than the lower limit value of the user threshold range may be determined as the third class. After the third-class determination step S30, the process proceeds to the third-overlay information generation step S32.

[0126] In the third-overlay information generation step S32, the overlay information generation unit 40 generates third-overlay information for the pixels classified as the third class. If the third-overlay information is not generated and the pixels classified as the third class are displayed as the medical image as it is, the third-overlay information generation step S32 may be omitted. After the third-overlay information generation step S32, the process proceeds to the composite image generation step S34.

[0127] In the composite image generation step S34, the composite unit 42 superimposes the heatmap image on the medical image, and further generates a composite image in which at least any one of the first overlay information, the second overlay information, and the third overlay information is superimposed. That is, in the composite image generation step S34, the superimposition of the second overlay information on the heatmap image may be omitted, or the superimposition of the third overlay information on the heatmap image may be omitted. After the composite image generation step S34, the process proceeds to the composite image display step S36.

[0128] In the composite image display step S36, the composite unit 42 causes the display device 44 to display the composite image. When the composite image is displayed on the display device 44, the procedure of the medical information processing method ends. Note that the medical information processing method whose procedure is illustrated in FIG. 4 is an example of an operation method of the information processing apparatus. Each step illustrated in FIG. 4 is an example of a step in which various processes are performed.

[0129] At least some of the steps in the procedure of the medical information processing method illustrated in FIG. 4 may include processing according to a signal input in response to an operator's operation. Further, the processing of the computer functioning as the information processing apparatus may include processing for acquiring information from an external computer, a measuring device, or the like. The same applies to the procedure of the medical information processing method according to other embodiments.

[0130] [Specific Example of Heat Map Display] FIG. 5 is a schematic diagram of a heat map image and a comprehensive finding integrated image according to the first embodiment. In the figure, a chest X-ray image is applied as a medical image MI on which the heat map image is superimposed. A composite image CI1 in which the heat map image HM1 is superimposed on the medical image MI, a composite image CI2 in which the heat map image HM2 is superimposed on the medical image MI, and a composite image CI3 in which the heat map image HM3 is superimposed on the medical image MI are shown. Further, in FIG. 5, a comprehensive finding integrated image MII in which the composite images CI1, CI2, and CI3 are integrated is shown. That is, the comprehensive finding integrated image MII shown in FIG. 5 is an example of a composite image generated in the composite image generation step S34 shown in FIG. 4 and displayed in the composite image display step S36.

[0131] The heat map image HM1 represents the distribution of the confidence level of a disease when a nodule or infiltrative shadow is detected as the disease. The heat map image HM2 represents the distribution of the confidence level of a disease when pneumothorax is detected as the disease. The heat map image HM3 represents the distribution of the confidence level of a disease when pleural effusion is detected as the disease.

[0132] In the heat map image HM1, the magnitude of the confidence level is represented using a plurality of colors with different colors such as red, orange, yellow, green, blue, indigo, and violet. For example, in the heat map image HM1, pixels with a relatively high confidence level use colors on the relatively red side, and pixels with a relatively low confidence level use colors on the relatively violet side. The same configuration as that of the heat map image HM1 is also applied to the heat map image HM2 and the heat map image HM3.

[0133] In the comprehensive finding integrated image MII, first superimposed information of nodules or infiltrative shadows to which a value of a specified color and a specified opacity is applied to the pixels classified into the first class in the heat map image HM1 is superimposed on the medical image MI.

[0134] In the overall finding integrated image MII, the first superimposed information of pneumothorax, in which the specified color and the value of the specified opacity are applied to the pixels classified into the first class in the heat map image HM2, is superimposed on the medical image MI.

[0135] Furthermore, in the overall finding integrated image MII, the first superimposed information of pleural effusion, in which the specified color and the value of the specified opacity are applied to the pixels classified into the first class in the heat map image HM3, is superimposed on the medical image MI.

[0136] FIG. 6 is a schematic diagram of a medical image on which the first superimposed information is superimposed. In FIG. 6, an enlarged view of the overall finding integrated image MII shown in FIG. 5 is shown. In FIG. 6, the heat map image HM1, the heat map image HM2, and the heat map image HM3 are schematically shown. In the heat map image HM1 and the like, the difference in confidence level is represented by applying the difference in hatching instead of color.

[0137] The first superimposed information SI11 of nodules or infiltrative shadows superimposed on the overall finding integrated image MII represents the boundary of the disease region of the nodules or infiltrative shadows. The first superimposed information SI12 of pneumothorax superimposed on the overall finding integrated image MII represents the boundary of the disease region of the pneumothorax. The first superimposed information SI13 of pleural effusion superimposed on the overall finding integrated image MII represents the boundary of the disease region of the pleural effusion. That is, the first superimposed information SI11 of nodules or infiltrative shadows, the first superimposed information SI12 of pneumothorax, and the first superimposed information SI13 represent the contour of the abnormal finding region.

[0138] The color applied to the first superimposed information SI11 of nodules or infiltrative shadows is a color that can be distinguished from the heat map image HM1 superimposed on the disease region of nodules or infiltrative shadows, the heat map image HM2 superimposed on the disease region of pneumothorax, and the heat map image HM3 superimposed on the disease region of pleural effusion. The color applied to the first superimposed information SI12 of pneumothorax and the color applied to the first superimposed information SI13 of pleural effusion are the same as the color applied to the first superimposed information SI11 of nodules or infiltrative shadows.

[0139] The colors applied to the first overlay information SI11 of nodules or infiltrative shadows, the colors applied to the first overlay information SI12 of pneumothorax, and the colors applied to the first overlay information SI13 of pleural effusion are preferably colors that can be distinguished from each other.

[0140] The opacity value applied to the first overlay information SI11 of nodules or infiltrative shadows is an opacity value that can be distinguished from the heat map image HM1, the heat map image HM2, and the heat map image HM3.

[0141] The opacity values applied to the first overlay information SI12 of pneumothorax and the opacity values applied to the first overlay information SI13 of pleural effusion are also the same as the opacity value applied to the first overlay information SI11 of nodules or infiltrative shadows.

[0142] The opacity values applied to the first overlay information SI11 of nodules or infiltrative shadows, the opacity values applied to the first overlay information SI12 of pneumothorax, and the opacity values applied to the first overlay information SI13 of pleural effusion are preferably opacity values that can be distinguished from each other.

[0143] The opacity value referred to here represents the degree of opacity of a specified color. The opacity is adjusted by changing the ratio of the area that transmits the lower color to the area that does not transmit it. In this embodiment, a plurality of consecutive pixels representing the contour of the abnormal finding region are classified into the first class. A specified color is applied to the pixels classified into the first class, and further, a specified opacity value representing the ratio of not transmitting the specified color is applied. The specified opacity value is realized using the ratio of the pixels that do not transmit the specified color to the pixels that transmit the specified color for a plurality of pixels.

[0144] Note that the specified color applied to each of the first overlay information SI11, the first overlay information SI12, and the first overlay information SI13 described in the embodiment is an example of the first color, and the specified opacity is an example of the first opacity.

[0145] FIG. 7 is a schematic diagram of a medical image on which the second superimposed information and the third superimposed information are superimposed. For the pixels classified into the second class in the pixels constituting the medical image MI, the specified color and opacity values corresponding to the maximum value of the confidence level are applied, and the specified color and opacity values corresponding to the maximum value of the confidence level are superimposed on the medical image MI.

[0146] In the overall finding integrated image MII, for the medical image MI on which the first superimposed information SI11 of nodules or infiltrative shadows, the first superimposed information SI2 of pneumothorax, and the first superimposed information SI13 of pleural effusion are superimposed, the second superimposed information SI21 of nodules or infiltrative shadows, the second superimposed information SI22 of pneumothorax, and the second superimposed information SI23 of pleural effusion are superimposed.

[0147] Each of the second superimposed information SI21 of nodules or infiltrative shadows, the second superimposed information SI22 of pneumothorax, and the second superimposed information SI23 of pleural effusion has a specified color corresponding to the maximum value of the confidence level in the pixels classified into the second class and a specified opacity corresponding to the maximum value of the confidence level.

[0148] The second superimposed information SI21 of nodules or infiltrative shadows shown in FIG. 7 is applied with the color applied to the maximum value of the confidence level in the heat map image HM1 shown in FIGS. 5 and 6. The second superimposed information SI22 of pneumothorax is applied with the color applied to the maximum value of the confidence level in the heat map image HM2. The second superimposed information SI23 of pleural effusion is applied with the color applied to the maximum value of the confidence level in the heat map image HM3.

[0149] The opacity value applied to the second superimposed information SI21 of nodules or infiltrative shadows is preferably less than the opacity value applied to the first superimposed information SI11 of nodules or infiltrative shadows. The opacity value applied to the second superimposed information SI21 of nodules or infiltrative shadows may be less than the opacity value applied to the first superimposed information SI12 of pneumothorax and may be less than the opacity value applied to the first superimposed information SI13 of pleural effusion.

[0150] The opacity value applied to the second superimposed information SI22 of pneumothorax is preferably less than the opacity value applied to the first superimposed information SI2 of pneumothorax. The opacity value applied to the second superimposed information SI22 of pneumothorax may be less than the opacity value applied to the first superimposed information SI1 of nodules or infiltrative shadows, and may be less than the opacity value applied to the first superimposed information SI13 of pleural effusion.

[0151] The opacity value applied to the second superimposed information SI23 of pleural effusion is preferably less than the opacity value applied to the first superimposed information SI13 of pleural effusion. The opacity value applied to the second superimposed information SI23 of pleural effusion may be less than the opacity value applied to the first superimposed information SI11 of nodules or infiltrative shadows, and may be less than the opacity value applied to the first superimposed information SI12 of pneumothorax.

[0152] Note that the specified color applied to each of the second superimposed information SI21, the second superimposed information SI22, and the second superimposed information SI23 described in the embodiment is an example of the second color, and the opacity value is an example of the second opacity.

[0153] In FIG. 7, a comprehensive view image MII is illustrated in which a medical image MI input is directly displayed for pixels that are not generated with the third superimposed information SI3 and are classified into the third class, with a symbol SI3 representing the third superimposed information attached. That is, when the medical image MI illustrated in FIG. 7 is displayed, the pixels classified into the third class are directly displayed as the pixels of the medical image.

[0154] For one pixel, if there are diseases classified into the first class and diseases classified into the second class, the first superimposed information for the diseases classified into the first class is generated. Also, for one pixel, if there are diseases classified into the first class and diseases classified into the third class, the first superimposed information for the diseases classified into the first class is generated. Further, for one pixel, if there are diseases classified into the second class and diseases classified into the third class, the first superimposed information for the diseases classified into the second class is generated.

[0155] [Operation and Effect of Medical Information Processing Apparatus and Medical Information Processing Method According to First Embodiment] The medical information processing apparatus 20 and the medical information processing method according to the first embodiment can obtain the following operation and effect.

[0156] [1] In the medical image MI, a plurality of disease regions are detected, a confidence level for each disease is calculated for each pixel, and a heat map image representing the distribution of the confidence levels is generated for each disease. Pixels having a confidence level value equal to the user threshold for each disease and each pixel are classified into the first class. For the pixels classified into the first class, first superimposed information to which values of a first color and a first opacity are applied is superimposed. Thereby, first superimposed information representing the contour of the disease region for each disease is superimposed on the medical image, and visualization in which each of the plurality of disease regions is distinguished is realized.

[0157] [2] As the values of the color and opacity applied to the first superimposed information, values of a color and an opacity that allow the first superimposed information to be distinguished from the heat map image are applied. Thereby, even when a plurality of disease regions overlap, the contour for each disease is distinguished from the medical image and the heat map.

[0158] [3] Pixels having a confidence level exceeding the user threshold are classified into the second class. For the pixels classified into the second class, second superimposed information to which values of a color and an opacity corresponding to the maximum value of the confidence levels among the plurality of pixels classified into the second class are applied is generated. The second superimposed information is superimposed on the medical image. Thereby, visualization of the disease region for each disease is realized.

[0159] [4] As the color applied to the second superimposed information, the color of the maximum value of the confidence level in the heat map is applied. Thereby, visualization of the disease region associated with the heat map for each disease is realized.

[0160] [5] The value of the opacity applied to the second overlapping information is a value less than the opacity applied to the first overlapping information for each disease. Thereby, the first overlapping information is preferentially visualized.

[0161] [6] Pixels with a confidence level less than the user threshold are classified into the third class. The pixels classified into the third class are displayed as the medical image as it is. Thereby, a display of a medical image in which a diseased area and a non-diseased area are distinguished is realized.

[0162] [7] The user threshold acquisition unit acquires a user threshold range including values in the vicinity of the user threshold. The classification unit classifies pixels whose confidence level for each disease for each pixel is within the user threshold range into the first class. Thereby, the number of pixels constituting the contour of the diseased area is relatively increased, and the contour of the diseased area is thickened and emphasized.

[0163] [First Modification Example of Visualization of Heatmap Image] In setting the value of the opacity for the pixels classified into the second class, when all the pixels in the vicinity of the pixels classified into the second class are classified into the first class for any of all the diseases, an opacity value lower than the opacity value applied to the pixels of the first class may be applied.

[0164] Thereby, the transmittance of the pixels of the first class visually recognized as the contour is relatively decreased, and the contour is emphasized. Note that the opacity value lower than the opacity value applied to the pixels of the first class described in the embodiment is an example of a third opacity lower than the first opacity set for the pixels of the second class.

[0165] The pixels in the vicinity of the pixels classified into the second class may include pixels adjacent to the pixels classified into the second class. The pixels in the vicinity of the pixels classified into the second class may be pixels in a range of 2 or more and 10 or less pixels from the pixels classified into the second class. Note that the upper limit value of 10 pixels in the range of the pixels in the vicinity is an example and can be defined as appropriate.

[0166] [Second Modified Example of Visualization of Heatmap Image] For each pixel constituting the heatmap image illustrated in FIG. 6 and the like, the maximum value of the confidence level in the local region located within the specified range from each pixel is used as the value of the confidence level of each pixel, and the value of the confidence level of each pixel is updated. The range of the vicinity of each pixel may be a range from 10 pixels to 20 pixels when one pixel is 0.25 millimeters square.

[0167] When there is only one pixel having the maximum value of the confidence level, there is a concern that the disease region to which the pixel having the maximum value of the confidence level belongs may be visually recognized as a normal region. Therefore, dilation processing is performed on the pixel having the maximum value of the confidence level, and the visibility of the region having the maximum value of the confidence level can be improved.

[0168] [Configuration Example of Medical Information Processing Apparatus According to Second Embodiment] FIG. 8 is a functional block diagram showing the electrical configuration of the medical information processing apparatus according to the second embodiment. The medical information processing apparatus 20C illustrated in the figure has a maximum value calculation unit 52 and a comparison unit 48 added to the medical information processing apparatus 20 illustrated in FIG. 2.

[0169] The maximum value calculation unit 52 calculates the maximum value of the confidence level for each disease. The comparison unit 48 compares the maximum value of the confidence level for each disease with the user threshold value, and generates a comparison result indicating whether the maximum value of the confidence level for each disease exceeds the user threshold value.

[0170] When the comparison unit 48 generates a comparison result indicating that the maximum value of the confidence level for each disease exceeds the user threshold value, the superimposed information generation unit 40 generates character information representing the disease name and character information representing the maximum value of the confidence level associated with the disease name.

[0171] The medical information processing apparatus 20C may include a character information generation unit that generates character information representing the disease name and character information representing the maximum value of the confidence level associated with the disease name, separately from the superimposed information generation unit 40.

[0172] The synthesizing unit 42 generates a comprehensive finding integrated image MII2 that superimposes character information representing a disease name and character information representing the maximum value of the confidence level associated with the disease name on a medical image MI on which a heat map image HM1 or the like and first superimposed information SI1 or the like are superimposed.

[0173] The display device 44 acquires a display signal representing a comprehensive finding integrated image in which a heat map image HM1 or the like and first superimposed information SI1 or the like are superimposed on a medical image MI, and character information representing the maximum value of the confidence level for each disease is superimposed, and the comprehensive finding integrated image on which the character information or the like is superimposed is displayed.

[0174] FIG. 9 is a schematic diagram of a medical image on which character information representing the maximum value of the confidence level for each disease is superimposed. In the comprehensive finding integrated image MII2 illustrated in the figure, character information T1 representing the maximum value of the confidence level for each disease is superimposed on the medical image MI. The character information T1 indicates that the maximum value of the confidence level for nodules or ground-glass opacities is 90, the maximum value of the confidence level for pneumothorax is 99, and the maximum value of the confidence level for pleural effusion is 68.

[0175] In the comprehensive finding integrated image MII2 illustrated in FIG. 9, the character information T1 is superimposed on the medical image MI. However, the comprehensive finding integrated image MII2 may include a region where the character information T1 is arranged outside the medical image MI. The character information T1 can be arranged at any position in the comprehensive finding integrated image MII2, but is preferably arranged at a position that does not interfere with the visibility of the heat map image HM1 or the like and the first superimposed information SI11 or the like.

[0176] FIG. 9 illustrates character information T1 on which characters to which white is applied against a black background are superimposed. The characters of the character information T1 may be applied with any color that can be distinguished from the background. The characters of the character information T1 may be applied with any font and any size.

[0177] The background of the character information T1 may be any color that can be distinguished from the characters. The background of the character information T1 may be transparent or translucent.

[0178] The hardware configuration of the electrical configuration of the medical information processing apparatus 20C according to the second embodiment is the same as the hardware configuration of the electrical configuration of the medical information processing apparatus 20 illustrated in FIG. 3. In the medical information processing apparatus 20C, the maximum value calculation program applied to the maximum value calculation unit 52 and the comparison program applied to the comparison unit 48 are stored in the memory 112 illustrated in FIG. 3.

[0179] The processor 102 executes the maximum value calculation program to realize the maximum value calculation function. Further, the processor 102 executes the comparison program to realize the comparison function.

[0180] In the medical image processing method applied to the medical information processing apparatus 20C according to the second embodiment, a maximum value calculation step executed by the maximum value calculation unit 52 and a comparison step executed by the comparison unit 48 are added to the flowchart illustrated in FIG. 4. In the composite image generation step S34, character information T1 representing the maximum value of the disease name and the confidence level for each disease is generated. In the composite image display step S36, a composite image including the character information T1 is displayed.

[0181] [Operation and Effect of Medical Information Processing Apparatus According to Second Embodiment] According to the medical information processing apparatus 20 and the medical information processing method according to the second embodiment, in the overall findings integrated image MII2, character information T1 representing the maximum value of the disease name and the confidence level for each disease is superimposed on the medical image MI. Thereby, visualization of the maximum value of the confidence level for each disease is realized.

[0182] For example, when there are a plurality of disease regions for the same disease in the medical image MI, the maximum confidence level among the plurality of disease regions for the same disease is visualized.

[0183] [Configuration Example of Medical Information Processing Apparatus According to Third Embodiment] FIG. 10 is a functional block diagram showing the electrical configuration of the medical information processing apparatus according to the third embodiment. The medical information processing apparatus 20D illustrated in the figure is obtained by adding a connected region labeling processing unit 50, a maximum value calculation unit 52, and a designated information acquisition unit 54 to the medical information processing apparatus 20 illustrated in FIG. 2.

[0184] The connected region labeling processing unit 50 performs a labeling process on a disease region whose confidence level for each disease exceeds the user threshold. The connected region labeling processing unit 50 assigns a label for identifying the disease region to the disease region for each disease.

[0185] When there are a plurality of connected regions isolated from each other inside the contour of a disease region composed of a plurality of pixels classified into the first class, the connected region labeling processing unit 50 assigns different labels to each connected region.

[0186] When there is a connected region where disease regions in different diseases overlap, the connected region labeling processing unit 50 assigns different labels for each disease to the connected region where the disease regions in different diseases overlap.

[0187] The maximum value calculation unit 52 calculates the maximum value of the confidence level for each connected region to which a label is assigned. The maximum value calculation unit 52 associates and stores the maximum value of the confidence level for each connected region and each disease with the label of the region. In FIG. 10, illustration of the storage unit in which the maximum value of the confidence level for each connected region and each disease is stored is omitted.

[0188] The specified information acquisition unit 54 acquires the position information of the pixel specified by the user operating the mouse. The position information of the pixel is applied with the coordinate values of the two-dimensional coordinate system defined for the medical image MI. The specified information acquisition unit 54 transmits the acquired position information of the pixel to the connected region labeling processing unit 50.

[0189] The connected region labeling processing unit 50 identifies the connected region to which the pixel specified by the user operating the mouse belongs. The maximum value calculation unit 52 transmits the maximum value of the confidence level for each disease in the specified connected region to the superimposed information generation unit 40.

[0190] The overlapping information generation unit 40 generates character information representing the maximum value of the confidence level for each disease in the specified connected region. The medical information processing apparatus 20D may include, separately from the overlapping information generation unit 40, a character information generation unit that generates character information representing the maximum value of the confidence level for each disease in the specified connected region.

[0191] The synthesizing unit 42 generates a comprehensive view integrated image in which character information representing the maximum value of the confidence level for each disease in the connected region to which the pixel designated by the user's mouse operation belongs is superimposed on the medical image MI on which the heat map image HM1 or the like and the first overlapping information SI1 or the like are superimposed.

[0192] The display device 44 acquires a display signal representing a comprehensive view integrated image in which the heat map image HM1 or the like and the first overlapping information SI1 or the like are superimposed on the medical image MI, and character information representing the maximum value of the confidence level for each disease in the specified connected region is superimposed, and the comprehensive view integrated image on which the character information or the like is superimposed is displayed.

[0193] FIG. 11 is a schematic diagram of a medical image on which the maximum value of the confidence level for each disease in the specified region is superimposed. In the comprehensive view integrated image MII3 shown in the figure, a cursor CU that moves in response to the user's mouse operation is displayed on the medical image MI.

[0194] Further, in the comprehensive view integrated image MII3, character information T2 representing the maximum value of the confidence level for each disease in the connected region to which the pixel designated using the cursor CU belongs is superimposed on the medical image MI.

[0195] FIG. 11 shows a comprehensive view integrated image MII3 in which character information T2 representing that the maximum value of the confidence level of a nodule or infiltrative shadow in the connected region to which the pixel in the disease region of the nodule or infiltrative shadow is designated and belongs is 90 is superimposed on the medical image MI.

[0196] When a connected region is specified using the cursor CU, the connected region to which the pixel overlaid with the cursor CU belongs may be the specified connected region. The pixel overlaid with the cursor CU may be a pixel having the same coordinate value as the coordinate value of the tip position of the cursor CU. Instead of the tip position of the cursor CU, any position of the cursor CU may be applied.

[0197] In the panoramic integrated image MII3 illustrated in FIG. 11, character information T2 is superimposed on the range of the medical image MI. The panoramic integrated image MII3 may include a region where the character information T2 is arranged outside the medical image MI. The character information T2 can be arranged at any position in the panoramic integrated image MII3, but it is preferably arranged at a position that does not interfere with the visibility of the heat map image HM1 and the like and the first superimposed information SI1 and the like.

[0198] Similar to the character information T1 illustrated in FIG. 9, the character information T2 illustrated in FIG. 11 may have its background color, character color, character font, and character size defined.

[0199] The hardware configuration of the electrical configuration of the medical information processing apparatus 20D according to the third embodiment is the same as the hardware configuration of the electrical configuration of the medical information processing apparatus 20 illustrated in FIG. 3. In the medical information processing apparatus 20D, the connected region labeling processing program applied to the connected region labeling processing unit 50, the maximum value calculation program applied to the maximum value calculation unit 52, and the designated information acquisition program applied to the designated information acquisition unit 54 are stored in the memory 112 illustrated in FIG. 3.

[0200] The processor 102 executes the connected region labeling processing program to realize the connected region labeling processing function. Further, the processor 102 executes the maximum value calculation program to realize the maximum value calculation function. Furthermore, the processor 102 executes the designated information acquisition program to realize the designated information acquisition function.

[0201] The medical image processing method applied to the medical information processing apparatus 20D according to the third embodiment adds a connected region labeling process executed by the connected region labeling unit 50, a maximum value calculation process executed by the maximum value calculation unit 52, and a designated information acquisition process executed by the designated information acquisition unit 54 to the flowchart shown in FIG. 4. In the composite image generation step S34, character information T2 representing the maximum value of the confidence level for each disease in the connected region to which the pixel specified by the user operating the mouse belongs is generated. In the composite image display step S36, the integrated finding image MII3 including the character information T2 is displayed.

[0202] [Operation and Effect of the Third Embodiment] According to the medical information processing apparatus 20D according to the third embodiment, visualization of the disease name and the maximum value of the confidence level for each disease is realized for the connected region to which the pixel specified by the user operating the mouse belongs.

[0203] [Configuration Example of Medical Information Processing Apparatus According to the Fourth Embodiment] FIG. 12 is a functional block diagram showing the electrical configuration of the medical information processing apparatus according to the fourth embodiment. The medical information processing apparatus 20E shown in the figure adds a designated information acquisition unit 54 and a comparison unit 56 to the medical information processing apparatus 20 shown in FIG. 2. The designated information acquisition unit 54 acquires information on the position in the medical image specified by the user operating the mouse.

[0204] The comparison unit 56 compares the confidence level for each disease with the user threshold for the pixel specified by the user, and determines whether the confidence level for each disease exceeds the user threshold. When the confidence level for each disease of the specified pixel exceeds the user threshold, the comparison unit 56 transmits the confidence level for each disease in the specified pixel to the superimposed information generation unit 40.

[0205] The superimposed information generation unit 40 generates character information representing the confidence level for each disease in the pixel specified by the user. The medical information processing apparatus 20E may include a character information generation unit that generates character information representing the confidence level for each disease in the pixel specified by the user separately from the superimposed information generation unit 40.

[0206] The display device 44 acquires a display signal representing an integrated image of all findings with character information indicating the confidence level for each disease at the pixel specified by the user superimposed on a medical image, and displays the integrated image of all findings with character information indicating the confidence level for each disease at the pixel specified by the user.

[0207] When the pixel specified by the user has a confidence level exceeding the user threshold for a plurality of diseases, the superimposed information generation unit 40 may generate character information indicating the confidence level for each of the plurality of diseases.

[0208] As an example of the character information indicating the confidence level for each disease at the pixel specified by the user, there is the character information T2 illustrated in FIG. 11. The character information may be arranged at a position in contact with the cursor CU or at a position separated from the cursor CU.

[0209] The hardware configuration of the electrical configuration of the medical information processing apparatus 20E according to the fourth embodiment is the same as the hardware configuration of the electrical configuration of the medical information processing apparatus 20 illustrated in FIG. 3. In the medical information processing apparatus 20E, the designation information acquisition program and the comparison program are stored in the memory 112 illustrated in FIG. 3. The processor 102 executes the designation information acquisition program to realize the designation information acquisition function. The processor 102 executes the comparison program to realize the comparison function.

[0210] In the medical information processing method applied to the medical information processing apparatus 20E according to the fourth embodiment, a designation information acquisition step executed by the designation information acquisition unit 54 and a comparison step executed by the comparison unit 56 are added to the flowchart illustrated in FIG. 4. In the composite image generation step S34, character information T2 indicating the confidence level for each disease at the pixel specified by the user is generated. In the composite image display step S36, the integrated image of all findings MII3 including the character information T2 is displayed.

[0211] [Operation and Effect of the Fourth Embodiment] According to the medical information processing apparatus 20E according to the fourth embodiment, for a pixel specified by a user operating a mouse, when the confidence level exceeds the user threshold, visualization of at least one of the disease name and the confidence level for each disease is realized.

[0212] [Configuration example of medical information processing apparatus according to fifth embodiment] FIG. 13 is a functional block diagram showing the electrical configuration of the medical information processing apparatus according to the fifth embodiment. The medical information processing apparatus 20F according to the fifth embodiment is applied when a user cannot operate an input device 16A such as a mouse.

[0213] The medical information processing apparatus 20F has a connected region labeling processing unit 50 added to the medical information processing apparatus 20 illustrated in FIG. 2. The medical information processing apparatus 20F assigns a label for each disease to each connected region using the connected region labeling processing unit 50 for a connected region where the confidence value for each disease exceeds the user threshold.

[0214] The connected region labeling processing unit 50 stores, in association with the label for each connected region, at least one of the disease name and the confidence level for each disease for each connected region. In FIG. 13, illustration of the memory in which at least one of the disease name and the confidence level for each disease for each connected region is stored is omitted.

[0215] The superimposed information generation unit 40 generates character information representing at least one of the disease name and the confidence level for each disease for each connected region. The synthesizing unit 42 generates a synthesized image in which character information representing at least one of the disease name and the confidence level for each disease for each connected region is superimposed on the medical image MI. The synthesizing unit 42 transmits a signal representing the generated synthesized image to the display device 44. The display device 44 displays the synthesized image.

[0216] FIG. 14 is a schematic diagram of a medical image on which the disease name and the confidence level for each disease for each connected region are superimposed. The overall findings integrated image MII5 illustrated in the figure has character information T51, character information T52, and character information T53 superimposed on the medical image MI.

[0217] The character information T51 includes character information representing at least one of the disease name and the confidence level in the connected region detected as the disease region of the nodule or infiltration shadow. The character information T52 includes character information representing at least one of the disease name and the confidence level in the connected region detected as the disease region of the pneumothorax. The character information T53 includes character information representing at least one of the disease name and the confidence level in the connected region detected as the disease region of the pleural effusion.

[0218] In addition, in FIG. 14, the specific illustration of the disease name and the confidence level in each of the character information T51, the character information T52, and the character information T53 is omitted, and the background of the character information T51, etc. is illustrated.

[0219] FIG. 15 is a schematic diagram showing a modified example of the character information illustrated in FIG. 14. In the overall findings integrated image MII51 illustrated in the figure, the character information T51, the character information T52, and the character information T53 are arranged outside the region of the medical image MI. The character information T51, the character information T52, and the character information T53 may be arranged at a position superimposed on the medical image MI as long as it is outside the region of the examination target site in the medical image MI. For example, when the medical image MI is a chest X-ray image, the character information T51, the character information T52, and the character information T53 may be arranged outside the region of the lung field. Note that the arrangement of the character information T51, the character information T52, and the character information T53 described in the embodiment is an example outside the detection target region in the medical image.

[0220] FIG. 16 is a schematic diagram showing another modified example of the character information illustrated in FIG. 14. In the overall findings integrated image MII52 illustrated in the figure, the character information T512, the character information T522, and the character information T532 are superimposed on the medical image MI.

[0221] The character information T512 includes all of the confidence levels for each disease in the connected region detected as the disease region of the nodule or infiltration shadow. The character information T522 includes all of the confidence levels for each disease in the connected region detected as the disease region of the pneumothorax. The character information T532 includes all of the confidence levels for each disease in the connected region detected as the disease region of the pleural effusion.

[0222] In addition, in FIG. 15, specific illustrations of the confidence levels and the like in each of the character information T512, the character information T522, and the character information T532 are omitted, and the background of the character information T512 and the like is illustrated.

[0223] The hardware configuration of the electrical configuration of the medical information processing apparatus 20F according to the fifth embodiment is the same as the hardware configuration of the electrical configuration of the medical information processing apparatus 20 illustrated in FIG. 3. In the medical information processing apparatus 20F, the connected region labeling processing program applied to the connected region labeling processing unit 50 is stored in the memory 112 illustrated in FIG. 3. The processor 102 executes the connected region labeling processing program to realize the connected region labeling processing function.

[0224] The medical information processing method applied to the medical information processing apparatus 20F according to the fifth embodiment adds a connected region labeling processing step executed by the connected region labeling processing unit 50 to the flowchart illustrated in FIG. 4. In the composite image generation step S34, the character information T51 and the like illustrated in FIG. 14 are generated. In the composite image display step S36, an overall findings integrated image MII5 and the like including the character information T51 and the like are displayed.

[0225] [Operational Effects of the Fifth Embodiment] According to the medical information processing apparatus 20F according to the fifth embodiment, when the user cannot operate the mouse, visualization of at least one of the disease name and the confidence level for each disease is realized for the connected region whose confidence level exceeds the user threshold.

[0226] [Configuration Example of Medical Information Processing Apparatus According to the Sixth Embodiment] FIG. 17 is a functional block diagram showing the electrical configuration of the medical information processing apparatus according to the sixth embodiment. The medical information processing apparatus 20G illustrated in the figure has a portion where different disease regions overlap, and when the overlap rate for each disease exceeds a specified value, integrated character information in which the disease names of the overlapping disease regions are integrated is displayed.

[0227] The medical information processing device 20G has a connected region labeling processing unit 50 and an overlap ratio determination unit 58 added to the medical information processing device 20 illustrated in FIG. 2. The connected region labeling processing unit 50 is a processing unit similar to the connected region labeling processing unit 50 illustrated in FIG. 10, and assigns a label to a region where the confidence level for each disease exceeds the user threshold value.

[0228] When regions of different diseases overlap, the overlap ratio determination unit 58 determines whether the overlap ratio for each disease exceeds a specified value. The overlap ratio determination unit 58 transmits a signal representing the determination result to the superimposed information generation unit 40.

[0229] The overlap ratio for each disease may be the Dice coefficient that takes a value in the range from 0 to 1.0. The connected region labeling processing unit 50 may include an overlap ratio calculation unit that calculates the overlap ratio. The medical information processing device 20G may include an overlap ratio calculation unit that calculates the overlap ratio separately from the connected region labeling processing unit 50.

[0230] The specified value applied to the determination of the overlap ratio functions as a determination threshold value for the overlap ratio. Any value greater than 0 and less than 1.0 may be applied as the specified value. For example, any value greater than or equal to 0.5 and less than 1.0 may be applied as the specified value.

[0231] The specified value may be defined according to the area of the disease region. When the area of the disease region is relatively large, the specified value may be made relatively large, and when the area of the disease region is relatively small, the specified value may be made relatively small.

[0232] When the overlap ratio for each disease exceeds the specified value, the superimposed information generation unit 40 generates integrated character information in which character information representing the disease name of the disease event is integrated. The synthesizing unit 42 generates an overall findings integrated image in which the integrated character information is superimposed on the medical image MI. The display device 44 displays the overall findings integrated image.

[0233] The overlapping information generation unit 40 may generate integrated character information when the overlap rate exceeds a specified value for all diseases among a plurality of diseases. The overlapping information generation unit 40 may also generate integrated character information when the overlap rate exceeds a specified value for at least one disease among a plurality of diseases.

[0234] FIG. 18 is a schematic diagram of integrated character information. In the overall finding integrated image MII6 shown in the figure, the disease area DA1 of pleural effusion and the disease area DA2 of pneumothorax overlap, and the integrated character information T63 generated in place of the character information T61 and the character information T62 representing each disease is superimposed on the medical image MI.

[0235] The arrow line AL1 from the character information T61 to the disease area DA1 of pleural effusion indicates that the character information T61 corresponds to the disease area DA1 of pleural effusion. Also, the arrow line AL2 from the character information T62 to the disease area DA2 of pneumothorax indicates that the character information T62 corresponds to the disease area DA2 of pneumothorax.

[0236] The integrated character information T63 combines the pleural effusion of the character information T61 and the pneumothorax of the character information T2. The arrow line AL3 extending from the integrated character information T63 points to the overlapping area OA between the disease area DA1 of pleural effusion and the disease area DA2 of pneumothorax.

[0237] FIG. 18 illustrates a case where two disease areas DA1 and DA2 with different diseases overlap each other. However, when three or more disease areas overlap, integrated text information in which the text information of each of the three disease areas is integrated may be generated.

[0238] The hardware configuration of the electrical configuration of the medical information processing apparatus 20G according to the sixth embodiment is the same as the hardware configuration of the electrical configuration of the medical information processing apparatus 20 illustrated in FIG. 3. In the medical information processing apparatus 20G, the connected region labeling processing program applied to the connected region labeling processing unit 50 and the overlap rate determination program applied to the overlap rate determination unit 58 are stored in the memory 112 illustrated in FIG. 3.

[0239] The processor 102 executes a connected region labeling processing program to realize a connected region labeling processing function. Further, the processor 102 executes an overlap ratio determination program to realize an overlap ratio determination function.

[0240] The medical image processing method applied to the medical information processing apparatus 20G according to the sixth embodiment adds a connected region labeling processing step executed by the connected region labeling processing unit 50 and an overlap ratio determination step executed by the overlap ratio determination unit 58 to the flowchart illustrated in FIG. 4. In the composite image generation step S34, integrated character information T63 corresponding to a plurality of disease regions is generated. In the composite image display step S36, the overall findings integrated image MII6 including the integrated character information T63 is displayed.

[0241] [Operational Effects of the Sixth Embodiment] According to the medical information processing apparatus 20G according to the sixth embodiment, when a plurality of disease regions with different diseases overlap each other, the visibility of the disease name for each disease can be improved.

[0242] [Configuration Example of Medical Information Processing Apparatus According to the Seventh Embodiment] FIG. 19 is a functional block diagram showing the electrical configuration of the medical information processing apparatus according to the seventh embodiment. The medical information processing apparatus 20H illustrated in the figure switches between the visualization display of the heat map image illustrated in FIG. 6 and the display of the heat map image for each disease illustrated in FIG. 5 according to the input information of the user.

[0243] That is, the medical information processing apparatus 20H adds an input information acquisition unit 60 and a display switching unit 62 to the medical information processing apparatus 20 illustrated in FIG. 2. The input information acquisition unit 60 acquires a signal representing the input information input by the user using the input device 16A illustrated in FIG. 1. The display switching unit 62 switches the display of the display device 44 according to the input information acquired using the input information acquisition unit 60.

[0244] The hardware configuration of the electrical configuration of the medical information processing device 20H according to the seventh embodiment is the same as the hardware configuration of the electrical configuration of the medical information processing device 20 illustrated in FIG. 3. In the medical information processing device 20H, the input information acquisition program applied to the input information acquisition unit 60 and the display switching program applied to the display switching unit 62 are stored in the memory 112 illustrated in FIG. 3.

[0245] The processor 102 executes the input information acquisition program to realize the input information acquisition function. Further, the processor 102 executes the display switching program to realize the display switching function.

[0246] In the medical image processing method applied to the medical information processing device 20H according to the seventh embodiment, an input information acquisition step executed by the input information acquisition unit 60 and a display switching step executed by the display switching unit 62 are added to the flowchart illustrated in FIG. 4.

[0247] [Operation and Effect of the Seventh Embodiment] According to the medical information processing device 20H according to the seventh embodiment, the user can selectively switch between the visualization display of the heat map image and the display of the heat map image for each disease.

[0248] [Function for Assisting in Creating a Multi-Findings Report] The medical information processing device 20 and the like may be provided with a function for assisting in creating a multi-findings report. A multi-findings report is a report in which a plurality of findings included in one medical image are described. As an example of assisting in creating a multi-findings report, an example is cited in which the character information generated using the medical information processing device 20 is automatically input into the multi-findings report.

[0249] Medical information processing apparatuses 20 and the like include an anatomical structure information acquisition unit that acquires anatomical structure information from a medical image MI, a confidence value maximum acquisition unit that acquires the maximum value of the confidence value for each disease with respect to regions for each anatomical structure, and a determination unit that determines whether the maximum value of the confidence value exceeds a specified threshold for each disease in a region for each anatomical structure. The confidence value maximum acquisition unit and the determination unit may be applied to the disease region detection unit 32 illustrated in FIG. 2.

[0250] Medical information processing apparatuses 20 and the like include a character information generation unit that generates character information representing a combination of an anatomical structure name, a disease name, and the maximum value of the confidence value when the maximum value of the confidence value exceeds a specified threshold for each disease in a region for each anatomical structure. The generated character information is stored so as to be searchable using the anatomical structure name and the disease name as search keys.

[0251] Medical information processing apparatuses 20 and the like include a character information transmission unit that transmits, to the user terminal device 16, character information representing a combination of an anatomical structure name, a disease name, and the maximum value of the confidence value in response to a request transmitted from the user terminal device 16 illustrated in FIG. 1. The character information generation unit and the character information transmission unit may be applied to the superimposed information generation unit 40.

[0252] [Other examples of diseases to be detected] In the disease region detection unit 32 illustrated in FIG. 2 and the like, nodules, infiltrative shadows, pneumothorax, and pleural effusion are detected as diseases, but the disease region detection unit 32 may detect tumors, tuberculosis, and the like as diseases.

[0253] The functions of the medical information processing apparatus 20 illustrated in FIG. 1 may be provided in the user terminal device 16. That is, the various processing units of the medical information processing apparatus 20 illustrated in FIG. 2 and the like may be provided in the user terminal device 16. The user terminal device 16 includes a memory 112 in which various programs illustrated in FIG. 3 are stored, and a processor may execute the various programs to realize various functions.

[0254] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined among the embodiments without departing from the gist of the present invention.

Explanation of Signs

[0255] 10 Medical information processing system 12 Medical image capturing device 14 Medical image database 16 User terminal device 16A Input device 16B Display device 18 Interpretation report database 20 Medical information processing device 20C Medical information processing device 20D Medical information processing device 20E Medical information processing device 20F Medical information processing device 20G Medical information processing device 20H Medical information processing device 22 Network 30 Medical image acquisition unit 32 Disease area detection unit 34 Heat map image generation unit 36 User threshold acquisition unit 38 Classification unit 40 Overlay information generation unit 42 Synthesis unit 44 Display device 48 Comparison unit 50 Labeling processing unit 52 Maximum value calculation unit 54 Designated information acquisition unit 56 Comparison unit 58 Overlap rate determination unit 60 Input upper information acquisition unit 62 Display switching unit 102 Processor 104 Computer-readable medium 106 Communication interface 108 Input / output interface 110 Bus 112 Memory 114 Storage 120 Medical Image Acquisition Program 122 Disease Region Detection Program 124 Heatmap Image Generation Program 126 User Threshold Acquisition Program 128 Classification Program 130 Overlay Information Generation Program 132 Synthesis Program 134 Confidence Level Acquisition Program AL1 Arrow Line AL2 Arrow Line AL3 Arrow Line CI1 Composite Image CI2 Composite Image CI3 Composite Image CU Cursor DA1 Disease Region DA2 Disease Region HM1 Heatmap Image HM2 Heatmap Image HM3 Heatmap Image MI Medical Image MII Comprehensive Findings Integrated Image MII2 Comprehensive Findings Integrated Image MII3 Comprehensive Findings Integrated Image MII5 Comprehensive Findings Integrated Image MII6 Comprehensive Findings Integrated Image MII51 Comprehensive Findings Integrated Image MII52 Comprehensive Findings Integrated Image OA Overlap Region SI1 First Overlay Region SI2 First Overlay Region SI3 First Overlay Region SI21 Second Overlay Region SI22 Second Overlay Region SI23 Second Overlay Region T1 Character Information T2 Character Information T51 Character Information T52 Character Information T53 Character Information T61 Character Information T62 Character Information T63 Character Information T512 Character Information T522 Character Information T532 Character Information Steps S10 to S36 of the Medical Information Processing Method

Claims

1. One or more processors, One or more memories storing instructions to be executed by the one or more processors, Comprising: The one or more processors: Acquire a medical image, Using a multi-disease detection model that detects a plurality of disease regions from the medical image, for each pixel of the medical image, obtain a confidence level representing at least one of the presence or absence of a disease and the degree of the disease for each disease, For each of the detected diseases, create confidence distribution information for visualizing the distribution of the confidence levels, Obtain a threshold value applied to the confidence level, For any one of the diseases, perform control to superimpose at least one of a first color and a first opacity on the medical image and display it on a display device for pixels of a first image having the confidence level equal to the threshold value. An information processing apparatus.

2. The one or more processors: Obtain a threshold range that is an upper limit value exceeding the threshold value, a pre-defined upper limit value, and a lower limit value less than the threshold value, a pre-defined lower limit value, Classify pixels having the confidence level within the threshold range as pixels of the first image. The information processing apparatus according to claim 1.

3. The one or more processors classify pixels whose distance from pixels classified as pixels of the first image is within a specified range as pixels of the first image. The information processing apparatus according to claim 1.

4. For any one of the diseases, the one or more processors perform control to superimpose and display at least one of a second color corresponding to the maximum value of the confidence level and a second opacity corresponding to the maximum value of the confidence level on the medical image for pixels of a second image whose confidence level exceeds the confidence level of the pixels of the first image. The information processing apparatus according to any one of claims 1 to 3.

5. When pixels adjacent to the pixels of the second image are classified as pixels of the first image, the one or more processors set a third opacity lower than the first opacity for the pixels of the second image. The information processing apparatus according to claim 4.

6. For any one of the diseases, the one or more processors perform control to display the medical image with the color and opacity for pixels of a third image whose confidence level is less than the confidence level of the pixels of the first image as non-overlapping. The information processing apparatus according to any one of claims 1 to 3.

7. The one or more processors: Obtain the maximum value of the confidence level for each of the diseases, Determine whether the maximum value of the confidence level for each of the diseases exceeds the threshold, When the maximum value of the confidence level exceeds the threshold, perform control to display text representing the disease name and the maximum value of the confidence level. The information processing apparatus according to any one of claims 1 to 3.

8. The one or more processors Perform labeling processing on one or more connected regions including a plurality of consecutive pixels for which the value of the confidence level for each of the diseases exceeds the threshold, For each of the connected regions for each of the diseases, determine whether the maximum value of the confidence level exceeds the threshold, Receive an input from the user designating the connected region, Perform control to display the maximum value of the confidence level for each of the diseases in the designated connected region. The information processing apparatus according to any one of claims 1 to 3.

9. The one or more processors Receive an input from the user designating a pixel in the medical image, When the confidence level for each of the diseases in the designated pixel exceeds the threshold, perform control to display the value of the confidence level. The information processing apparatus according to any one of claims 1 to 3.

10. The one or more processors Perform labeling processing on one or more connected regions including a plurality of consecutive pixels for which the value of the confidence level for each of the diseases exceeds the threshold, For each of the connected regions for each of the diseases, perform control to display at least one of the disease name and the confidence level of the connected region. The information processing apparatus according to any one of claims 1 to 3.

11. The one or more processors perform control to display at least one of the disease name and the confidence level of each connected region outside the detection target region in the medical image. The information processing apparatus according to claim 10.

12. The one or more processors Perform labeling processing on one or more connected regions including a plurality of consecutive pixels for which the confidence level for each of the diseases exceeds the threshold, Perform control to display all of the confidence levels for each of the diseases for each of the connected regions. The information processing apparatus according to any one of claims 1 to 3.

13. The one or more processors Perform labeling processing on one or more connected regions including a plurality of consecutive pixels for which the confidence level for each of the diseases exceeds the threshold, For the connection area for each disease, when the overlapping part for each disease exceeds a specified value, perform control to display character information obtained by integrating a plurality of disease names in the overlapping part. The information processing apparatus according to any one of claims 1 to 3.

14. The one or more processors replace the confidence value of the pixel constituting the confidence distribution information with the maximum confidence value in a local area located within a specified range from the specified pixel. The information processing apparatus according to any one of claims 1 to 3.

15. The one or more processors receive an input from a user to change the control to display the confidence distribution information. The information processing apparatus according to any one of claims 1 to 3.

16. A computer functioning as an information processing apparatus acquires a medical image, using a multi-disease detection model that detects a plurality of disease regions from the medical image, obtains, for each pixel of the medical image, a confidence representing at least one of the presence or absence of a disease and the degree of the disease for each disease, creates confidence distribution information for visualizing the distribution of the confidence for each of the detected diseases, obtains a threshold value applied to the confidence, for any one of the diseases, performs control to superimpose at least one of a first color and a first opacity on the medical image and display it on a display device for pixels of a first image having the same value of the confidence as the threshold value. A method of operating an information processing apparatus that executes the above.

17. A computer functioning as an information processing apparatus has a function of acquiring a medical image, using a multi-disease detection model that detects a plurality of disease regions from the medical image, has a function of obtaining, for each pixel of the medical image, a confidence representing at least one of the presence or absence of a disease and the degree of the disease for each disease, has a function of creating confidence distribution information for visualizing the distribution of the confidence for each of the detected diseases, has a function of obtaining a threshold value applied to the confidence, A program that realizes a function of performing control to superimpose at least one of a first color and a first opacity on the medical image and display it on a display device for pixels of a first image having the same value of the confidence as the threshold value for any one of the diseases.

18. One or more processors and one or more memories storing instructions to be executed by the one or more processors, comprising The one or more processors: acquire medical images, for each pixel of the medical images, obtain, for each disease, a confidence level representing at least one of the presence or absence of the disease and the degree of the disease, using a multi-disease detection model that detects a plurality of disease regions from the medical images, obtain a threshold value applied to the confidence level, obtain information on regions for each anatomical structure from the medical images, for each region for each anatomical structure, obtain the maximum value of the confidence level for each disease, an information processing apparatus that stores a combination of an anatomical structure name, a disease name, and the maximum value of the confidence level when the maximum value of the confidence level for each disease in the region for each anatomical structure exceeds the threshold value.

Images