Image processing device, image processing system, image processing method, and program

Abstract

To provide users with appropriate information to support their decision regarding the presence of a lesion. [Solution] The image processing device includes a medical image acquisition unit that acquires a medical image, an inference unit that infers the probability of the presence of a lesion from the medical image and acquires a first inference result, and extracts the region where the lesion exists from the medical image and acquires a second inference result, an inference basis information acquisition unit that acquires inference basis information which is information that influenced the acquisition of the first inference result, and a display target determination unit that determines at least one of the inference basis information and the region based on the second inference result as a display target based on at least one of the first inference result and the second inference result, determines the information based on the second inference result as a display target if the second inference result includes information indicating the presence of a lesion, and determines the inference basis information as a display target if the second inference result does not include information indicating the presence of a lesion and the first inference result includes information indicating the presence of a lesion.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program.

Background Art

[0002] A method of presenting the basis for judgment of machine learning represented by deep learning to a user is known. For example, in the method disclosed in Patent Document 1, a task of inferring and classifying the probability of the presence of a lesion in an image is executed, the contribution rate of the feature amount in the image serving as the basis for the judgment is obtained, and this is presented to the user. Also, in a task of extracting the region of a lesion in an image, typified by semantic segmentation, it is known that the region extracted by machine learning indicates the basis for judgment.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the method described in Patent Document 1, if the lesion in the image is difficult to see during the region extraction task, the region extraction itself may fail, and it may not be possible to provide appropriate information to support the decision.

[0006] This invention is made in view of the above circumstances, and one of its objectives is to provide users with suitable information that suggests the presence of a lesion. [Means for solving the problem]

[0007] To solve the above problems, an image processing apparatus according to one aspect of the present invention is: A medical image acquisition unit that acquires medical images, An inference unit obtains a first inference result by performing a first task of inferring the probability of the presence of a lesion on the medical image, and obtains a second inference result by performing a second task, which is different from the first task, of extracting the region where the lesion is located on the medical image. An inference basis information acquisition unit acquires inference basis information, which is information that influenced the inference unit when it obtained the first inference result. The system includes a display target determination unit that determines, based on at least one of the first inference result and the second inference result, to be a display target, and the inference basis information and at least one of the region based on the second inference result. The display target determination unit determines the information based on the second inference result as the display target if the second inference result includes information indicating the presence of a lesion, and determines the inference basis information as the display target if the second inference result does not include information indicating the presence of a lesion, and the first inference result includes information indicating the presence of a lesion. [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to provide the user with suitable information that suggests the presence of a lesion. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example configuration of the image processing system in the first embodiment. [Figure 2] This flowchart shows an example of the overall image processing in the first embodiment. [Figure 3] This figure illustrates an example of the overall process in the first embodiment. [Figure 4] This diagram illustrates an example of a method for obtaining inference basis information in the first embodiment. [Figure 5] This diagram illustrates screen displays that vary depending on the target object being displayed. [Figure 6] This is a block diagram showing an example configuration of an image processing system in the second embodiment. [Figure 7] This flowchart shows an example of the overall image processing in the second embodiment. [Figure 8] This is an example of a generated SPR image. [Figure 9] This is a block diagram showing an example configuration of an image processing system in the third embodiment. [Figure 10] This diagram illustrates a screen display that includes report generation results tailored to the displayed items. [Figure 11] This flowchart shows an example of the overall image processing in the third embodiment. [Modes for carrying out the invention]

[0010] Hereinafter, an exemplary embodiment of an image processing apparatus, an image processing method, and an image processing system for implementing the disclosures of this specification will be described with reference to the drawings. The dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be modified according to the configuration of the apparatus to which this disclosure applies or various conditions. Furthermore, identical or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals across the drawings, and redundant descriptions will be omitted as appropriate. Additionally, some components, members, and processes will be omitted from the drawings as appropriate.

[0011] In the following embodiments, the present invention will be described using, as an example of medical image data, CT image data obtained by an X-ray computed tomography (CT) apparatus. However, the present invention is not limited to the following exemplary embodiments, and can also be applied to images obtained by, for example, a magnetic resonance imaging (MRI) apparatus, a positron emission tomography (PET) apparatus, or an ultrasonic diagnostic apparatus.

[0012] <First Embodiment> In the image processing apparatus according to the first embodiment, an area extraction process for extracting an area suspected of being a lesion in the input image from the input image input to the image processing apparatus can be executed. In this embodiment, a CT image is used as an example of the input image, and the case where pancreatic cancer is targeted as the lesion in the image is described. That is, in the image processing apparatus of this embodiment, an area extraction process, which is a process for extracting an area suspected of pancreatic cancer in the CT image, is executed.

[0013] Next, in the image processing apparatus, an area extraction result determination process for determining whether information suggesting a lesion is included in the image as a result of the area extraction process can be executed. When it is determined by the area extraction result determination process that information suggesting a lesion is included in the image, a display target determination process for determining, as a display target, to present the area extracted by the area extraction process to the user by a display unit such as a monitor can be executed.

[0014] In addition, in this image processing apparatus, when it is determined by the region extraction result determination process that the result of the region extraction process does not include information suggesting a lesion, an existence probability inference process for inferring the probability (e.g., likelihood) that a lesion exists in the image can be executed. Then, based on the result of the existence probability inference process, it is determined whether a lesion is included in the image. As a result, when it is determined that a lesion is included, an inference basis information acquisition process for acquiring information on the inference basis, which is information that influenced the existence probability inference process, can be executed. Then, a display target determination process for determining the obtained inference basis information as a display target to be presented to the user by a monitor or the like can be executed.

[0015] Note that in the following embodiments, pancreatic cancer is exemplified as the processing target. However, the site or lesion targeted by the present invention is not limited to this example, and even for other organs or diseases, such as diseases of various organs such as the liver and kidneys, the essential effects of the present invention are not lost. Also, regarding the lesion, as long as it is a lesion other than the exemplified cancer and it is possible to know the extraction of the lesion region and the possibility of the existence of the lesion by an image, it can be a lesion extractable by the present invention.

[0016] Hereinafter, the configuration of the image processing apparatus and the image processing system according to this embodiment, and the processes executed in the image processing apparatus will be described with reference to FIG. 1. FIG. 1 is a block diagram showing an example of the overall configuration of an image processing system 101 including an image processing apparatus 111 according to this embodiment. As shown in FIG. 1, the image processing system 101 includes an image processing apparatus 111, a communication unit 121, a storage unit 131, an operation unit 141, a display processing unit 151, and a display unit 161 according to this embodiment. Also, the image processing system 101 can be connected to an external database 102 via the communication unit 121.

[0017] The image processing device 111 may consist of a CPU (Central Processing Unit) or a dedicated or general-purpose processor. The image processing device 111 may also consist of a GPU (Graphic Processing Unit) or an FPGA (Field-Programmable Gate Array). Alternatively, the image processing device 111 may consist of an ASIC (Application Specific Integrated Circuit) or the like. The image processing device 111 controls each part of the image processing system 101 and executes the processing described below to realize the image processing method related to the technology disclosed herein.

[0018] The image processing apparatus 111 according to this embodiment includes an image acquisition unit 112, a region extraction unit 113, a region extraction result determination unit 114, an existence probability inference unit 115, a determination unit 116, an inference basis information acquisition unit 117, a display target determination unit 118, and an output unit 119. The following describes each part of the image processing apparatus 111.

[0019] The image acquisition unit 112 can acquire the image to be inferred from, for example, the memory unit 131. The image to be inferred is, for example, an image of the subject (patient) acquired by a CT scanner. In this embodiment, as an example of the image to be inferred, a contrast-enhanced CT image of the pancreatic parenchymal phase of the abdominal region, approximately 40 to 50 seconds after injection of contrast agent, is acquired. However, the contrast-enhanced phase of the CT image is not limited to the pancreatic parenchymal phase described above; the effects of the present invention can also be expected in other phases such as the portal venous phase, arterial phase, equilibrium phase, and late phase. Furthermore, a non-contrast-enhanced CT image may be used as the image to be inferred. The image acquisition unit 112 may also acquire the image to be inferred directly from a diagnostic device such as a CT scanner (not shown), in which case the image processing system 101 may be implemented in the diagnostic device as part of the functions of the diagnostic device.

[0020] The region extraction unit 113 can extract region information suggestive of pancreatic cancer from the image acquired by the image acquisition unit 112. The process of extracting region information may be performed using a computational model implemented in the image processing system 101 or the image processing device 111, or the model structure and parameters of the computational model may be stored in the database 102 or the storage unit 131 and retrieved for execution. As an example, the region information is a binary label image in which the pancreatic cancer region is 1 and other regions are 0, but if the values ​​differ for each region, these values ​​can be freely set. Note that the binary label image is only one example of region information, and the region information may take any form as long as it indicates a location suggestive of pancreatic cancer. For example, a pancreatic cancer likelihood image in which the value of pancreatic cancer likelihood is stored for each pixel may be used as region information, or the coordinates of a representative location of pancreatic cancer (for example, the coordinates of the centroid or the coordinates of the rectangle surrounding the pancreatic cancer) may be used as region information.

[0021] The region extraction result determination unit 114 can determine whether or not pancreatic cancer is present in the image acquired by the image acquisition unit 112, based on the region information suggesting pancreatic cancer extracted by the region extraction unit 113. Details of the actual determination process will be described later.

[0022] The probability inference unit 115 can infer the probability value (likelihood) of the presence of pancreatic cancer from the image acquired by the image acquisition unit 112. The probability value calculated by the inference can be obtained as a continuous value between 0 and 1, for example, with 1 representing a high probability of the presence of pancreatic cancer and 0 representing a low probability. This inference process may be performed using a computational model implemented in the image processing system 101, similar to the region extraction unit 113. Alternatively, the model structure and parameters of the computational model may be stored in the database 102 or the storage unit 131, and then retrieved and executed.

[0023] The determination unit 116 can determine whether or not pancreatic cancer is present in the image acquired by the image acquisition unit 112, based on the inference result inferred by the probability inference unit 115. Details of the actual determination process will be described later.

[0024] The inference basis information acquisition unit 117 acquires inference basis information, which is information that influenced the inference process of the existence probability inference unit 115, from the image acquired by the image acquisition unit 112 and the inference results output by the existence probability inference unit 115. The information that influenced the inference process described here is information that suggests the presence of pancreatic cancer, and image features can be cited as an example. In this embodiment, image features include, for example, information on the pancreatic cancer region in the image, information indicating the location of occlusion of the pancreatic duct, which is an indirect finding that suggests pancreatic cancer, or information indicating atrophy of a part or the whole of the pancreas.

[0025] The display target determination unit 118 determines the display target according to the following conditions, for example: If the region extraction result determination unit 114 determines that pancreatic cancer is included in the region information suggesting pancreatic cancer extracted by the region extraction unit 113, it determines that region information to be displayed. On the other hand, if the region extraction result determination unit 114 determines that pancreatic cancer is not included in the region information, and the determination unit 116 determines that pancreatic cancer is present, it determines that the inference basis information obtained by the inference basis information acquisition unit 117 to be displayed.

[0026] The output unit 119 outputs the display target determined by the display target determination unit 118 to the display processing unit 151, causing the display unit 161 to display it. The determined display target may also be output to the database 102 or the storage unit 131 and stored there.

[0027] Next, in this embodiment, components other than the image processing device 111 that may be included in the image processing system 101 will be described. The communication unit 121 enables the image processing system 101 to be connected to external devices and networks, and enables communication by predetermined communication means. The communication unit 121 may be composed of wireless devices such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or wired devices such as wired LAN (Local Area Network) or USB (Universal Serial Bus). The communication unit 121 communicates with an external database or the like by predetermined communication means, enabling the image processing system 101 to acquire one or more images, region information, and other various data.

[0028] The storage unit 131 consists of one or more data recording media, such as an HDD (Hard Disk Drive) or RAM (Random Access Memory), and is used for storing various types of data and various calculation results. The storage unit 131 may be configured by separating a main memory consisting of volatile memory for temporarily storing read data and the like, from an auxiliary storage unit for long-term data storage.

[0029] The operation unit 141 can consist of input devices such as a keyboard, mouse, touch panel, or remote control, and can be used to input user instructions to various devices. Furthermore, the operation unit 141 can be integrated with, for example, the display unit 161 (described later), and the display screen of the display unit 161 can be used as a touch panel.

[0030] The display processing unit 151 processes images and various calculation results output from the image processing device 111 in a format that can be displayed by the display unit 161, and can send the processing results to the display unit 161. The display unit 161 consists of output devices such as monitors and displays, and can be used to display data such as calculation results and various images processed by the display processing unit 151 to the user.

[0031] In this embodiment, the database 102, which is connected to the image processing system 101, stores, for example, images used by the image processing system 101. The database 102 is connected to the image processing system 101 via the communication unit 121 described above, and transmits the stored information to the image processing device 111 in response to requests from the image processing system 101. It can also store the results calculated by the image processing system 101 and output images.

[0032] In the image processing system 101 illustrated in Figure 1, the image processing device 111, the communication unit 121, the storage unit 131, the operation unit 141, the display processing unit 151, and the display unit 161 are shown as separate components. However, the image processing system 101 is not limited to the configuration example shown in Figure 1, and the image processing device 111 and some or all of the other components may be configured as a single unit. Furthermore, for example, some or all of the components shown separately from the image processing device 111 may be included in the image processing device 111.

[0033] Next, an example of a series of processes from image processing to image display actually performed in the image processing system 101 described above will be explained in detail using the flowchart in Figure 2. In the following explanation, it is assumed that the input image input to the image processing device 111 is stored in the storage unit 131 of the image processing system 101. For example, when a user inputs an instruction to start image processing via the operation unit 141, the image processing device 111 moves the flow to step S201 in order to execute the image processing method according to this embodiment.

[0034] (Step S201: Image acquisition) In step S201, the image acquisition unit 112 acquires an image specified by the user via the operation unit 141 from the storage unit 131 as the input image. The input image may be acquired by other means. For example, the system may be configured to search for and read an image that satisfies predetermined conditions from the database 102 in response to the user's instructions and then process it. In this case, the display processing unit 151 may display the acquired input image on the display unit 161. In this embodiment, a three-dimensional contrast-enhanced CT image taken with a CT device is given as an example of the input image, but as mentioned above, the input image is not limited to this and may be an image taken with another modality, a non-contrast CT image, a two-dimensional image, etc. It may also be a group of multi-phase images taken at multiple contrast-enhanced phases, in which case the system may further select a suitable phase to be used as the target for processing from step S202 onwards, or each of the images at multiple phases may be used as the input image and the processing from step S202 onwards may be executed. After the image acquisition is performed, the image processing device 111 moves the flow to step S202.

[0035] (Step S202: Extraction of lesion area) In step S202, the region extraction unit 113 extracts region information suggestive of pancreatic cancer from the input image acquired by the image acquisition unit 112, identifying it as a lesion region. In this embodiment, as an example of a method for extracting region information, semantic segmentation using a Convolutional Neural Network (CNN), a type of machine learning, is used. However, the method for extracting region information is not limited to this; other neural network methods such as Transformers using Self-Attention mechanisms, or other machine learning methods such as Random Forest, may also be used.

[0036] In this step, the extracted region information may take the form of a binary image, for example, where the pancreatic cancer region is represented as 1 and other regions as 0. However, the extraction method of region information is not limited to this; it may also be a likelihood image, where the pixel values ​​are continuous values ​​between 0 and 1, representing the likelihood of the region's existence. Alternatively, it may be a multi-level image resembling a Gaussian distribution with the center of the pancreatic cancer as the peak position. In this case, region information of the pancreas encompassing the pancreatic cancer may be further acquired, and the spread of the distribution may be modified based on the distance between the center of the pancreatic cancer and the contour of the pancreatic region. The region information may also be a bounding rectangle (3D cuboid) surrounding the pancreatic cancer. For example, it may be a binary image where the internal region of the bounding rectangle is represented as 1 and other regions as 0, or the coordinate values ​​of the vertex positions of the bounding rectangle may be used instead. It should be noted that the process of extracting region information in this step does not guarantee that region information will be extracted in all cases. That is, if the input image does not contain pancreatic cancer, or if the extraction process does not extract region information related to pancreatic cancer, the result may be that there is no region information.

[0037] Here, with reference to Figure 3, the details of the region extraction process using CNN will be explained. Figure 3 shows an example of pancreatic cancer region segmentation processing using CNN. In Figure 3, the input image 301, which is a CT image, shows the pancreatic region 302 and the pancreatic duct region (pancreatic duct 303). The input image 301 also contains the pancreatic cancer region 304. The region extraction unit 305 in this embodiment (corresponding to the region extraction unit 113 in Figure 1) is configured using a CNN and performs convolution processing multiple times on the input image 301 to obtain the region extraction result 306.

[0038] In this embodiment, the region extraction unit 305 is a CNN comprising an Encoder, which is a feature extraction unit, and a Decoder, which reconstructs an image such as a desired region mask from the extracted features. For example, it is a CNN such as U-net. The region extraction unit 305 is not limited to the example configuration, and may also be configured to execute only the feature extraction results from the Encoder and obtain the region extraction result 306 by converting the acquired features into the input image space by upsampling. The features in this embodiment are the result of performing a convolution operation on the image and include tensor data with a spatial distribution. When region extraction is completed, the image processing device 111 moves the flow to step S203.

[0039] (Step S203: Determination of the extracted region result) In step S203, the region extraction result determination unit 114 determines whether or not pancreatic cancer is present based on the region information suggesting pancreatic cancer extracted by the region extraction unit 113. For example, in the case of a likelihood image where the region information has continuous values ​​from 0 to 1 as pixel values, the likelihood image can be thresholded by a predetermined threshold (e.g., 0.5), and if there is one or more pixels exceeding the threshold, it can be determined that "pancreatic cancer is present in the image." However, when determining whether pancreatic cancer is present, the condition for the number of pixels exceeding the threshold is not limited to the above "one or more," and by changing the condition, it is possible to adjust the determination process to be highly sensitive or robust to noise. In addition, it is also possible to determine whether pancreatic cancer is present if the number of pixels in the maximum connected region of pixels exceeding the threshold (the largest region where pixels exceeding the threshold are continuous) is equal to or greater than a predetermined number of pixels (e.g., 100 pixels). Furthermore, the determination may be made based on the volume of the maximum connected region, for example, if the volume of the maximum connected region is 100 mm 3In the above cases, it may be determined that pancreatic cancer is present. The volume of the region can be calculated by multiplying the number of pixels by the size (volume) of one pixel. The pixel size may be obtained, for example, from the storage unit 131 or database 102 as information attached to the image, or it may be received from the user via the operation unit 141. The threshold used in thresholding may be obtained from the storage unit 131 or received from the user via the operation unit 141.

[0040] Furthermore, the region extraction result determination unit 114 can also determine whether or not pancreatic cancer is present using a method other than the threshold processing method. In other words, any method can be used to determine whether or not pancreatic cancer is present from the region information suggestive of pancreatic cancer extracted by the region extraction unit 113. For example, the determination may be made using a machine learning-based inference system that takes the region information as input and outputs whether or not pancreatic cancer is present.

[0041] In addition to the above determination process, the pixel value with the highest value among the pixel values ​​of the region extraction results may be stored as the probability of the presence of pancreatic cancer. In this case, in step S204 described later, the stored probability of presence can be included as part of the display target for the image determined to be displayed on the screen. If the above determination determines that pancreatic cancer is present in the image, the image processing device 111 proceeds to step S204. If it is determined that pancreatic cancer is not present, the process proceeds to step S205.

[0042] (Step S204: Determine the region extraction results to be displayed) In step S204, the display target determination unit 118 executes processing according to whether the region extraction result determination unit 114 determined in step S203 that pancreatic cancer is included in the image acquired by the image acquisition unit 112. Specifically, it determines the region information extracted by the region extraction unit 113 as the display target. After the determination, the image processing device 111 moves the flow to step S210.

[0043] (Step S205: Execution of existence probability inference process) In step S205, the existence probability inference unit 115 infers the probability (existence probability) of whether or not pancreatic cancer is present in the image acquired by the image acquisition unit 112. In this embodiment, as a result of the inference process, the existence probability inference unit 115 outputs the probability of pancreatic cancer being present as a continuous value between 0 and 1. In this case, the existence probability takes a value of 1 or close to 1 if the probability of pancreatic cancer being present is high, and a value of 0 or close to 0 if it is low.

[0044] Here, as a method for inferring the probability of existence, for example, a mechanism similar to the Encoder of the region extraction unit 113 used in step S202 can be used. That is, by performing processing by an Encoder that performs a convolution operation on the input image, feature quantities can be calculated, and the calculated feature quantities can be input to various classification networks such as fully connected layers to output the probability of existence of pancreatic cancer. In this case, the same Encoder as the Encoder of the region extraction unit 113 used in step S202 may be used. In this case, the convolution operation on the input image may be performed in this step, or the calculation result of the Encoder, which is an intermediate result of the processing in step S202, may be used as is.

[0045] Here, referring again to Figure 3, we will explain a specific example of the processing in this step using a CNN. In this embodiment, the region extraction unit 305 is a CNN, and the Encoder unit performs convolution processing on the input image 301 multiple times to obtain a feature map as a result of the convolution. In step S202, the obtained feature map was input to the Decoder unit, but in this step, unlike that, the feature map is input to the feature analysis unit 307, which is composed of fully connected layers, etc., in order to analyze the features. In this embodiment, the feature analysis unit 307 infers the probability of existence 308 of pancreatic cancer based on the feature map, but it may also simultaneously infer the probability of existence of other tumors, such as cystic tumors and intraductal papillary mucinous neoplasms, or other different classes (tumor types). In this case, the number of output classes of the feature analysis unit 307 is changed so that the classes to be inferred exist independently for each type of tumor. In Figure 3, as an example, an example in which the probability of existence of pancreatic cancer was inferred to be 0.984 is shown. Once the probability of existence is inferred, the image processing device 111 moves the flow to step S206.

[0046] (Step S206: Determination of the probability inference result) In step S206, the determination unit 116 determines whether or not pancreatic cancer is present in the image acquired by the image acquisition unit 112, based on the probability of presence inferred by the probability of presence inference unit 115. Specifically, if the inferred probability of presence exceeds a predetermined reference value (e.g., 0.5), it is determined that pancreatic cancer is present in the image. Here, the reference value used for the determination may be obtained from the storage unit 131 or received from the user via the operation unit 141. In addition to the method based on comparison with the reference value, any method that determines whether or not pancreatic cancer is present from the probability of presence inferred by the probability of presence inference unit 115 may be used. For example, a machine learning-based inference unit that takes the probability of presence as input and outputs whether or not pancreatic cancer is present may be further provided and used to make the determination. If it is determined that pancreatic cancer is present in the image, the image processing device 111 moves the flow to step S207; otherwise, it moves the flow to step S209.

[0047] (Step S207: Obtaining information supporting the inference) In step S207, the inference basis information acquisition unit 117 acquires inference basis information, which is information that influenced the inference processing of the existence probability inference unit 115, from the image acquired by the image acquisition unit 112 and the inference result output by the existence probability inference unit 115. Here, the inference basis information in this embodiment is spatial feature quantities that contributed to the output of the inference result. Specifically, examples include image features of the pancreatic cancer itself depicted in the image, and image features that represent indirect findings suggesting the presence of pancreatic cancer. Examples of indirect findings include pancreatic duct dilation, in which the flow of pancreatic juice flowing from the tail to the head of the pancreas is obstructed by cancer cells with hypovascularity, causing the pancreatic juice to remain in the pancreatic duct and the pancreatic duct to expand, and pancreatic duct obstruction, which is the location where the flow is obstructed. Many other things can also be considered.

[0048] It should be noted that individual indirect findings do not necessarily indicate the presence of pancreatic cancer. However, when multiple indirect findings are detected together (appearing simultaneously), it can be used as strong evidence to suspect the presence of pancreatic cancer. For example, even if pancreatic duct obstruction is confirmed on a CT image, if the pancreatic duct is not dilated and remains narrow, it cannot be ruled out that the obstruction may not be caused by cancer cells, but rather by a misidentification of a region of the pancreatic duct that is too narrow to be visible on the CT image. However, if findings of pancreatic duct dilation are also shown at the same time, the possibility of pancreatic duct obstruction caused by cancer cells increases. In this way, by combining image features that indicate the presence of indirect findings, it is possible to suspect the presence of pancreatic cancer, and image features representing indirect findings can be used as inferred evidence.

[0049] The inference basis information can be obtained using known feature analysis methods, such as Grad-CAM described in Non-Patent Literature 1. As an example, the method for obtaining the inference basis information will be explained using Figure 4. In this embodiment, the existence probability inference unit 115 (region extraction unit 305 in the figure) is a CNN, so it performs convolution processing multiple times on the input image 301 to obtain multiple feature maps 409 as convolution results. In the next convolution layer, this feature map 409 is reduced and further convolution processing is performed to obtain the next feature map 410. Each feature map (409, 410) obtained in this way has various values, regardless of whether they are positive or negative. In general, the feature maps (409, 410) show various features of the image, and the CNN is learned to combine multiple of these feature maps to calculate a feature that indicates the probability of existence of pancreatic cancer. Figure 4 shows, as an example, the case in which the probability of existence of pancreatic cancer is inferred to be 0.984 as a result of performing all convolutions.

[0050] Finally, the existence probability inference unit 115 obtains inference basis information 411 by integrating each feature map (409, 410). Various methods exist for the integration process; in one example, inference basis information can be obtained by acquiring the strongly outputted regions within each feature map (409, 410). When acquiring inference basis information, instead of simply selecting the strongly outputted regions, as disclosed in Non-Patent Literature 1, the existence probability value may be differentiated with respect to each feature map, and a weighting process may be performed on the resulting gradient. Also, although feature maps can take both positive and negative values, it is desirable to acquire only the feature maps with positive values.

[0051] In the case of a CNN with multiple convolutional layers, selecting the feature map 410, which is the convolution result closer to the output layer, allows for the extraction of abstract features of the image, making it possible to obtain noise-robust inference basis information 411. If the resolution of the feature map (409,410) differs from the resolution of the input image 301, it is advisable to apply a resolution transformation to the feature map (409,410) to match the resolution of the input image 301 before obtaining the inference basis information 411. In addition to selecting one of the multiple feature maps (409,410), it is also possible to integrate multiple feature maps (409,410) to obtain the inference basis information 411. For example, each feature map (409,410) in each layer can be transformed to the same resolution, then superimposed, and the maximum value of each layer can be obtained to obtain the inference basis information 411. In this case, it is possible to obtain inference basis information 411 that has more accurate positional information (coordinate information) for the input image 301. Once the features that form the basis for inference are identified, the image processing device 111 moves the flow to step S208.

[0052] (Step S208: Determine which information to display as the basis for the inference) In step S208, the display target determination unit 118 determines the inference basis information extracted by the inference basis information acquisition unit 117 as the display target. After the determination, the image processing device 111 moves the flow to step S210.

[0053] (Step S209: Determine the predetermined items to be displayed) If the flow proceeds to step S209, it means that no region has been extracted as pancreatic cancer, and no features suggesting the presence of pancreatic cancer have been obtained. Therefore, it is preferable to inform the user that the possibility of pancreatic cancer being present is low. For this reason, in step S209, the display target determination unit 118 determines a predetermined item, such as "The possibility of pancreatic cancer being present is low," as the display target to inform the user that there is no display target. Alternatively, an image with all pixels filled with 0s may be determined as the display target. After determining the display target, the image processing device 111 moves the flow to step S210. Note that in step S209, it is also possible to skip the next step S210 by choosing not to determine a display target and not to perform any display at all.

[0054] (Step S210: Output processing) In step S210, the output unit 119 outputs the display target determined by the display target determination unit 118 to the display processing unit 151. At this time, the output unit 119 can superimpose the determined display target onto, for example, the image acquired in step S201 (the input image 301 shown in Figure 3) and output it. The output result may also be stored in the storage unit 131 or the database 102. Once the display target is output, the image processing described above is completed.

[0055] Next, with reference to Figure 5, an example of an image displayed on the display unit 161 by the display processing unit 151 through the image processing described above will be explained. Figure 5 shows a schematic diagram of an image that can be displayed on the display unit 161 after superposition processing by the output unit 119, etc., according to the display target determined by the display target determination unit 118. Note that in Figures 5(a) and (b), only a single image of the pancreas and related information are shown as display examples. However, in actual displays, various information or images such as patient information and information about the modality used to acquire the image may also be included as display targets, and these may be considered as at least one of the items determined to be displayed.

[0056] Figure 5(a) shows an example in which the region extraction result 306 determined by the display target determination unit 118 in step S204 is superimposed on the input image 301 in Figure 3. Since the region extraction result 306 output by the region extraction unit 113 is located on the pancreatic cancer region 304, the basis for the region extraction result is visualized to the user. Furthermore, to emphasize the region extraction result 306, a bounding box 502 surrounding the region extraction result 306 may be displayed as an example. In addition, if the probability of pancreatic cancer presence is obtained from the region extraction result 306 in step S203, the obtained probability value may be displayed on the screen. For example, if the maximum pixel value of the region extraction result 306, which has a value range of 0 to 1, is 0.921, the obtained probability of presence can be displayed as 0.921 or 92.1% in the pancreatic cancer presence probability display frame 501. Figure 5(a) illustrates the case where the probability of existence display frame 501 is placed outside the input image 301, but it may also be displayed superimposed on the input image 301 in correspondence with the region extraction result 306 and the bounding rectangle 502.

[0057] Figure 5(b) shows an example in which the inference basis information 411, which the display target determination unit 118 decided to display in step S208, is superimposed on the input image 301. The inference basis information 411 acquired by the inference basis information acquisition unit 117 indicates the location of pancreatic cancer (pancreatic cancer region 304) or the location of indirect findings indicating the location of pancreatic cancer (for example, the location of occlusion of the pancreatic duct 303), thereby visualizing the inference basis for the probability of existence (area of ​​interest) to the user. Here, since the inference basis information 411 is information generated from the feature map 409, it can take both positive and negative values, but for example, the area with a high pixel value (luminance value) of this feature map (light-colored area in the figure) can be notified to the user that it is an area of ​​interest. In the example shown in Figure 5(b), an attention scale 503 is displayed to make it easier for the user to understand how much attention the displayed inference basis information 411 is worth. The attention scale at this time continuously changes brightness, for example, so that when the value of the inference basis information 411 is low, it becomes closer to black, and when the value is high, it becomes closer to white. Note that the manner in which the area of ​​interest is notified to the user is not limited to the examples given here, as long as the user can see the area in question. Also, if the probability of presence of pancreatic cancer is obtained in step S205, the probability of presence of pancreatic cancer may be displayed on the screen as in Figure 5(a).

[0058] The processing according to the first embodiment of the present invention is performed by the processing procedure described above. This makes it possible to provide the user with suitable inference-based information that suggests the presence of a lesion.

[0059] (Extreme variation 1-1) In the first embodiment described above, the region extraction unit 113 and the existence probability inference unit 115 used a common encoder for feature extraction as the processes in steps S202 and S205. However, embodiments of the present invention are not limited to the above example. For example, independent encoders may be used for the feature extraction units of the region extraction unit 113 and the existence probability inference unit 115. As an example, the region extraction unit 113 may use a CNN with a U-net structure, similar to the first embodiment, and the existence probability inference unit 115 may use a classification model such as a Residual Network or VGG.

[0060] Thus, by having the region extraction unit 113 and the existence probability inference unit 115 have different neural network (NN) models, it is possible to obtain different features for each task, namely the region extraction task and the existence probability inference task. This configuration is expected to increase the variety of inference basis information presented to the user. It is also expected to allow for flexible setting of learning rates, model sizes, etc., appropriate for each task.

[0061] <Second Embodiment> In the first embodiment, images were used as input information for both the region extraction process and the probability of existence inference process, and the feature map obtained by convolution on the image was described as an example of a feature with spatial information. In contrast, the image processing device according to the second embodiment visualizes the feature that serves as the basis for inference based on the probability of existence of a lesion inferred using features that do not have spatial information, and then performs a display target determination process.

[0062] In this embodiment, features without spatial information include, for example, the volume of the pancreas and representative values ​​(maximum, minimum, median, etc.) of the diameter of the pancreatic duct throughout the pancreas. Features without spatial information are not limited to these; for example, representative values ​​of the diameter of the pancreatic region throughout the pancreas and radiomic features are also included as features without spatial information. In this embodiment, an example is shown in which representative values ​​of features calculated from an image, such as radiomic features, are used as input information for existence probability inference processing. By using representative values ​​of each feature, existence probability inference processing can be performed with simpler processing compared to the first embodiment.

[0063] The configuration of the image processing apparatus and image processing system according to this embodiment, and the processes performed in the image processing apparatus, will be described below with reference to Figure 6. Figure 6 is a block diagram showing an example of the overall configuration of the image processing system 601, including the image processing apparatus 611, according to this embodiment. As shown in Figure 6, the image processing system 601 further includes an image feature acquisition unit 620 and a feature projection unit 621 in addition to the configuration of the image processing apparatus 111 described in the first embodiment.

[0064] The image feature acquisition unit 620 acquires one or more features from the image acquired by the image acquisition unit 112 for use in the existence probability inference unit 115. The features acquired in this embodiment are values ​​that do not necessarily have spatial information, such as the volume of the pancreas or representative values ​​(maximum value, minimum value, median, etc.) of the diameter of the pancreatic duct.

[0065] The feature projection unit 621 projects the image features determined by the inference basis information acquisition unit 117 as the basis for inference onto the spatial image acquired by the image acquisition unit 112. In other words, the feature projection unit 621 projects the features to be displayed for use in the output unit 119 onto the spatial information.

[0066] Next, an example of a series of processes from image processing to image display actually performed in the image processing system 601 described above will be explained in detail using the flowchart in Figure 7. Here, the processes from step S701 to step S704 are the same as the processes from step S201 to step S204 explained in the flowchart in Figure 2 illustrated in the first embodiment. Also, the processes from step S711 and step S712 are the same as the processes from step S209 and step S210. For this reason, the explanation of the processes performed in these steps will be omitted here, and the processes that differ from those performed in the first embodiment will be explained. Furthermore, in the following explanation, it will be assumed that the input image is stored in the storage unit 131 of the image processing system 601.

[0067] (Step S705: Image feature acquisition process) In step S705, the image feature acquisition unit 620 acquires features from the image acquired by the image acquisition unit 112 that the existence probability inference unit 115 uses to infer the existence probability. In this embodiment, various features can be used, such as radiomic features in the field of radiological diagnosis. Specifically, examples of these features include shape features such as the volume of the target organ (e.g., the pancreas), the major or minor axis of the target organ in a cross-sectional image, and features based on the frequency distribution (histogram) of pixel values. In addition, various other features, such as features based on the spatial distribution of pixel values ​​such as texture, can also be used. The above features may be calculated directly from the input image, or they may be calculated from an image obtained by spatially transforming the input image. Furthermore, for these features, for example, they may be calculated from an image obtained after transforming the image into an SPR (stretched curved planar reconstruction) image, which is generated by virtually stretching the pancreatic duct core line running through the pancreas into a straight line.

[0068] In this embodiment, as an example, we will describe the case in which features related to the pancreas are obtained in the SPR image space. First, pancreatic duct core line information is obtained based on user input via the operation unit 141. At this time, the pancreatic duct core line information may be stored in the storage unit 131 or the database 102, or it may be obtained from the input image by inference using machine learning such as CNN.

[0069] Here, an example of SPR image generation is shown using Figure 8. Figure 8(a) was obtained using the input image 301 exemplified in Figure 3, and the same reference numbers are used for the same parts as those described in Figure 3 in the following explanation. The dotted line in Figure 8(a) represents the pancreatic duct core line information 805. The pancreatic duct core line information 805 is obtained by image processing of the input image 301 or by user input. Based on the acquired pancreatic duct core line information 805, points are set discretely at equal intervals along the core line from the head end of the pancreas 807 to the tail end of the pancreas 808. Then, a cross-sectional image 809 is generated by cutting out the input image 301 in a cross section perpendicular to the direction of the pancreatic duct core line (pancreatic duct core line information 805) passing through each of these points. Figure 8(b) is a view of the cut-out cross-sectional image 809 from the direction of the core line. Furthermore, by stacking these generated cross-sectional images 809, an SPR image 806 like the one in Figure 8(c) is obtained. Figure 8(c) shows a cross-sectional view of the stacked SPR image, which is a section cut along the core line.

[0070] Next, the pancreatic duct diameter, which is an example of a feature quantity used in this embodiment, is obtained from the obtained SPR image 806. From one end of the SPR image 806 or the other end of the pancreatic duct core line information 805 to the other end, the pancreatic duct diameter 810 is obtained for each generated cross-sectional image as shown in Figure 8(b), and this value is used as a feature quantity. Note that the feature quantity obtained in this step is not limited to the pancreatic duct diameter as exemplified; for example, the pancreatic diameter may also be obtained, or the pixel value (luminance value) on the pancreatic duct core line may be obtained and used as a feature quantity. Furthermore, multiple feature quantities from these may be obtained. In this embodiment, the case in which the pancreatic duct diameter and pancreatic diameter are obtained will be described as an example.

[0071] Here, the range of values ​​obtained differs significantly between different features (for example, pancreatic duct diameter and pancreatic diameter). Therefore, a normalization process is applied to roughly equalize the distribution of each feature, and the values ​​of each feature are converted so that they fall within the range of 0 to 1. The normalization parameters used at this time may be obtained from the database 102 or the storage unit 131, or they may be obtained from the user via the operation unit 141. Alternatively, multiple images present in the database 102 or the storage unit 131 may be obtained and normalized using the maximum and minimum values ​​for each parameter. For each feature, the values ​​are normalized to the range of 0 to 1 by dividing by the range between the maximum and minimum values ​​that the feature can take.

[0072] Finally, a representative value is obtained for each acquired feature. In this embodiment, the maximum value of the feature obtained for each generated cross-sectional image is used. However, the representative value is not limited to the maximum value; for example, the median may also be used.

[0073] The volume of the pancreas and the diameter of the pancreatic duct may be obtained, for example, from the input image using a known image processing method, or from the user via the operation unit 141. Alternatively, information on the pancreatic duct core and pancreatic core may be obtained as auxiliary information for obtaining the volume of the pancreas and the diameter of the pancreatic duct, and the volume of the pancreas and the diameter of the pancreatic duct may be obtained based on this information. Information on the pancreatic duct core and pancreatic core may also be obtained using image processing, or from the user via the operation unit 141.

[0074] (Step S706: Execution of existence probability inference process) In step S706, the probability of presence inference unit 115 infers the probability of presence of pancreatic cancer from the representative values ​​of the features acquired by the image feature acquisition unit 620 in step S705. The inference process can use, for example, known machine learning methods for inferring classification problems. Specifically, the representative values ​​of the features can be input into a machine learning model such as a neural network to infer the probability of presence. In addition, logistic regression or SVM (support-vector machine) may be used. These machine learning models are assumed to have been trained using training data in advance so that appropriate inference can be performed. Furthermore, the probability of presence is not limited to the method of obtaining it by inference using a machine learning method, but may also be obtained by calculation using a mathematical formula with arbitrary coefficients, for example. In this embodiment, the probability of pancreatic cancer being included is obtained as a continuous value from 0 to 1 by the inference process. If the probability of presence of pancreatic cancer is high, it will take a value of 1 or close to 1, and if it is low, it will take a value of 0 or close to 0. When the probability of presence is inferred in step S706, the image processing device 611 moves the flow to step S707.

[0075] (Step S707: Determination of the probability inference result) In step S707, the determination unit 116 determines whether or not pancreatic cancer is present in the image acquired by the image acquisition unit 112, based on the probability of presence inferred by the probability of presence inference unit 115. Specifically, if the inferred probability of presence is greater than or equal to a predetermined threshold value (e.g., 0.5), it is determined that pancreatic cancer is present in the image; otherwise, it is determined that pancreatic cancer is not present in the image. If the determination determines that pancreatic cancer is present in the image, the image processing device 611 moves the flow to step S708; if it determines that pancreatic cancer is not present, it moves the flow to step S711.

[0076] (Step S708: Obtaining information supporting the inference) In step S708, the inference basis information acquisition unit 117 acquires inference basis information, which is information that influenced the inference process of the existence probability inference unit 115, from the image acquired by the image acquisition unit 112 and the inference result output by the existence probability inference unit 115. Here, the inference basis information in this embodiment is exemplified by information that, among the input information consisting of representative values ​​of multiple existing features, is retained without being excluded even after thresholding or weighting processing, and contributes to increasing the existence probability. For example, when a neural network is used as the inference process of the existence probability inference unit 115, several pieces of information such as representative values ​​of pancreatic diameter and pancreatic duct diameter are input, and product or sum operations are performed with other input information in each layer. The inference basis information is the information after weighting and bias processing has been performed on the activation function obtained from such product or sum operations with other input information, and which was able to fire the nodes of the output layer. To obtain this inference basis information, the loss value (1 - inferred existence probability) obtained by forward propagation can be backpropagated, and the obtained gradient can be used as the contribution. Then, features with a large contribution can be obtained as inference basis information. In addition, in regression analysis methods such as logistic regression, if there are features with large regression coefficients that exceed a predetermined value, the representative value of the relevant feature can be obtained as inference basis information. Once the inference basis information is obtained, the image processing device 611 moves the flow to step S709.

[0077] (Step S709: Projection of inference basis information) In step S709, the feature projection unit 621 projects the inference basis information obtained in step S708 onto the image space acquired by the image acquisition unit 112. In this embodiment, since the representative value of the feature selected as the inference basis information is a representative value of multiple feature obtained from multiple cross-sectional images, the unit searches for coordinate values ​​that indicate where the representative value is located in the image space. More specifically, it acquires the coordinate values ​​in the SPR space that have the same value as the representative value obtained in step S705 from among the multiple feature values ​​obtained from multiple cross-sectional images. At this time, if there are multiple coordinate values ​​in the SPR space that have the same value as the representative value, each of the multiple coordinate values ​​is acquired. In addition to acquiring the coordinate values ​​of the positions that have the same value as the representative value, it is also possible to acquire coordinate values ​​that have feature values ​​whose difference from the representative value is within a predetermined range. Then, the reverse procedure of the procedure (spatial transformation) used to generate the SPR image in step S705 is performed on the acquired coordinate values ​​to obtain the corresponding positions in the image space of the image acquired by the image acquisition unit 112. That is, the coordinate values ​​on the SPR image are projected onto the corresponding positions on the input image.

[0078] Next, the feature projection unit 621 generates a display image to be determined as the display target in step S712. The feature projection unit 621 generates an image with the same shape (pixel arrangement) as the image acquired by the image acquisition unit 112 and initializes the pixel values ​​with an arbitrary value (for example, 0). Then, using the coordinate values ​​in the image space obtained by the method described above, it draws a likelihood map (heatmap) of an arbitrary size on the display image. Here, the likelihood map is a multi-valued image with a Gaussian distribution-like structure where the coordinate values ​​in the image space are the peak positions, and the spread of the distribution can be freely set. In this embodiment, the information indicating a region is not limited to a likelihood map; it can take various forms, such as an arrow or rectangle indicating the coordinates. By drawing such a likelihood map, an image can be generated in which the position that serves as the basis for inference can be visually identified. When the feature projection process is completed, the image processing device 611 moves the flow to step S710.

[0079] (Step S710: Determine which information to display as the basis for the inference) In step S710, the display target determination unit 118 determines the information indicating the inference basis location projected by the feature projection unit 621 as the display target. At this time, the image on which the inference basis information is projected becomes the display image determined for step S712 described above. After determination, the image processing device 611 moves the flow to step S712.

[0080] The processing according to the second embodiment of the present invention is performed by the processing procedure described above. According to this, even if the feature quantities used for inferring the probability of the existence of a lesion are feature quantities that do not contain spatial information, suitable inference basis information accompanied by spatial information that suggests the existence of a lesion can be provided to the user.

[0081] (Variation 2-1) In the second embodiment described above, as the process in step S705, the image feature acquisition unit 620 acquires representative values ​​of features such as the diameter of the pancreatic duct from the SPR image and performs existence probability inference processing. However, embodiments of the present invention are not limited to the above example. For example, without acquiring representative values ​​for each feature, some or all of the features acquired at each crosscut on the SPR image can be input to the existence probability inference unit 115 to infer the probability of pancreatic cancer. In this case, by associating the position of each crosscut in the SPR image with the acquired features, the features can be made to have spatial information. As a result, the feature projection processing in step S709 can be performed on the position of the crosscut corresponding to the feature selected as the inference basis information, making it possible to execute the process in step S709 with simpler processing and improving processing speed.

[0082] <Third Embodiment> In the first embodiment, a method was described in which the user can visually determine which of the above the displayed image is by displaying the results of the region extraction process obtained in step S202 and the results of the inference basis obtained in step S207 in different ways. However, the embodiments of the present invention are not limited to this. In the image processing apparatus according to the third embodiment, an example of a method is shown in which the user can visually determine which of the above the displayed image is by performing report generation according to the display target.

[0083] The configuration of the image processing apparatus and image processing system according to this embodiment, and the processes performed in the image processing apparatus, will be described below with reference to Figure 9. Figure 9 is a block diagram showing an example of the overall configuration of the image processing system 901, including the image processing apparatus 911, according to this embodiment. As shown in Figure 9, the image processing apparatus 911 further includes a report generation unit 920 in addition to the configuration of the image processing apparatus 111 described in the first embodiment.

[0084] The report generation unit 920 acquires information about the display targets determined by the display target determination unit 118 and generates a report based on this information. The generated report may be included in the display targets determined later.

[0085] Next, an example of the series of processes from image processing to image display actually performed in the image processing system 901 described above will be explained in detail using the flowchart in Figure 11. Here, the processes from step S1101 to step S1108 are the same as the processes from step S201 to step S208 described in the flowchart of the first embodiment. Similarly, the processes from step S1110 and step S1111 are the same as the processes of step S209 and S210. For this reason, the explanation of the processes performed in these steps will be omitted here, and the processes that differ from those performed in the first embodiment will be explained. Also, in the following explanation, it will be assumed that the input image is stored in the storage unit 131 of the image processing system 901.

[0086] (Step S1109: Report generation) After the image processing device 911 has executed the processes in steps S1101 to S1108, in step S1109, the report generation unit 920 generates a report according to the display target determined by the display target determination unit 118. If the display target determination unit 118 determines that the display target is the region output by the region extraction unit 113, the report generation unit 920 generates a report that includes a statement indicating that a region suspected of being pancreatic cancer has been found. As an example, as shown in Figure 10(a), a report 1001 such as "A region suspected of being pancreatic cancer has been found" is generated. In addition, Figure 10(a) also displays the bounding rectangle surrounding the region extraction result generated in step S209 of the first embodiment. In such cases, it is also possible to add a statement to report 1001 indicating a highlighted region, such as "The area within the frame has a high probability of being pancreatic cancer." Note that the items stated in report 1001 are examples, and the displayed items are not limited to these, as similar items can be presented to the user.

[0087] Furthermore, if the display target determination unit 118 determines that the inference basis information acquired by the inference basis information acquisition unit 117 is the display target, the report generation unit 920 generates a report that includes a statement indicating that the presence of pancreatic cancer is suspected. For example, as shown in Figure 10(b), it generates a report 1002 such as "The presence of pancreatic cancer is suspected." Also, in Figure 10(b), as described in the first embodiment, the inference basis information is shown as a continuous value. In such cases, the values ​​of the inference basis information may be used as indicators of attention, and items indicating suspicious areas, such as "Please visually check the areas of high attention," may be added to the report 1002. In any case, the report is generated in a way that differs from when the display target determination unit 118 determines that the display target is an area output by the area extraction unit 113. Note that the items described in the report 1002 are examples, and the displayed items are not limited to these if similar items can be presented to the user. By displaying such a report in addition to images, users who see the report text will be able to determine which information the display target determination unit 118 has decided to display. Once the report to be displayed is generated, the image processing device 911 moves the flow to step S1111.

[0088] (Step S1111: Output) In step S1111, if there is a display target determined by the display target determination unit 118, the output unit 119 outputs the display target to the display processing unit 151. At that time, the output unit 119 can superimpose the determined display target onto, for example, the image acquired in step S1101 (input image 301 shown in Figure 3), and further output the report generated in step S1109. The output result may also be stored in the storage unit 131 or the database 102. It should be noted that there may be cases where, after steps S1103 and S1106, no region is extracted as pancreatic cancer, and no features suggesting the presence of pancreatic cancer are obtained. In such cases, there is no need to generate a report, so predetermined items are output in step S1111 via step S1110.

[0089] The processing according to the third embodiment of the present invention is performed by the processing procedure described above. This allows for the appropriate provision of region extraction results or inference basis information suggesting the presence of a lesion to the user in the form of images and text information (report text).

[0090] As described above, an image processing apparatus according to one aspect of the present invention comprises a medical image acquisition unit, an inference unit, an inference basis information acquisition unit, and a display target determination unit. The image processing apparatus may also include an output unit 119 that outputs the display target determined by the display target determination unit to a display device exemplified by the display unit 116. The medical image acquisition unit, exemplified as the image acquisition unit 112 in the first embodiment, can acquire medical images from a storage unit 131, a database 102, or a medical image acquisition device exemplified by an X-ray CT apparatus (not shown). The inference unit may also include a first inference unit and a second inference unit. In this case, the first inference unit may include the existence probability inference unit 115 and the determination unit 116 exemplified in the first embodiment, and can infer (acquire) the existence probability of a lesion exemplified by pancreatic cancer as a first inference result as a first task (steps S205, S206). The second inference unit may include a region extraction unit 113 and a region extraction result determination unit 114, and as a second task different from the first task, it can extract (acquire) the region where a lesion exemplified by pancreatic cancer exists as a second inference result (steps S202, S203). The inference basis information acquisition unit 117 can acquire inference basis information, which is information that influenced the first inference unit when it acquired the first inference result (step S207). The display target determination unit 118 determines, based on at least one of the first inference result and the second inference result (steps S203, S206), at least one of the inference basis information (step S208) and the region based on the second inference result (step S204) as the display target. At that time, if the display target determination unit 118 includes information indicating the presence of a lesion in the second inference result, it determines the information based on the second inference result as the display target (step S204). Furthermore, if the second inference result does not include information indicating the presence of a lesion, and the first inference result does include information indicating the presence of a lesion, the inference basis information is determined to be displayed (step S208). In the first embodiment, the cases in which the image exemplified in Figure 5(a) is displayed and the cases in which the image exemplified in Figure 5(b) is displayed are illustrated as the cases after going through step S204 and step S208, respectively.However, these images may not only be displayed separately, but both images may also be displayed simultaneously or switched between. For example, in the example of Figure 5(b), if there are multiple pieces of inference-based information, or if it is difficult to distinguish which image is superimposed by multiple determined display targets, the image in Figure 5(a) can be easily viewed, which is expected to improve user convenience. The display of both images in this manner can be determined, for example, in step S208 of the flowchart in Figure 2, under specific conditions such as when there are multiple pieces of inference-based information. Alternatively, after display, a display mode for instructing the user to switch between images may be added to the display target output in step S210.

[0091] The image processing apparatus described above may further include an image feature acquisition unit. The image feature acquisition unit acquires multiple image features related to the organ (pancreas in this embodiment) or lesion (pancreatic cancer in this embodiment) to be processed from a medical image. In the first embodiment, the image feature acquisition unit is exemplified by an inference basis information acquisition unit 117, and in the second embodiment, it is exemplified as a configuration that further includes an image feature acquisition unit 620. Such an inference basis information acquisition unit can acquire inference basis information based on at least one image feature that contributed to the first inference result among the image features. In the first embodiment, the inference basis information acquisition unit can acquire spatial features in the medical image that contributed to the inference by the first inference unit as inference basis information.

[0092] Furthermore, the inference basis information described above can be obtained as spatial features consisting of features whose contribution to the inference by the first inference unit, calculated from the first inference result, is greater than a predetermined value (step S207). In addition, the contribution in this case can be based on the gradient obtained based on the inference processing by the first task, and the spatial features in this case can be obtained based on the gradient.

[0093] Furthermore, the inference unit described above can perform the first task on the medical image and obtain the first inference result if the second inference result does not contain information indicating the presence of a lesion (no step S203). The inference unit described above may also include a first inference unit that obtains the first inference result and a second inference unit that obtains the second inference result (modification 1-1). These first and second inference units may be composed of the same CNN as illustrated in the first embodiment, but they may also be composed of different CNNs, as described in modification 1-1.

[0094] As described in the second embodiment, the image processing apparatus according to the present invention may further include a feature projection unit 621 that projects image features onto the image space of a medical image. In such a configuration, it is preferable that the inference basis information acquisition unit 117 converts non-spatial features into spatial features and acquires them for the image features projected onto the image space.

[0095] The image processing apparatus according to the present invention may further include a report generation unit 920, as described in the third embodiment. The report generation unit 920 can generate text suggesting the presence of a lesion when the display target determination unit 118 determines an area based on the second inference result as a display target (Figure 10(a)). It can also generate text indicating the existence of a region of interest suggesting a lesion when the display target determination unit 118 determines the inference basis information of the first inference result as a display target (Figure 10(b)).

[0096] In the embodiments described above, X-ray CT (Computed Tomography) images obtained by an X-ray CT (Computed Tomography) device are exemplified as suitable medical images when applying the present invention. However, the medical images to which the present invention can be applied are not limited to these, and may include medical images obtained by various modalities or conditions. Furthermore, while images of the abdominal region including the pancreas are exemplified as suitable images when applying the present invention, the target region is not limited to the pancreas or the abdomen. Any part of the human body that requires medical observation can be treated as a target region in the present invention.

[0097] Furthermore, the image processing device described above may further include a display unit 161, which is separate from the image processing device 111 in the image processing system 101. In this case, the display unit can display the display target determined by the display target determination unit 118. Alternatively, the image processing device may further include a display processing unit 151, which is separate from the image processing device 111 in the image processing system 101. In this case, when the display target determination unit 118 determines a region based on the second inference result as a display target, the display processing unit 151 can highlight the region where the lesion exists and display the display target on the display unit 161 (Figure 5(a)). Also, when the display target determination unit 118 determines the inference basis information of the first inference result as a display target, the display processing unit 151 can display the inference basis information as the display target on the display unit 161 in a display format different from the region where the lesion exists (Figure 5(b)). Furthermore, the display target determination unit 118 or the output unit 119 may differentiate the display format depending on whether the region based on the second inference result is determined to be the display target or whether the inference basis information for the first inference result is determined to be the display target. For example, by changing the display format such as the display color (brightness of a black and white display, etc.) or the transparency of the image, the user can easily distinguish whether the first inference result or the second inference result is being displayed. In addition, such display commands may be issued by the display processing unit 151 to the display unit 161 according to the inference result determined to be the display target.

[0098] Furthermore, the image processing devices described above can be configured as an image processing system 101,601,901, which includes a display unit 161 that displays the display target determined by the display target determination unit 118, separately from the image processing devices 111,611,911. In this case, for example, individual PCs including the display unit 161 can be connected to each image processing device 111,611,911, creating a system that allows for the individual confirmation of processed images in different examination rooms. Such an image processing system 101,601,901 may also include a display processing unit 151 that causes the display target determined by the display target determination unit 118 to be displayed on the display unit 161 in a display format appropriate to the display target. The image processing system 101,601,901 may also further include an operation unit 141 that inputs at least one of a threshold for evaluating a first inference result and a threshold for evaluating a second inference result to the image processing device when the display target determination unit 118 determines the display target.

[0099] <Other Embodiments> Furthermore, the disclosed technology can take the form of, for example, a system, apparatus, method, program, or recording medium (storage medium). Specifically, it may be applied to a system consisting of multiple devices (e.g., a host computer, interface devices, imaging devices, web applications, etc.) or to an apparatus consisting of a single device.

[0100] Furthermore, it goes without saying that the object of the present invention can be achieved as follows: a program that implements one or more of the functions of the above embodiments can be supplied to a system or device via a network or storage medium, and one or more processors in the computer of that system or device can read and execute the program. It can also be achieved by a circuit (e.g., ASIC) that implements one or more functions. Various recording media can be used as recording media, such as flexible disks, optical disks (e.g., CD-ROM, DVD-ROM), magneto-optical disks, magnetic tapes, non-volatile memory (e.g., USB memory), ROM, etc. Alternatively, a program that implements the above functions can be downloaded via a network and executed by a computer.

[0101] Furthermore, the functionality of the above embodiment is not limited to being realized solely by the execution of program code read by a computer. It also includes cases where the OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program code, and the functionality of the above embodiment is realized through that processing.

[0102] Furthermore, program code read from a recording medium may be written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer. This also includes cases where the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the above-mentioned functions are realized through that processing.

[0103] The present invention described above includes the following configuration, method, and program. (Composition 1) A medical image acquisition unit that acquires medical images, An inference unit obtains a first inference result by performing a first task of inferring the probability of the presence of a lesion on the medical image, and obtains a second inference result by performing a second task, which is different from the first task, of extracting the region where the lesion is located on the medical image. An inference basis information acquisition unit acquires inference basis information, which is information that influenced the inference unit when it obtained the first inference result. The system includes a display target determination unit that determines, based on at least one of the first inference result and the second inference result, to be a display target, and the inference basis information and at least one of the region based on the second inference result. The display target determination unit determines the information based on the second inference result as the display target if the second inference result includes information indicating the presence of a lesion, and determines the inference basis information as the display target if the second inference result does not include information indicating the presence of a lesion, and the first inference result includes information indicating the presence of a lesion. (Configuration 2) The system further comprises an image feature acquisition unit that acquires multiple image features related to the organ or lesion to be processed from the aforementioned medical image, The image processing apparatus according to configuration 1, wherein the inference basis information acquisition unit acquires the inference basis information based on at least one image feature from the image features that contributed to the first inference result. (Composition 3) The image processing apparatus according to configuration 1 or 2, wherein the inference basis information acquisition unit acquires spatial feature quantities in the medical image that contributed to the first inference result as the inference basis information. (Composition 4) The image processing apparatus according to configuration 3, wherein the spatial feature quantity is a feature quantity whose contribution to the inference by the inference unit calculated from the first inference result is greater than a predetermined value. (Composition 5) The image processing apparatus according to configuration 4, wherein the contribution is based on a gradient obtained based on the inference processing by the first task. (Composition 6) The spatial feature quantities are based on the gradient, as described in the image processing apparatus of configuration 5. (Composition 7) The image processing apparatus according to any one of configurations 1 to 6, wherein the inference unit performs the first task on the medical image to obtain the first inference result if the second inference result does not include information indicating the presence of a lesion. (Composition 8) The image processing apparatus according to any one of configurations 1 to 7, wherein the inference unit includes a first inference unit for acquiring the first inference result and a second inference unit for acquiring the second inference result. (Composition 9) The system further comprises a feature projection unit that projects the aforementioned image features onto the image space of the medical image, The image processing apparatus according to any one of configurations 2 to 5, wherein the inference basis information acquisition unit acquires image features projected onto the image space as spatial features. (Composition 10) It also includes a report generation unit, The image processing apparatus according to any one of configurations 1 to 9, wherein the report generation unit generates a sentence suggesting the presence of the lesion when the display target determination unit determines the region based on the second inference result as the display target, and generates a sentence indicating that there is a region of interest suggesting the lesion when the display target determination unit determines the inference basis information of the first inference result as the display target. (Composition 11) The image processing apparatus according to any one of configurations 1 to 10, wherein the medical image is an X-ray CT (Computed Tomography) image obtained by an X-ray CT (Computed Tomography) device. (Composition 12) The image processing apparatus according to any one of configurations 1 to 11, wherein the medical image is an image taken of the abdominal region. (Composition 13) It also includes a display unit, The image processing apparatus according to any one of configurations 1 to 12, wherein the display target determined by the display target determination unit is superimposed on the acquired medical image and displayed on the display unit. (Composition 14) Further equipped with a display processing unit, The image processing apparatus according to any one of configurations 1 to 12, wherein the display processing unit, when the display target determination unit determines a region based on the second inference result as a display target, displays the display target on the display unit with emphasis on the region where the lesion exists, and when the display target determination unit determines the inference basis information of the first inference result as a display target, displays the display target on the display unit in a display format different from the region where the lesion exists. (Composition 15) An image processing apparatus as described in any of configurations 1 to 12, An image processing system comprising: a display unit that displays the display target determined by the display target determination unit; (Composition 16) The image processing system according to configuration 15, further comprising a display processing unit, which, when the display target determination unit determines a region based on the second inference result as a display target, emphasizes the region where the lesion exists and displays the display target on the display unit, and when the display target determination unit determines the inference basis information of the first inference result as a display target, displays the display target on the display unit in a display format different from the region where the lesion exists, the inference basis information. (Composition 17) The image processing system according to configuration 15 or 16, further comprising an operation unit that inputs at least one of a threshold for evaluating the first inference result and a threshold for evaluating the second inference result to the image processing device when the display target determination unit determines the display target. (Method 1) Acquiring medical images and, A second inference result is obtained by performing a second task on the aforementioned medical image to extract the area where the lesion is located, If the second inference result includes information indicating the presence of a lesion, then at least one piece of information based on the second inference result shall be determined to be the item to be displayed. If the second inference result does not include information indicating the presence of a lesion, a first inference result is obtained by performing a first task on the medical image that indicates the probability of the presence of a lesion, unlike the second task, and inference basis information, which is the information that influenced the acquisition of the first inference result, is obtained. If the first inference result includes information indicating the presence of a lesion, then the inference basis information is determined to be at least one of the items to be displayed. Image processing methods, including those mentioned above. (Program 1) A program that, when executed by a computer, causes the computer to perform each step of the image processing method described in Method 1.

[0104] Although embodiments of the present invention have been described above, the present invention is not limited to these, and can be modified or altered within the scope described in the claims. Furthermore, in addition to the above-described objectives of the present invention, it is also possible to describe the effects and advantages derived from each configuration shown in the above-described embodiments for carrying out the invention, which cannot be obtained by the prior art, and which can be described as another objective of the disclosure of this specification. [Explanation of Symbols]

[0105] 101 Image Processing System 102 Databases 111 Image Processing Device 112 Image acquisition unit 113 Region extraction part 114 Area extraction result judgment unit 115 Probability Inference Unit 116 Judgment section 117 Inference Basis Information Acquisition Unit 118 Display Target Determination Unit 119 Output section 121 Communications Department 131 Storage section 141 Operation section 151 Display Processing Unit 161 Display section

Claims

1. A medical image acquisition unit that acquires medical images, An inference unit obtains a first inference result by performing a first task of inferring the probability of the presence of a lesion on the medical image, and obtains a second inference result by performing a second task, which is different from the first task, of extracting the region where the lesion is located on the medical image. An inference basis information acquisition unit acquires inference basis information, which is information that influenced the inference unit when it obtained the first inference result. The system includes a display target determination unit that determines, based on at least one of the first inference result and the second inference result, to be a display target, and the inference basis information and at least one of the region based on the second inference result. The display target determination unit determines the information based on the second inference result as the display target if the second inference result includes information indicating the presence of a lesion, and determines the inference basis information as the display target if the second inference result does not include information indicating the presence of a lesion, and the first inference result includes information indicating the presence of a lesion.

2. The system further comprises an image feature acquisition unit that acquires multiple image features related to the organ or lesion to be processed from the aforementioned medical image, The image processing apparatus according to claim 1, wherein the inference basis information acquisition unit acquires the inference basis information based on at least one image feature from the image features that contributed to the first inference result.

3. The image processing apparatus according to claim 1, wherein the inference basis information acquisition unit acquires spatial feature quantities in the medical image that contributed to the first inference result as the inference basis information.

4. The image processing apparatus according to claim 3, wherein the spatial feature quantity is a feature quantity whose contribution to the inference by the inference unit calculated from the first inference result is greater than a predetermined value.

5. The image processing apparatus according to claim 4, wherein the contribution is based on a gradient obtained based on the inference processing by the first task.

6. The spatial feature quantity is based on the gradient, as described in claim 5.

7. The image processing apparatus according to claim 1, wherein the inference unit performs the first task on the medical image to obtain the first inference result if the second inference result does not include information indicating the presence of a lesion.

8. The image processing apparatus according to claim 1, wherein the inference unit includes a first inference unit for acquiring the first inference result and a second inference unit for acquiring the second inference result.

9. The system further comprises a feature projection unit that projects the aforementioned image features onto the image space of the medical image, The image processing apparatus according to claim 2, wherein the inference basis information acquisition unit acquires image features projected onto the image space as spatial features.

10. It also includes a report generation unit, The image processing apparatus according to claim 1, wherein the report generation unit generates a sentence suggesting the presence of the lesion when the display target determination unit determines the region based on the second inference result as a display target, and generates a sentence indicating that there is a region of interest suggesting the lesion when the display target determination unit determines the inference basis information of the first inference result as a display target.

11. The image processing apparatus according to claim 1, wherein the medical image is an X-ray CT (Computed Tomography) image obtained by an X-ray CT (Computed Tomography) apparatus.

12. The image processing apparatus according to claim 1, wherein the medical image is an image taken of the abdominal region.

13. It also includes a display unit, The image processing apparatus according to any one of claims 1 to 12, wherein the display target determined by the display target determination unit is superimposed on the acquired medical image and displayed on the display unit.

14. Further equipped with a display processing unit, The image processing apparatus according to any one of claims 1 to 12, wherein the display processing unit, when the display target determination unit determines a region based on the second inference result as a display target, causes the display target to be displayed on the display unit with emphasis on the region where the lesion exists, and when the display target determination unit determines the inference basis information of the first inference result as a display target, causes the display target to be displayed on the display unit in a display form different from the region where the lesion exists.

15. An image processing apparatus according to any one of claims 1 to 12, An image processing system comprising: a display unit that displays the display target determined by the display target determination unit;

16. The image processing system according to claim 15, further comprising a display processing unit, which, when the display target determination unit determines a region based on the second inference result as a display target, emphasizes the region where the lesion exists and displays the display target on the display unit, and when the display target determination unit determines the inference basis information of the first inference result as a display target, displays the display target on the display unit in a display format different from the region where the lesion exists, the inference basis information.

17. The image processing system according to claim 15, further comprising an operation unit that inputs at least one of a threshold for evaluating the first inference result and a threshold for evaluating the second inference result to the image processing device when the display target determination unit determines the display target.

18. Acquiring medical images and, A second inference result is obtained by performing a second task on the aforementioned medical image to extract the area where the lesion is located, If the second inference result includes information indicating the presence of a lesion, then at least one piece of information based on the second inference result shall be determined to be the item to be displayed. If the second inference result does not include information indicating the presence of a lesion, a first inference result is obtained by performing a first task on the medical image that indicates the probability of the presence of a lesion, unlike the second task, and inference basis information, which is the information that influenced the acquisition of the first inference result, is obtained. If the first inference result includes information indicating the presence of a lesion, then the inference basis information is determined to be at least one of the items to be displayed. Image processing methods, including those mentioned above.

19. A program that, when executed by a computer, causes the computer to perform each step of the image processing method described in claim 18.

Images