Image segmentation system and image segmentation method

Abstract

An image segmentation system has: an attention heat map generation section that generates a first attention heat map and a second attention heat map from a first image of a scan image and a second image of a scan image, respectively, of different modalities of a target lesion of a patient, each pixel of the first attention heat map indicating a degree of attention given to a pixel at the same position in the first image when performing image segmentation, each pixel of the second attention heat map indicating a degree of attention given to a pixel at the same position in the second image when performing image segmentation; a difference value calculation section that calculates an attention difference image indicating a difference between the first attention heat map and the second attention heat map; and a first neural network that performs image segmentation on the second image based on at least the first image and the attention difference image generated by the difference value calculation section, and obtains a segmentation result of the second image.

Description

TECHNICAL FIELD

[0001] The present application relates to an image segmentation system and an image segmentation method. BACKGROUND

[0002] In order to grasp the condition of a cancer patient, it is necessary to evaluate the treatment effect of the tumor of the patient. As an evaluation standard of the treatment effect of the tumor, RECIST (Response Evaluation Criteria in Solid Tumors) can be cited. RECIST classifies the target lesion of the patient into one of Complete Response, Partial Response, Stable Disease, and Progressive Disease. In RECIST, it is necessary to determine whether a new tumor has appeared in the target lesion of the patient compared with the previous stage, and at present, this detection process is mainly performed by a doctor by manually comparing the scan images of the current stage and the previous stage.

[0003] In the conventional technology, there is a technology of automatically performing tumor detection based on the scan image of the target lesion of the patient using a deep learning technique. However, compared with the organ detection technique based on deep learning, the accuracy of the tumor detection technique based on deep learning is generally low, and in particular, for small tumors, there are often cases where the tumor region cannot be correctly divided, and therefore the conventional tumor detection technique using a deep learning technique cannot be directly applied to the evaluation of the treatment effect of the tumor such as RECIST.

[0004] In order to solve this problem, a segmentation network using a cascade neural network is proposed, which first inputs the scan image to a low-resolution first segmentation network to obtain an image segmentation result, and then inputs the segmentation result together with the scan image to a high-resolution second segmentation network to obtain an image segmentation result with higher accuracy. However, the accuracy improvement obtained by this method is still limited and cannot meet the needs of the evaluation of the treatment effect of the tumor.

[0005] Patent Document 1 discloses a breast cancer MRI segmentation method based on a hierarchical convolutional neural network, which performs image segmentation on the scan image based on pre-enhanced images and contrast-enhanced images to determine whether a tumor exists. However, in this document, it is not possible to automatically and accurately detect whether a new tumor has appeared in the target lesion of the patient compared with the previous stage.

[0006] Prior Art Documents

[0007] Patent Documents

[0008] Patent Document 1: CN110796672 SUMMARY

[0009] PROBLEMS TO BE SOLVED BY THE INVENTION

[0010] The present application is to provide an image segmentation system and an image segmentation method capable of automatically and accurately detecting whether a new tumor has appeared in a target lesion of a patient by comparing different scan images of the target lesion.

[0011] MEANS FOR SOLVING THE PROBLEMS

[0012] The image segmentation system of the embodiment has: an attention heat map generation section that generates a first attention heat map and a second attention heat map from a first image and a second image that are different modalities of scan images of a target lesion of a patient, the first attention heat map being an image of the same size as the first image, each pixel of the first attention heat map indicating a degree of attention to be given to a pixel at the same position in the first image when image segmentation is performed, the second attention heat map being an image of the same size as the second image, each pixel of the second attention heat map indicating a degree of attention to be given to a pixel at the same position in the second image when image segmentation is performed; a difference value calculation section that calculates an attention difference image indicating a difference between the first attention heat map and the second attention heat map; and a first neural network that performs image segmentation on the second image based on at least the first image and the attention difference image generated by the difference value calculation section, to obtain a segmentation result of the second image.

[0013] According to the present application, an image segmentation system and an image segmentation method capable of automatically and accurately detecting whether a new tumor has appeared in a target lesion of a patient by comparing different scan images of the target lesion can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a diagram showing an example of the structure of the image segmentation system of the embodiment.

[0015] Figure 2 is a flowchart showing the image segmentation method of the embodiment.

[0016] Figure 3 is a diagram showing an example of the first image and the second image of the embodiment.

[0017] Figure 4 is a diagram showing an example of the first attention heat map and the second attention heat map of the embodiment.

[0018] Figure 5 is a diagram showing an example of the attention difference image of the embodiment.

[0019] Figure 6 is a diagram indicating an example of a final result of image segmentation of the second image of the present embodiment.

[0020] Figure 7 is a diagram comparing a pixel value difference image and an attention difference image between the first image and the second image of the present embodiment. DETAILED DESCRIPTION

[0021] Hereinafter, an image segmentation system and an image segmentation method of the present embodiment will be described with reference to the drawings.

[0022] (First Embodiment)

[0023] Figure 1 is a diagram indicating an example of a structure of the image segmentation system of the present embodiment.

[0024] The image segmentation system 1 has a communication section 11, an input section 12, a display section 13, a storage section 14, and an image segmentation section 15. The communication section 11, the input section 12, the display section 13, the storage section 14, and the image segmentation section 15 are communicably connected to each other, for example, by a bus not shown.

[0025] The communication section 11 includes a communication interface such as a NIC, for example. The communication section 11 is capable of communicating with an external device through a network, transmitting and receiving various information. The communication section 11 can output the received information to the image segmentation section 15. The communication section 11 can transmit the information of the image segmentation section 15 to an external device connected through a network.

[0026] The input section 12 receives an input operation of a user such as a doctor and an expert, for example, and outputs a signal based on the received input operation to the image segmentation section 15. The input section 12 is realized by a mouse and a keyboard, a trackball, a switch, a button, a lever, a touch panel, and the like, for example. The input section 12 can be realized by a user interface that receives a sound input through a microphone and the like, for example. In the case where the input section 12 is a touch panel, the display section 13 described later can be integrated with the input section 12.

[0027] The display section 13 displays various information. For example, the display section 13 displays an image generated by the image segmentation section 15, and displays a GUI (Graphical User Interface) and the like for receiving an input operation from an operator. The display section 13 is an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display, and the like, for example.

[0028] The storage section 14 is realized by, for example, a storage device such as a ROM, a flash memory, a RAM (Random Access Memory), a HDD (Hard Disc Drive), a SSD (Solid State Drive), a register, or the like. The flash memory, the HDD, the SSD, and the like are nonvolatile storage media. These nonvolatile storage media can be realized by other storage devices connected via a network, such as a NAS (Network Attached Storage) and an external storage server device. The data required for constructing a neural network is stored in the storage section 14, and the details thereof will be described later.

[0029] The image segmentation section 15 performs image segmentation on the scan image of the current stage of the target lesion of the patient, on the basis of the scan image of the previous stage and the scan image of the current stage of the target lesion of the patient. In the present application, image segmentation refers to classification of each pixel in a scan image to generate an image segmentation result containing classification information of each pixel, in which the classes of the classification include at least three classes of "tumor" and "organ" and "background". Hereinafter, the scan image of the previous stage of the target lesion of the patient will be referred to as a first image, and the scan image of the current stage of the target lesion of the patient will be referred to as a second image. The image segmentation section 15 includes an attention heat map generation section 151, a difference value calculation section 152, a first deep learning section 153, and a post-processing section 154.

[0030] Each of the constituent elements of the image segmentation section 15 can be realized by executing a program (software) stored in the storage section 14 by a hardware processor such as a CPU and a GPU. Some or all of these constituent elements can also be realized by hardware such as an LSI, an ASIC, and an FPGA, and can also be realized by a cooperative operation of software and hardware. The above-mentioned program can be stored in advance in the storage section 14, and can also be stored in a removable storage medium such as a DVD and a CD-ROM, and mounted to the storage section 14 from the storage medium by mounting the storage medium to a drive device of the image segmentation system 1.

[0031] The attention heat map generation section 151 acquires the first image and the second image from an external device through the communication section 11 or from the storage section 14, generates an image segmentation result and an attention heat map for the first image on the basis of the first image, and generates an image segmentation result and an attention heat map for the second image on the basis of the second image.

[0032] The attention heat map generation section 151 includes a second deep learning section 1511 and a mapping section 1512. The second deep learning section 1511 constructs a second neural network by reading the structure parameters of the second neural network that has been trained and stored in the storage section 14, and performs image segmentation on an input scan image based on the second neural network. The mapping section 1512 calculates, from the scan image input to the second deep learning section 1511 and the image segmentation result thereof, the attention information of each pixel of the scan image with respect to a certain specific classification by the second neural network, and generates an attention heat map. The attention information of each pixel with respect to a certain specific classification indicates the degree of attention of the neural network to each pixel when determining which pixels should be classified into the specific classification in the above-described image segmentation. The higher the value of the attention information of a pixel, the more the neural network pays attention to the pixel when performing the specific classification, and also means that the pixel has a higher possibility of being classified into the specific classification.

[0033] The difference value calculation section 152 calculates the difference value between the attention heat map of the first image and the attention heat map of the second image generated by the attention heat map generation section 151, generates an attention difference image, and outputs the attention difference image to the first deep learning section 153.

[0034] The first deep learning section 153 constructs a first neural network by reading the structure parameters of the first neural network that has been trained and stored in the storage section 14, and performs image segmentation on an input scan image and the attention difference image calculated by the difference value calculation section 152 by the first neural network, and outputs the generated image segmentation result to the post-processing section 154.

[0035] The post-processing section 154 performs post-processing on the image segmentation result output from the first deep learning section 153, and determines whether there is a new tumor based on the post-processed image segmentation result.

[0036] Figure 2 is a flowchart of an image segmentation method according to the embodiment. Hereinafter, the image segmentation method according to the embodiment will be described with reference to Figure 2 The image segmentation method according to the present application will be described. In the present embodiment, a CT scan image of a previous stage of a lesion region of a liver of a patient is used as a first image, and a CT scan image of a current stage of the lesion region of the liver of the patient is used as a second image, but the present application is not limited thereto. For example, the lesion region can be an organ such as a lung, a stomach, or a brain, and the scan image can be obtained by MRI scanning, ultrasonic scanning, or the like. In the present embodiment, both the first image and the second image are two-dimensional planar gray scale images, and indicate information of one cross section of the lesion region of the patient, which is stored in a matrix of width W x height H, for example.

[0037] Further, in the present embodiment, for the convenience of explanation, a case where only one set of first and second images is used for image segmentation is explained, but a plurality of sets of first and second images can be used simultaneously, and image segmentation processing can be performed in parallel.

[0038] If an instruction to start image segmentation processing by a user is received by the input unit 12, the following processing of steps S100 to S106 is performed.

[0039] In step S100, the display unit 13 prompts the user for information that first and second images are needed to be input, and based on the input operation of the user, the first and second images are acquired and input to the attention heat map generation unit 151. Here, for example, a dialog box including a text of "Please input first and second images" is displayed on the display unit 13, and then based on the operation of the user, the first and second images are read from an external device or the storage unit 14 and input to the attention heat map generation unit 151.

[0040] Figure 3 is a drawing showing an example of first and second images of the embodiment. In Figure 3 an example of the first image is shown in (a) of Figure 3 an example of the second image is shown in (b) of In the present embodiment, the first and second images are images that have been preprocessed. The above-mentioned preprocessing includes registration of the first and second images, image pixel value normalization, image resampling, image denoising, and the like.

[0041] By observing Figure 3 it can be seen that in the previous stage shown in (a) of Figure 3 there was already a tumor (shaded part) in the region A of the liver, and in the current stage shown in (b) of Figure 3 a new tumor (shaded part) appeared in the region B of the liver. In Figure 3 (a) and (b) of it can be considered that the large gray region represents the liver parenchyma part, and the large black part in the upper part represents the background part.

[0042] In step S101, the attention heat map generation section 151 performs image segmentation on the first image and the second image input in step S100, and classifies each pixel in the first image and the second image into one of "tumor", "liver parenchyma", and "background", respectively. First, the attention heat map generation section 151 causes the second deep learning section 1511 to read the parameters of the second neural network that has been trained and stored in the storage section 14, and constructs the second neural network. The second neural network here is a deep convolutional neural network that performs image segmentation, such as U-net or attention U-net. The second deep learning section 1511 then inputs the first image and the second image to the second neural network, respectively, and calculates the image segmentation results of the first image and the second image by forward propagation, while recording the intermediate data generated in each layer of the neural network in order to calculate the image segmentation results. Note that, although the first image and the second image are subjected to image segmentation by the second neural network in step S101, this image segmentation is performed in order to generate the attention heat map of the first image and the second image, and the image segmentation result of the second image obtained here is not the final image segmentation result of the second image.

[0043] The second neural network of the present embodiment and the training process thereof will be described below. The second neural network of the present embodiment is a multi-layered deep convolutional neural network that outputs an image segmentation result corresponding to a scan image as an input through calculation in each layer. In image segmentation by a deep convolutional neural network, each pixel is classified based on all the image information in a certain range centered on the pixel. In order to train the second neural network, a large number of scan images and their correct answers for image segmentation, which are data containing correct classification information for each pixel in the image, are required. Then, in each training, a plurality of sets of scan images are input to the second neural network, and the distance between the output image segmentation result and the correct answer is calculated, and the parameters of the neural network are corrected in such a manner that the distance is shortened by gradient descent, stochastic gradient descent, or the like. This training is repeated until the distance between the image segmentation result output by the second neural network and the correct answer satisfies a prescribed accuracy. Here, the distance described above is, for example, Manhattan distance or Euclidean distance, or the like.

[0044] In step S102, the mapping unit 1512 of the attention heatmap generation unit 151 uses, for example, attention heatmap generation methods such as CAM (class activation mapping) or grad-CAM (grad-class activation mapping) to perform backpropagation through the second neural network using the image segmentation results and intermediate data of the first and second images calculated in step S101. This allows the unit to calculate the attention information of each pixel in the first and second images for the category "liver parenchyma" in image segmentation, and generate a first attention heatmap as an attention heatmap of the first image and a second attention heatmap as an attention heatmap of the second image.

[0045] Figure 4 This diagram illustrates an example of a first attention heatmap and a second attention heatmap of an embodiment. The attention heatmap is an image of the same size as the scanned image, and each pixel in the attention heatmap represents the attention information of a pixel at the same position in the scanned image corresponding to that attention heatmap. Figure 4 (a) represents the first attention heatmap. Figure 4 (b) represents the second attention heatmap. Figure 4 In this model, the higher the brightness of a pixel and the closer it is to white, the higher its attention information value. This means the neural network pays more attention to that pixel when classifying it as "liver parenchyma," assigns it higher weight when deciding whether it and its surrounding pixels belong to "liver parenchyma," and thus has a higher probability of being classified as such. Conversely, the lower the brightness of a pixel and the closer it is to black, the lower its attention information value. This means the neural network pays less attention to that pixel when classifying it as "liver parenchyma," assigns it lower weight when deciding whether it and its surrounding pixels belong to "liver parenchyma," and thus has a lower probability of being such such a pixel. Through observation... Figure 4 It can be seen that, in Figure 4 In (b), the neural network pays less attention to region B where new tumors appear, which suggests that the neural network considers this part to be less likely to be "liver parenchyma" than in the previous stage.

[0046] In step S103, the difference value calculation section 152 generates an attention difference image based on the first attention heat map and the second attention heat map generated by the attention heat map generation section 151 in step S102. Specifically, the difference value calculation section 152 first performs normalization processing on the first attention heat map and the second attention heat map generated in step S102, and then extracts a region of interest of the liver based on the normalized first attention heat map and the normalized second attention heat map, respectively, performs weighting processing on the first attention heat map and the second attention heat map based on the regions of interest, and then calculates a distance between corresponding pixels of the weighted first attention heat map and the weighted second attention heat map to generate the attention difference image. Here, the distance is, for example, a Manhattan distance or a Euclidean distance, or the like.

[0047] Figure 5 is a diagram showing an example of the attention difference image of the present embodiment. In Figure 5 As can be seen in FIG. 10B, the first attention heat map and the second attention heat map have a clear difference in the region B in which a new tumor has appeared.

[0048] In step S104, the first deep learning section 153 performs image segmentation on the second image based on the attention difference image and the second image. First, the first deep learning section 153 reads the parameters of the first neural network that has been trained, which are stored in the storage section 14, and constructs the first neural network. The first neural network here is a deep convolutional neural network that performs image segmentation, such as a U-net or an attention U-net. Then, the first deep learning section 153 inputs the second image and the attention difference image, which represents the difference between the first attention heat map and the second attention heat map, to the first neural network, and performs image segmentation on the second image based on the information of the second image and the attention difference image.

[0049] The first neural network of the present embodiment and the training process thereof will be described below. The first neural network of the present embodiment is a multi-layered deep convolutional neural network that takes a scan image and an attention difference image as inputs, and outputs an image segmentation result corresponding to the scan image through the calculation of each layer. Since the attention difference image that represents the difference between the attention heat map for the scan image of the current stage and the attention heat map for the scan image of the previous stage is used, the first neural network is able to grasp that there is a significant change in those regions in the current stage compared to the previous stage. For example, in this case, the neural network grasps the information that there is a significant change in region B. In order to train the first neural network, a large number of scan images, attention difference images corresponding to the scan images, and correct answers of image segmentation of the scan images, which are data containing correct classification information of each pixel in the image, are required. Then, in each training, a plurality of sets of scan images and attention difference images corresponding thereto are input to the first neural network, and the distance between the output image segmentation result and the correct answer is calculated, and the parameters of the neural network are corrected in such a manner that the distance is shortened by gradient descent, stochastic gradient descent, or the like. This training is repeated until the distance between the image segmentation result output by the first neural network and the correct answer satisfies a prescribed accuracy. Here, the distance is, for example, Manhattan distance or Euclidean distance, or the like.

[0050] Figure 6 is an example of a final result of image segmentation of the second image of the present embodiment. In Figure 6 , the newly appearing tumor in region B is accurately image segmented, and the outline of the tumor is accurately segmented.

[0051] Figure 7 is a graph comparing the pixel value difference image and the attention difference image of the first image and the second image of the present embodiment. Figure 7 (a) of is a pixel value difference image that represents the difference between the pixel values of each pixel of the first image and the second image of the present embodiment, in which the higher the brightness and the closer to white, the smaller the difference between the pixel values of the first image and the second image at the pixel, and the lower the brightness and the closer to black, the greater the difference between the pixel values of the first image and the second image at the pixel, Figure 7 (b) of is the same image as the attention difference image of Figure 5 By comparing (a) and (b) of Figure 7 , it can be seen that the pixel value difference image of (a) of Figure 7As understood from (b) of FIG. 9, the pixel value difference image shows a large difference not only in the region of the newly generated tumor but also in the region irrelevant to the newly generated tumor due to the influence of noise, whereas the attention difference image shows a large difference only in the region of the newly generated tumor. As understood from this, the attention difference image contains important information for detecting the generation of a new tumor but hardly contains noise information. Whereas the pixel value difference image also contains information for detecting the generation of a new tumor, it contains a large amount of noise information. If the pixel value difference image is used instead of the attention difference image for image segmentation, it is possible to misclassify a region not belonging to a tumor as a tumor region due to the influence of noise.

[0052] In step S105, the post-processing section 154 performs post-processing including smoothing, correction of erroneously segmented regions, and the like on the image segmentation result of the second image obtained in step S104, and outputs the post-processed image segmentation result as the final result of the image segmentation of the second image.

[0053] In step S106, the post-processing section 154 determines the tumor region already present in the first image based on the correct answer of the image segmentation of the first image, judges whether a newly appearing tumor region is present based on the final result of the image segmentation of the second image, and performs processing such as tumor texture analysis and tumor size evaluation in the case where a newly appearing tumor region is present. Here, for example, the image segmentation result of the first image calculated in step S101 can be used as the correct answer of the image segmentation of the first image, or an image segmentation result of the first image obtained by the present image segmentation method or another image segmentation method can be used as the correct answer of the image segmentation of the first image.

[0054] After the image segmentation processing is completed, the image segmentation section 15 can display the final result of the image segmentation of the second image on the display section 13, store it in the storage section 14, or transmit it to an external device via the communication section 11.

[0055] According to the present embodiment, when performing image segmentation on a scan image of a current stage of a target lesion of a patient, image segmentation is performed not only based on the scan image of the current stage but also based on an attention difference image that is a difference between an attention heat map of the scan image of the current stage and an attention heat map of a scan image of a previous stage. Therefore, image segmentation can be accurately performed. Furthermore, the image segmentation method of the present application can accurately obtain a stage change of a lesion such as a tumor, and thus can be more suitably applied to evaluation of a tumor treatment effect such as RECIST.

[0056] (Second Embodiment)

[0057] Next, the second embodiment will be described. In the second embodiment, the first image and the second image are different modalities of scan images of the same stage of the target lesion, which is different from the first embodiment described above. Hereinafter, the description will be focused on the difference from the first embodiment, and the commonalities with the first embodiment will be omitted. In the description of the second embodiment, the same reference numerals will be used to describe the same parts as those of the first embodiment.

[0058] In the present embodiment, the first image and the second image are different modalities of scan images, i.e., the generation means of the first image and the second image are different. For example, the first image is a CT image of the current stage of the target lesion of the patient, and the second image is an MRI image of the current stage of the target lesion of the patient. Alternatively, for example, both the first image and the second image are CT images of the current stage of the target lesion of the patient, but the contrast parameters or imaging parameters used to generate the first image and the second image are different.

[0059] In the case where the first image and the second image are images generated by different scanning methods, the second neural network can be a multi-layered deep convolutional neural network having an input layer corresponding to the first image and an input layer corresponding to the second image, and the scan image is segmented by inputting the data of the scan image into the corresponding input layer. Alternatively, two different second neural networks can be provided to segment the first image and the second image, respectively.

[0060] According to the present embodiment, when the scan image of the target lesion of the patient is segmented, the scan image is segmented by the difference in the attention heat map of different modalities of scan images, i.e., the attention difference image, and thus the difference in the different modalities of scan images can be compared, and the segmentation of the image can be accurately performed.

[0061] (Third Embodiment)

[0062] Next, the third embodiment will be described. In the third embodiment, the first image and the second image are different modalities of scan images of the previous stage and the current stage of the target lesion, which is different from the first embodiment described above. Hereinafter, the description will be focused on the difference from the first embodiment, and the commonalities with the first embodiment will be omitted. In the description of the third embodiment, the same reference numerals will be used to describe the same parts as those of the first embodiment.

[0063] In the present embodiment, the first image and the second image are scanning images of different modalities, i.e., the first image and the second image are generated by different means. For example, the first image is a CT image of a previous stage of a target lesion of a patient, and the second image is an MRI image of a current stage of the target lesion of the patient. Or, for example, the first image is a CT image of a previous stage of a target lesion of a patient, and the second image is a CT image of a current stage of the target lesion of the patient, but the contrast parameters or imaging parameters used to generate the first image and the second image are different.

[0064] In the case where the first image and the second image are images generated by different scanning methods, the second neural network can be a multi-layer deep convolutional neural network having an input layer corresponding to the first image and an input layer corresponding to the second image, and the scanning image is segmented by inputting the data of the scanning image into the corresponding input layer. In addition, two different second neural networks can also be provided to segment the first image and the second image, respectively.

[0065] According to the present embodiment, when performing image segmentation on a scanning image of a target lesion of a patient, the image is segmented by the difference between the attention heat maps of the scanning images of different stages of different modalities, i.e., the attention difference image, so that the differences between the scanning images of different modalities can be compared, and the image segmentation can be performed accurately. In addition, the image segmentation method of the present application can accurately obtain the stage changes of lesions such as tumors, and thus can be better applied to the evaluation of tumor treatment effects such as RECIST.

[0066] (Variation)

[0067] In addition, in the above embodiments, the case where the first image and the second image have been pre-processed is described, but the first image and the second image can also be images before pre-processing, and the first image and the second image are pre-processed by the image segmentation system of the present application after inputting the first image and the second image before pre-processing.

[0068] In addition, in the above embodiments, the case where the first neural network takes the scanning image and the attention difference image as input is described, but the first neural network can also take the scanning image of the previous stage, the image segmentation result of the scanning image of the previous stage output by the second neural network, the image segmentation result of the scanning image of the current stage output by the second neural network, etc. as input on this basis.

[0069] In addition, in the above embodiments, the case where the scanning image is a two-dimensional planar image is described, but the scanning image can also be a three-dimensional stereoscopic image containing information of multiple sections of the lesion area of the patient. In addition, the scanning image can also be an RGB image.

Claims

1. An image segmentation system for performing image segmentation on a scan image, comprising: an attention heat map generation section that generates a first attention heat map and a second attention heat map from a first image and a second image of different modalities of a scan image of a target lesion of a patient, respectively, the first attention heat map being an image of the same size as the first image, each pixel of the first attention heat map indicating a degree of attention to be given to a pixel at the same position in the first image when performing image segmentation, the second attention heat map being an image of the same size as the second image, each pixel of the second attention heat map indicating a degree of attention to be given to a pixel at the same position in the second image when performing image segmentation; a difference value calculation section that calculates an attention difference image indicating a difference between the first attention heat map and the second attention heat map; and a first neural network that performs image segmentation on the second image based on at least the first image and the attention difference image generated by the difference value calculation section, to obtain a segmentation result of the second image, the attention heat map generation section including a second neural network that performs image segmentation on the first image and the second image to obtain a segmentation result of the first image and a segmentation result of the second image, and a mapping section that generates the first attention heat map using the segmentation result of the first image obtained by the second neural network via the second neural network, generates the second attention heat map using the segmentation result of the second image obtained by the second neural network via the second neural network, normalizes the first attention heat map and the second attention heat map, respectively, extracts a region of interest, performs weighting processing on the first attention heat map and the second attention heat map based on the region of interest, and calculates a distance between each corresponding pixel of the weighted first attention heat map and the second attention heat map to generate the attention difference image.

2. The image segmentation system of claim 1, wherein, The second neural network performs image segmentation on the first image and the second image to obtain a segmentation result of the first image and first intermediate data and a segmentation result of the second image and second intermediate data, the first intermediate data and the second intermediate data being intermediate data generated in each layer of a neural network when calculating the segmentation result of the first image and the segmentation result of the second image, respectively, the mapping section generates the first attention heat map based on the second neural network by class activation mapping or gradient class activation mapping using the segmentation result of the first image and the first intermediate data obtained by the second neural network, and generates the second attention heat map based on the second neural network by class activation mapping or gradient class activation mapping according to the segmentation result of the second image and the second intermediate data obtained by the second neural network.

3. The image segmentation system of claim 1 or 2, wherein, The first image is a scan image of a previous stage of the target lesion of the patient, and the second image is a scan image of a current stage of the target lesion of the patient.

4. The image segmentation system of claim 1 or 2, wherein, The image segmentation system further has a post-processing unit that performs post-processing including smoothing, correction of erroneous segmentation regions, and the like, on the segmentation result of the second image.

5. The image segmentation system of claim 2, wherein, The first neural network further performs image segmentation on the second image based on a segmentation result of image segmentation of the second image obtained by the second neural network.

6. The image segmentation system of claim 2, wherein, The first neural network further performs image segmentation on the second image based on the first image.

7. An image segmentation method for segmenting a tumor in a scan image, having: an attention heat map generation step of generating a first attention heat map and a second attention heat map from a first image and a second image of different modalities of scan images of a target lesion of a patient, respectively, the first attention heat map being an image of the same size as the first image, each pixel of the first attention heat map representing a degree of attention to be given to a pixel at the same position in the first image when performing image segmentation, the second attention heat map being an image of the same size as the second image, each pixel of the second attention heat map representing a degree of attention to be given to a pixel at the same position in the second image when performing image segmentation. a difference value calculation step that calculates an attention difference image that represents a difference between the first attention heat map and the second attention heat map; and a segmentation step that performs image segmentation on the second image based on at least the first image and the attention difference image generated by the difference value calculation step, and obtains an output segmentation result of the second image, the attention heat map generation step including a second neural network that performs image segmentation on the first image and the second image, and obtains a segmentation result of the first image and a segmentation result of the second image, and a mapping step in which the first attention heat map is generated from the segmentation result of the first image obtained by the second neural network via the second neural network, and the second attention heat map is generated from the segmentation result of the second image obtained by the second neural network via the second neural network, the difference value calculation step in which the first attention heat map and the second attention heat map are subjected to normalization processing and an area of interest is extracted, respectively, the first attention heat map and the second attention heat map are subjected to weighting processing based on the area of interest, and a distance between respective corresponding pixels of the weighted first attention heat map and the second attention heat map is calculated, thereby generating an attention difference image.

Images