Image processing device, image processing method, and storage medium

Abstract

An image processing device 1X includes: an image-of-interest extraction means; 31X and a lesion evaluation means 32X. The image-of-interest extraction means 31X extracts, from a second medical image group, an image-of-interest that is the subject of analysis related to a lesion, the extraction performed on the basis of a comparison between: a first medical image group which is related to a patient and to which note information pertaining to a first lesion is attached; and the second medical image group which is related to the patient and which was generated after the first medical image group. On the basis of the image-of-interest, the lesion evaluation means 32X performs evaluation related to a first lesion in the second medical image group.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical field of an image processing device, an image processing method, and a storage medium that perform processing related to evaluation of a lesion using medical images.

[0002] Image processing systems that evaluate lesions based on images of the inside of a patient have been known for some time. For example, Patent Literature 1 discloses a diagnostic aid that calculates feature values ​​from past and present image data obtained by CT and detects suspected lung cancer based on the feature values.

[0003] Japanese Patent Application Laid-Open No. 2001-137230

[0004] When evaluating lesions using a group of images obtained in a previous examination and a group of images obtained in a target examination, it is necessary to perform processing to detect lesions detected in the previous examination in the target examination as well. In this case, if a lesion detection model is applied to all images in the group of images obtained in the target examination, the possibility of false positive detection of lesions increases, and there is a possibility that lesions detected in the previous examination cannot be properly detected from the group of images obtained in the target examination.

[0005] In view of the above-mentioned problems, one object of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that are capable of accurately evaluating lesions.

[0006] One aspect of the image processing device is an image processing device having: an image-of-interest extraction means for extracting an image of interest from a second group of medical images of a patient, the second group of medical images having annotation information about a first lesion added thereto, based on a comparison between the first group of medical images of the patient and the second group of medical images of the patient that were generated after the first group of medical images; and a lesion evaluation means for evaluating the first lesion in the second group of medical images based on the image of interest.

[0007] One aspect of the image processing method is an image processing method in which a computer compares a first group of medical images of a patient, which have annotation information regarding a first lesion, with a second group of medical images of the patient which were generated after the first group of medical images, and extracts an image of interest from the second group of medical images to be analyzed for the lesion; and evaluates the first lesion in the second group of medical images based on the image of interest.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute a process of: extracting an image of interest from a second group of medical images of a patient that has annotation information about a first lesion added thereto, based on a comparison between the first group of medical images of the patient and a second group of medical images of the patient that was generated after the first group of medical images, to perform an analysis of the lesion; and evaluating the first lesion in the second group of medical images based on the image of interest.

[0009] As an example of an effect of the present disclosure, it becomes possible to accurately evaluate lesions.

[0010] 1 shows a schematic configuration of a lesion evaluation system; FIG. 2 is an example of a functional block of a processor of an image processing device; FIG. 3 is a diagram in which a group of pre-examination images and a group of target examination images of a certain patient are arranged in correspondence with the patient's imaged body parts; FIG. 4 is a diagram showing an overview of processing performed by a first lesion evaluation unit when a pair of an image of interest and a corresponding image to be annotated is given; FIG. 5 shows an example of display of information based on evaluations of a first lesion and a second lesion; FIG. 6 is an example of a flowchart showing an overview of processing performed by an image processing device; FIG. 7 shows a schematic configuration of a lesion evaluation system; FIG. 8 is a block diagram of an image processing device; FIG. 9 is an example of a flowchart showing the procedure of processing performed by an image processing device.

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] <First embodiment> (1) System configuration Fig. 1 shows a schematic configuration of a lesion evaluation system 100. The lesion evaluation system 100 shown in Fig. 1 is a system that evaluates the lesion (condition) of a patient undergoing treatment for a disease such as cancer by comparing it with past conditions, and presents the evaluation results to a medical professional such as a doctor as information indicating the effectiveness of the treatment, etc. The lesion evaluation system 100 mainly includes an image processing device 1, a display device 3, and an input device 4.

[0013] The image processing device 1 evaluates the condition of a patient based on information about the patient's most recent examination, which is the subject of lesion evaluation, and information about the patient's examination conducted prior to the examination (also referred to as a "preliminary examination"). Specifically, the image processing device 1 evaluates a lesion detected in the preliminary examination (also referred to as a "first lesion") and a lesion not detected in a previous examination (i.e., a lesion other than the first lesion, also referred to as a "second lesion"). Examples of evaluations of the first lesion include RECIST (Response Evaluation Criteria in Solid Tumors) evaluation, drug efficacy evaluation, and therapeutic effect assessment for patients currently undergoing treatment. The image processing device 1 also controls display of the evaluation results of the first lesion and the evaluation results of the second lesion on the display device 3, and performs various processes based on user input signals received from the input device 4.

[0014] The display device 3 performs a predetermined display based on a display signal supplied from the image processing device 1. Examples of the display device 3 include displays such as a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), as well as a projector.

[0015] The input device 4 generates a user input signal based on an operation by a user, such as a doctor, of the image processing device 1. Examples of the input device 4 include a button, a keyboard, a pointing device such as a mouse, a touch panel, a remote controller, a voice input device, and any other user interface.

[0016] 1 also shows an example of the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, and an interface 13. These elements are connected via a data bus 19.

[0017] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0018] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for the processing of the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a storage medium such as a removable flash memory. The memory 12 stores programs and other information necessary for the image processing device 1 to execute each process in this embodiment.

[0019] The memory 12 stores first lesion detection model information D1, second lesion detection model information D2, and examination information D3. The first lesion detection model information D1 is model information including parameters necessary to configure a first lesion detection model, which is a model used to detect a first lesion. The second lesion detection model information D2 is model information including various parameters necessary to configure a second lesion detection model, which is a model used to detect a second lesion. The first lesion detection model and the second lesion detection model are, for example, machine learning models (including statistical models; the same applies hereinafter), and parameters necessary for these models are stored in the first lesion detection model information D1 and the second lesion detection model information D2, respectively. When the first lesion detection model is configured using a neural network, the first lesion detection model information D1 includes parameters such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weight of each element of each filter. Similarly, when the second lesion detection model is configured by a neural network, the second lesion detection model information D2 includes parameters such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weight of each element of each filter. Details of the first lesion detection model information D1 and the second lesion detection model information D2 will be described later.

[0020] The examination information D3 is examination information obtained by examining a patient, and includes pre-examination information D31 and target examination information D32. The examination information D3 is associated with, for example, a patient ID or the like for identifying the patient.

[0021] The pre-examination information D31 is information related to a pre-examination of a patient. In this embodiment, the pre-examination information D31 includes a group of images of the patient generated by an image generating device during the pre-examination (also referred to as a "pre-examination image group") and metadata associated with an image in the pre-examination image group in which a lesion (i.e., a first lesion) is detected. The image generating device described above is a device that generates images (slices) of the inside of a living body while changing its position, and examples of such devices include CT and MRI. The pre-examination image group is an example of a "first medical image group."

[0022] The metadata is data regarding the diagnosis result of the first lesion that is attached to an image showing the first lesion, and is generated, for example, by annotation by the doctor in charge of the pre-examination. The metadata includes at least area information indicating the area (range) of the first lesion within the image. Note that the metadata may also include the name of the disease corresponding to the lesion, the condition, and any other diagnostic results. Hereinafter, an image in the pre-examination image group to which metadata is attached is also referred to as an "annotated image."

[0023] The annotation is, for example, a process in which a doctor in charge of a pre-examination refers to a group of pre-examination images displayed on a display or the like, identifies an image in which a lesion appears, and then inputs, on a computer, a designation of the lesion area in the image and other inputs related to the diagnosis results. The metadata may be data based on a diagnosis result obtained by applying CAD (Computer Aided Diagnosis). The metadata is an example of "annotation information related to the first lesion."

[0024] The target examination information D32 is data obtained from an examination to be evaluated (e.g., the most recent examination) conducted after the preliminary examination, and includes at least a group of patient images generated by an image generating device. Hereinafter, the examination to be evaluated conducted after the preliminary examination will also be referred to as the "target examination," and the group of patient images included in the target examination information D32 will also be referred to as the "target examination image group." The target examination image group is an example of a "second medical image group."

[0025] The interface 13 performs an interface operation between the image processing device 1 and an external device. For example, the interface 13 is electrically connected to the display device 3 and the input device 4. The interface 13 may be a communication interface such as a network adapter for performing wired or wireless communication with the external device, or may be a hardware interface conforming to USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like. The interface 13 may also perform an interface operation with external devices such as the display device 3 and the input device 4 via a communication network such as the Internet.

[0026] The configuration of the lesion evaluation system 100 shown in FIG. 1 is an example, and various modifications may be made.

[0027] For example, the image processing device 1 may be integrated with at least one of the display device 3 and the input device 4. In another example, the image processing device 1 may include an audio output device that outputs information by voice. In yet another example, the image processing device 1 may be composed of multiple devices. In another example, instead of storing the target examination information D32 in the memory 12 in advance, the image processing device 1 may receive the target examination images from the above-mentioned image generation device that generates the patient images.

[0028] Next, the first and second lesion detection models will be described in detail.

[0029] The first lesion detection model is a machine-learned model that, when an image generated by an image generation device is input, outputs an inference result regarding a lesion area in the input image. In other words, the first lesion detection model is a machine-learned model that determines the relationship between the image input to the first lesion detection model and the lesion area in the input image. The lesion detection model may be a model (including a statistical model, the same applies hereinafter) that includes an architecture employed in any machine learning method, such as a neural network or a support vector machine. Representative models of such neural networks include, for example, Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab. The first lesion detection model is trained in advance based on a pair of an input image conforming to the input format of the lesion detection model and correct answer data (in the above example, a correct answer confidence map or bounding box) indicating the correct inference result that the lesion detection model should output when the input image is input.

[0030] Here, a first example of an inference result output by the first lesion detection model is a reliability map (including a mask image that represents the lesion area using binary values) that indicates the reliability of each unit area of ​​the input image as being a lesion area. The unit area may be an area of ​​one pixel, an area of ​​multiple pixels, or an area smaller than one pixel (a sub-pixel area). In this case, for example, the image processing device 1 determines that a connected area of ​​unit areas (which may be limited to those of a predetermined size or larger) whose reliability is equal to or greater than a predetermined threshold is a detected lesion area. The above-mentioned predetermined threshold is a threshold for detecting a lesion area (also called a "detection threshold") and corresponds to a hyperparameter set by the user.

[0031] A second example of an inference result output by the first lesion detection model is a set of a bounding box indicating the extent of a lesion area in an input image and a degree of confidence (certainty) that the area identified by the bounding box is a lesion area. In this case, for example, the image processing device 1 determines that a bounding box with a confidence level equal to or greater than a predetermined threshold is a detected lesion area. The predetermined threshold is a detection threshold for detecting a lesion area and corresponds to a hyperparameter set by the user.

[0032] The first lesion detection model may be a plurality of models each generated to be specialized for a specific body part (e.g., organ) of the imaged patient. In this case, model information for the plurality of first lesion detection models specialized for each body part is stored in first lesion detection model information D1.

[0033] When an image generated by the image generation device is input, the second lesion detection model outputs an inference result regarding the lesion area in the input image. In other words, the second lesion detection model is a model that has learned the relationship between the image input to the second lesion detection model and the lesion area in the input image. The lesion detection model may be a model including an architecture adopted in any machine learning method, such as a neural network or a support vector machine. Note that the first lesion detection model and the second lesion detection model may be the same model. In this case, lesion detection model information regarding a single model that functions as both the first lesion detection model information D1 and the second lesion detection model information D2 is pre-stored in memory 12.

[0034] The first and second lesion detection models may include a feature extraction model that extracts features from an image, or may be separate models from the feature extraction model. In the latter case, the first and second lesion detection models are machine-learned models that output the above-described inference results when a feature (a tensor with a predetermined number of dimensions) output by the feature extraction model to which an image is input is input.

[0035] (2) Functional Blocks Figure 2 shows an example of functional blocks of the processor 11 of the image processing device 1. Functionally, the processor 11 of the image processing device 1 has an acquisition unit 30, an image-of-interest extraction unit 31, a first lesion evaluation unit 32, a second lesion evaluation unit 33, and a display control unit 34. Note that in Figure 2, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to this. The same applies to other functional block diagrams described below.

[0036] The acquiring unit 30 acquires the pre-examination information D31 and the target examination information D32 included in the patient's examination information D3 via the interface 13. Then, the acquiring unit 30 supplies the acquired pre-examination information D31 and target examination information D32 to the image-of-interest extraction unit 31 and the second lesion evaluation unit 33, respectively. For example, the acquiring unit 30 identifies a patient to be evaluated in accordance with a user input signal received from the input device 4, and acquires the pre-examination information D31 and the target examination information D32 corresponding to the identified patient.

[0037] The image-of-interest extraction unit 31 extracts an image (also referred to as an "image of interest") to be analyzed for the first lesion from the group of target examination images included in the target examination information D32, and supplies the image of interest and the annotated image and metadata in the group of pre-examination images corresponding to the image of interest to the first lesion evaluation unit 32. Details of the processing performed by the image-of-interest extraction unit 31 will be described later. In other words, the image of interest is an image that should be noted in the evaluation of the first lesion.

[0038] The first lesion evaluation unit 32 evaluates the first lesion in the target examination image group based on the image of interest, the annotated image corresponding to the image of interest, and the metadata. In this case, the first lesion evaluation unit 32 first identifies a lesion area in the image of interest using a first lesion detection model constructed with reference to the first lesion detection model information D1. The first lesion evaluation unit 32 then identifies the presence or absence of a lesion area in the image of interest that is the same as the first lesion indicated by the metadata of the annotated image, and the size of the lesion area if present, and evaluates changes in the first lesion (e.g., disappearance, shrinkage, stability, progression). In this way, the first lesion evaluation unit 32 evaluates changes in the first lesion that occurred between the preliminary examination and the target examination. The first lesion evaluation unit 32 supplies the evaluation result for the first lesion to the display control unit 34. Details of the process of generating the evaluation result for the first lesion by the first lesion evaluation unit 32 will be described later.

[0039] The second lesion evaluation unit 33 evaluates a second lesion not indicated in the metadata based on the pre-examination information D31 and the target examination information D32 supplied from the acquisition unit 30. In this case, the second lesion evaluation unit 33 identifies a lesion area in each image of the target examination image group using a second lesion detection model constructed with reference to the second lesion detection model information D2, and identifies a lesion area among the identified lesion areas that corresponds to a second lesion not indicated in the metadata of the pre-examination information D31. At this time, the second lesion evaluation unit 33 determines whether the detected lesion area is a second lesion using the image-by-image matching results between the pre-examination image group and the target examination image group, as described below. The second lesion detected by the second lesion evaluation unit 33 is either a lesion that was already present at the time of the pre-examination but was overlooked without being detected and therefore not recorded in the metadata (also referred to as an "overlooked lesion"), or a lesion that was not present at the time of the pre-examination but has newly developed by the time of the target examination (also referred to as a "new lesion"). Examples of new lesions include lesions that have metastasized from a first lesion. The second lesion evaluation unit 33 supplies the evaluation result regarding the second lesion to the display control unit 34 .

[0040] The display control unit 34 generates display information based on the evaluation results for the first lesion supplied from the first lesion evaluation unit 32, the evaluation results for the second lesion supplied from the second lesion evaluation unit 33, and the examination information D3. Then, the display control unit 34 supplies the generated display information to the display device 3, thereby causing the display device 3 to display information based on the evaluation results for the first lesion and / or the second lesion.

[0041] Here, we will provide additional information about the effects of the block configuration shown in FIG. 2 . When measuring the effectiveness of treatment, such as in a RECIST evaluation, based on a physician's diagnosis, the burden on the physician increases. Specifically, the physician must find the same lesion in images captured at different times and must perform a full-body examination to determine whether new lesions have occurred. Furthermore, in new drug clinical trials, the discovery of new lesions is particularly important because the discovery of a new lesion is deemed to indicate progression and the trial is terminated. Taking the above into consideration, the image processing device 1 in this embodiment performs evaluation of a first lesion detected in a preliminary examination and evaluation of a second lesion, such as a new lesion, without requiring a physician to perform image diagnosis. This allows the image processing device 1 to optimally support diagnosis and clinical trials.

[0042] The components of the acquisition unit 30, the image-of-interest extraction unit 31, the first lesion evaluation unit 32, the second lesion evaluation unit 33, and the display control unit 34 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not necessarily be realized by software programs, but may also be realized by any combination of hardware, firmware, and software. At least some of these components may also be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0043] (3) Extraction of an Image of Interest The extraction of an image of interest will now be described in detail. The image of interest extraction unit 31 performs matching (i.e., correspondence) on an image-by-image basis between the pre-inspection image group and the target inspection image group, and extracts at least an image from the target inspection image group that is associated with the annotated image as an image of interest.

[0044] 3 is a diagram illustrating a group of pre-examination images and a group of target examination images of a certain patient, each arranged in association with a region to be imaged of the patient. In the example of FIG. 3, the group of pre-examination images includes at least one annotated image Ip1.

[0045] In this case, the image-of-interest extraction unit 31 determines the similarity of each image between the pre-examination image group and the target examination image group, and generates a matching result by matching the most similar images. In the example of FIG. 3 , the annotated image Ip1 is most similar (matches) with image It2 in the target examination image group, and the image-of-interest extraction unit 31 extracts image It1 and image It3, which represent the patient's body parts before and after image It2, as images of interest. For example, the image-of-interest extraction unit 31 calculates a similarity index for each image in the pre-examination image group that indicates the degree of similarity between the image and each image in the target examination image group, and identifies matching images based on the similarity index. When calculating a similarity index representing the degree of similarity between images in the above matching, the image-of-interest extraction unit 31 may calculate the similarity index by directly comparing the images, or may extract image features and calculate the similarity index based on the extracted features. In this case, the image-of-interest extraction unit 31 may acquire feature amounts output from the feature extraction model (or the feature extraction layer of the first lesion detection model or the second lesion detection model) used for lesion detection when each image is input to the feature extraction model (or the feature extraction layer of the first lesion detection model or the second lesion detection model). Note that the image-of-interest extraction unit 31 may perform matching on an image-by-image basis between the pre-examination image group and the target examination image group using any time-series data matching method not limited to the above-mentioned examples.

[0046] In addition, the image of interest extraction unit 31 may determine only the image It2 that matches the annotated image Ip1 as the image of interest, or may determine two or more images before and after the image It2 that matches the annotated image Ip1 as the image of interest.

[0047] In this way, the image-of-interest extraction unit 31 determines the image of interest based on matching between the preliminary examination image group and the target examination image group. As a result, the image processing device 1 performs the detection process for the first lesion only on the image of interest, which reduces erroneous detection of the first lesion and improves detection accuracy compared to when the detection process for the first lesion is performed on all images in the target examination image group.

[0048] (3) Evaluation of the First Lesion Next, a method for evaluating the first lesion will be described. The first lesion evaluation unit 32 evaluates the first lesion based on the lesion area indicated by the metadata of the annotated image and the lesion area detected using the first lesion detection model from the image of interest that matches the annotated image. In the evaluation of the first lesion, the first lesion evaluation unit 32 determines the presence or absence of a lesion area in the image of interest that represents the first lesion indicated by the metadata, and the size of the lesion area if present. Note that in the RECIST evaluation performed when the disease to be evaluated is cancer, it is necessary to identify the lesion detected in the preliminary examination (i.e., the first lesion) in the image generated in the target examination and measure the size of the identified lesion.

[0049] Here we will provide additional information about RECIST evaluation. RECIST evaluation is an evaluation standard used to determine whether treatment for solid cancers is effective. Before treatment begins, the size of the tumor (target lesion) is measured using diagnostic imaging such as CT, and the condition during treatment is classified into the following four states: Complete Response (CR) Partial Response (PR) Stable Disease (SD) Progressive Disease (PD)

[0050] Here, a complete response is defined as the disappearance of all target lesions during the course of treatment, or in the case of lymph nodes, a reduction in their short axis to less than 10 mm. A partial response is defined as a reduction in the size of target lesions by 30% or more compared to before the start of treatment. Stable disease is defined as a state between partial response and progression. Progression is defined as an increase in the size of target lesions by 20% or more compared to their smallest size during the course of treatment, or an increase in the diameter of target lesions by 5 mm or more. Thus, RECIST evaluation requires the identification of target lesions in images taken before and after treatment, and then the measurement and comparison of their size.

[0051] In clinical trials, RECIST evaluation is used to define the ratio of complete response and partial response as the response rate, and to evaluate the period from complete response, partial response, or stable state to progression as the progression-free survival period.

[0052] 4 is a diagram showing an outline of the processing executed by the first lesion evaluation unit 32 when a pair of an image of interest and a corresponding annotated image is given. Here, it is assumed that the annotated image has metadata attached thereto indicating that the area within a bounding box 80 is a lesion area.

[0053] First, the first lesion evaluation unit 32 aligns the patient region 78 displayed in the annotated image with the patient region 79 displayed in the image of interest (i.e., matches the positions of the patient regions). In this case, the first lesion evaluation unit 32 translates at least one of the patient region 78 and the patient region 79 within the image so that the positions of the patient region 78 and the patient region 79 match within the image. In this case, for example, the first lesion evaluation unit 32 moves at least one of the patient region 78 and the patient region 79 within the image so that the degree of overlap between the regions is maximized. In this case, an index indicating the degree of overlap between any regions, such as IOU (Intersection over Union), may be used. In the example of FIG. 4 , the first lesion evaluation unit 32 moves the patient region 79 within the image of interest. If the image sizes of the image of interest and the image to be annotated are different, the first lesion evaluating unit 32 may change the image size of at least one of the image of interest and the image to be annotated, and normalize the image size.

[0054] Next, first lesion evaluation unit 32 inputs the image of interest into a first lesion detection model constructed by referencing first lesion detection model information D1, and detects a lesion area in the image of interest based on the inference result output by the first lesion detection model. In the example of Figure 4, first lesion evaluation unit 32 detects the area within bounding boxes 81 to 83 as a lesion area.

[0055] Next, the first lesion evaluation unit 32 determines whether the lesion area detected in the image of interest (also referred to as the "detected lesion area") is identical to the lesion area in the annotated image indicated by the metadata (also referred to as the "annotated lesion area"). In this case, the first lesion evaluation unit 32 recognizes, for example, a detected lesion area whose degree of overlap with the annotated lesion area (e.g., IOU) is equal to or greater than a predetermined degree as the lesion area of ​​the first lesion. The predetermined degree is set to a default value stored in the memory 12, for example. In the example of FIG. 4 , the first lesion evaluation unit 32 determines that the bounding box 80 and the bounding box 82 indicate the same lesion area of ​​the first lesion because the bounding box 82 of the image of interest overlaps the bounding box 80 of the annotated image by a degree equal to or greater than a predetermined degree.

[0056] If the first lesion indicated by the metadata is not detected in the image of interest, the first lesion evaluation unit 32 determines that the first lesion has disappeared as a result of treatment after the preliminary examination. In this case, for example, if there is an annotated lesion area whose overlapping degrees with each detected lesion area are less than a predetermined degree, the first lesion evaluation unit 32 determines that the first lesion corresponding to the annotated lesion area has disappeared.

[0057] Thereafter, first lesion evaluation unit 32 compares the sizes of bounding box 80 and bounding box 82, and performs RECIST evaluation, etc. On the other hand, first lesion evaluation unit 32 ignores the lesion detection results for bounding boxes 81 and 83 of the image of interest that are located away from bounding box 80 of the annotated image (i.e., the degree of overlap is less than a predetermined degree) and does not use them in the evaluation of the first lesion.

[0058] As described above, the first lesion evaluation unit 32 aligns the image of interest with the image to be annotated and evaluates the first lesion based on the detected lesion area that overlaps the annotated lesion area. This allows the first lesion evaluation unit 32 to appropriately identify the first lesion detected in the preliminary examination in the image of interest and accurately evaluate the first lesion. Furthermore, this method can appropriately identify the lesion area of ​​the first lesion even when the first lesion detection model detects multiple lesion areas, as in the example shown in FIG. 4 . Therefore, the detection threshold value set in the first lesion detection model can be lowered (i.e., the sensitivity of the first lesion detection model can be increased). As a result, it is possible to prevent the first lesion from being overlooked in the image of interest and improve the detection accuracy of the first lesion.

[0059] (4) Evaluation of the second lesion The second lesion evaluation unit 33 performs lesion detection on each image of the target examination image group obtained in the target examination, and determines whether the detected lesion area is a second lesion different from the first lesion based on the metadata of the pre-examination information D31 and the matching results on an image-by-image basis between the pre-examination image group and the target examination image group.

[0060] In a first example of the evaluation of the second lesion, the second lesion evaluation unit 33 performs evaluation of the second lesion based on the result of applying the second lesion detection model to each image in the target examination image group.

[0061] In this case, the second lesion evaluation unit 33 first sequentially selects images from the target examination image group as processing targets, inputs the selected images (also simply referred to as "target examination images") to a second lesion detection model constructed based on the second lesion detection model information D2, and detects a lesion area in the target examination image based on the inference results output by the second lesion detection model. If a lesion area is detected in the target examination image, the second lesion evaluation unit 33 identifies an image from the pre-examination image group (also simply referred to as a "corresponding pre-examination image") that is associated with the target examination image through matching by the image-of-interest extraction unit 31. The second lesion evaluation unit 33 then references the metadata attached to the corresponding pre-examination image to determine whether the detected lesion area is a second lesion. If there is no metadata attached to the corresponding pre-examination image, or if the annotated lesion area indicated by the metadata differs from the lesion area detected in the target examination image, the second lesion evaluation unit 33 determines that the lesion area detected in the target examination image may be a second lesion. In addition, the identity of the lesion area detected in the target examination image and the annotated lesion area can be determined, for example, using the same method as the method for determining the same lesion by the first lesion evaluation unit 32 (see Figure 4).

[0062] In a second example of evaluation of the second lesion, the second lesion evaluation unit 33 performs evaluation of the second lesion based on a comparison of the results of applying each image in the target test image group and each image in the pre-test image group to the second lesion detection model.

[0063] In this case, the second lesion evaluation unit 33 inputs the target examination images selected in order from the group of target examination images into the second lesion detection model, and detects the lesion area in the target examination image based on the inference result output by the second lesion detection model. The second lesion evaluation unit 33 also inputs the corresponding pre-examination image associated with the target examination image into the second lesion detection model, and detects the lesion area in the corresponding pre-examination image based on the inference result output by the second lesion detection model.

[0064] The second lesion evaluation unit 33 then determines whether the lesion area detected in the target examination image is likely to be a second lesion based on whether the lesion area detected in the target examination image corresponds to the lesion area indicated by the metadata and the lesion area detected in the corresponding preliminary examination image (i.e., whether they are the same lesion). For example, if a lesion area in the corresponding preliminary examination image is identical to the lesion area detected in the target examination image, but the metadata does not indicate this lesion area, the second lesion evaluation unit 33 determines that the lesion area detected in the target examination image is likely to be an overlooked lesion (i.e., a second lesion). Instead of determining the lesion area as an overlooked lesion, the second lesion evaluation unit 33 may determine that the lesion area is likely to be a site that is prone to erroneous detection. In another example, if there is no lesion area in the corresponding preliminary examination image that is identical to the lesion area detected in the target examination image, the second lesion evaluation unit 33 determines that the lesion area detected in the target examination image is likely to be a new lesion (i.e., a second lesion), regardless of the metadata.

[0065] On the other hand, if the second lesion evaluation unit 33 determines that the lesion area detected in the target examination image is the same lesion as the lesion area in the corresponding pre-examination image and the lesion area indicated by the metadata, it determines that the lesion area detected in the target examination image is the first lesion.

[0066] Then, the second lesion evaluation unit 33 supplies the evaluation result for the second lesion together with information generated in association with the evaluation for the second lesion to the display control unit 34. In this case, for example, the second lesion evaluation unit 33 supplies to the display control unit 34 information on the lesion area detected by the second lesion inspection model, information on the reliability (certainty) of the lesion area detected by the second lesion inspection model, information on the lesion area determined to be possibly the second lesion, and the like.

[0067] (5) Display Example Fig. 5 shows a display example of information based on the evaluations of the first lesion and the second lesion. The display control unit 34 generates display information based on the information supplied from the first lesion evaluation unit 32 and the second lesion evaluation unit 33, the examination information D3, etc., and transmits the generated display information to the display device 3, thereby causing the display device 3 to display the display screen shown in Fig. 5. In the display example shown in Fig. 5, the display control unit 34 of the image processing device 1 provides an image display area 70, an image selection area 71, a display switch selection area 72, a lesion of interest determination result display area 73, and a warning display area 74 on the display screen.

[0068] The display control unit 34 displays an arbitrary image selected by the user from the group of target examination images based on an operation on the input device 4 in the image display area 70. In the example of FIG. 5 , the display control unit 34 displays the 23rd image out of the group of 135 target examination images. Furthermore, in response to a selection made in a display switching selection area 72 (described later), the display control unit 34 displays a bounding box 91 indicating the region of the lesion of interest corresponding to the first lesion, a bounding box 92 indicating a region of a possible overlooked lesion, and a bounding box 93 indicating a region of a possible new lesion on the image. The display control unit 34 also displays a score (here, a range of 0 to 100) indicating the reliability (certainty) of the region being a lesion, in association with each of the bounding boxes 91 to 93. In this case, for example, the display control unit 34 displays the bounding box 91 indicating the region of the lesion of interest and the score based on the inference result output by the first lesion detection model supplied from the first lesion evaluation unit 32. On the other hand, the display control unit 34 displays a bounding box 92 indicating an area where a missed lesion may exist and a score, and a bounding box 93 indicating an area where a new lesion may exist and a score based on the inference results output by the second lesion detection model supplied from the second lesion evaluation unit 33.

[0069] The display control unit 34 may highlight, for example, the bounding boxes (bounding boxes 92 and 93 in the example of FIG. 5 ) related to the second lesions whose scores are equal to or greater than a predetermined threshold. In this case, the predetermined threshold is pre-stored in the memory 12 or the like. In this way, the display control unit 34 allows the viewer to preferably recognize the presence of a second lesion that is highly likely to exist.

[0070] The display control unit 34 also displays, in image units, thumbnail images of a portion of the target examination image group corresponding to a region close to the image displayed in the image display area 70 in the image selection area 71. The display control unit 34 may also display all thumbnail images of the target examination image group in the image selection area 71. The display control unit 34 may also sequentially switch the images displayed in the image display area 70 when detecting a selection operation in the image selection area 71 or a predetermined operation such as a scrolling operation from the input device 4. The display control unit 34 may also highlight thumbnail images of images with warnings in the image selection area 71. In the example of FIG. 5 , since the 23rd and 26th images have warnings, the display control unit 34 highlights these images with a border effect. An image with a warning is, for example, an image for which warning information is displayed in the warning display area 74 when selected as an image to be displayed in the image display area 70.

[0071] The display control unit 34 displays check boxes for selecting whether or not to display various types of information in the display switching selection area 72. In the example of Fig. 5, the display control unit 34 provides a first check box CH1 for selecting whether or not to display notable lesions, a second check box CH2 for selecting whether or not to display overlooked lesions, a third check box CH3 for selecting whether or not to display new lesions, and a fourth check box CH4 for selecting whether or not to display pre-examination results.

[0072] 5, because the first check box CH1 to the third check box CH3 are checked, the display control unit 34 displays bounding boxes 91 to 93 and their corresponding scores on the image in the image display area 70. When the fourth check box CH4, which selects whether or not to display the pre-examination results, is checked, the display control unit 34 refers to the pre-examination information D31 and superimposes area information of the lesion area indicated by the metadata attached to the image in the pre-examination image group corresponding to the image currently displayed in the image display area 70 on the image display area 70. The above-mentioned area information may be an image obtained by cutting out the lesion area indicated by the metadata from the image in the pre-examination image group, or may be a bounding box indicating the lesion area, etc.

[0073] The display control unit 34 also displays the RECIST evaluation (here, "PD") determined by the first lesion evaluation unit 32 in the lesion-of-interest evaluation result display area 73. Since the first lesion evaluation unit 32 has calculated that the size of the first lesion indicated by the bounding box 91 accompanying the RECIST evaluation has increased by x%, the display control unit 34 displays the calculation result, "x% increase," in the lesion-of-interest evaluation result display area 73.

[0074] Furthermore, the display control unit 34 displays the stage of the first lesion (here, "3") indicated by the bounding box 91 in the lesion determination result display area 73 of interest. Note that, for example, if the inference result output by the first lesion detection model includes an inference result regarding the stage of the lesion, the display control unit 34 recognizes the stage of the lesion based on the inference result of the first lesion detection model regarding the first lesion indicated by the bounding box 91. In another example, the display control unit 34 may input an image including the image region within the bounding box 91 to a machine learning model other than the first lesion detection model and the second lesion detection model, and recognize the stage of the lesion based on the inference result output by the machine learning model. In this case, the above-mentioned machine learning model is a model trained by machine learning to output an inference result regarding the stage of the lesion when an image including a lesion region is input, and trained model information is stored in advance in the memory 12, etc.

[0075] Furthermore, when predetermined criteria for outputting a warning are met, the display control unit 34 displays warning information corresponding to the met criteria in the warning display area 74. Here, because a lesion area (the area indicated by the bounding box 93) that may be a new lesion has been detected, the display control unit 34 displays a warning message indicating the possibility of a new lesion in the warning display area 74. The predetermined criteria for outputting a warning are not limited to the presence of a lesion area that may be a new lesion, but may be any criteria indicating a dangerous condition. For example, when the rate of increase in size of the first lesion displayed in the lesion-of-interest determination result display area 73 is equal to or greater than a predetermined rate, the display control unit 34 may display warning information in the warning display area 74 warning that the first lesion is growing.

[0076] 5, the viewer (the doctor in charge of the patient) can easily understand the evaluation results for the first lesion and the evaluation results for the second lesion (overlooked lesion, new lesion). Therefore, the image processing device 1 can effectively support the viewer, the doctor, in making decisions such as determining a treatment plan for the patient. The information based on the evaluation results for the first lesion and the second lesion described above is an example of information that supports decision-making.

[0077] 5 is merely an example, and various modifications may be applied. For example, instead of indicating the reliability score numerically, the display control unit 34 may determine the display mode of the corresponding bounding box (e.g., color, line type, and / or line thickness, etc.) based on the reliability score. In another example, instead of displaying the RECIST evaluation result, the growth rate of the first lesion, and the stage of the lesion using text, the display control unit 34 may display them in any visually distinguishable manner (e.g., color-coded).

[0078] (6) Processing Flow FIG. 6 is an example of a flowchart showing an outline of the processing executed by the image processing device 1.

[0079] First, the image processing device 1 acquires the pre-examination information D31 and the target examination information D32 of a target patient (step S11). In this case, the image processing device 1 may receive, via the input device 4, a user input specifying the target patient, a user input specifying the pre-examination information D31, a user input specifying the first lesion evaluation unit 32, and the like. Thereafter, the image processing device 1 sequentially or in parallel executes processes corresponding to steps S12 to S15 and a process corresponding to step S16.

[0080] In the process corresponding to steps S12 to S15, the image processing device 1 first extracts an image of interest corresponding to the annotated image from the target inspection image group included in the target inspection information D32 (step S12). In this case, the image processing device 1, for example, performs matching between the pre-inspection image group and the target inspection image group on an image-by-image basis, and extracts an image (and an adjacent image) from the target inspection image group that is associated with the annotated image included in the pre-inspection image group as the image of interest.

[0081] Next, the image processing device 1 aligns the position of the subject in the image of interest based on the image to be annotated (step S13). As a result, the image processing device 1 translates the subject in the image of interest so that the subject overlaps most closely between the image to be annotated and the image of interest. The image processing device 1 then detects a lesion area in the image of interest (step S14). In this case, the image processing device 1 inputs the image of interest into a first lesion detection model constructed with reference to first lesion detection model information D1, and detects a lesion area in the image of interest based on the inference result output by the first lesion detection model.

[0082] The image processing device 1 then evaluates the first lesion in the target examination image group (step S15). In this case, the image processing device 1 compares the lesion area indicated by the metadata in the annotated image with the lesion area detected in step S14, and measures changes in size and the like by regarding lesion areas that overlap to a predetermined degree or more as the same lesion.

[0083] On the other hand, in the process corresponding to step S16, the image processing device 1 evaluates each image in the group of target examination images for a second lesion (new lesion, overlooked lesion) (step S16). In this case, the image processing device 1 detects a lesion area corresponding to a second lesion different from the first lesion registered in the pre-examination information D31 from the group of target examination images, based on the second lesion detection model constructed with reference to the second lesion detection model information D2, the group of target examination images, and the pre-examination information D31.

[0084] Then, the image processing device 1 causes the display device 3 to display information based on the evaluation results in steps S15 and S16 (step S17).

[0085] (7) Modifications Next, preferred modifications of the first embodiment will be described. The following modifications may be applied in combination to the first embodiment.

[0086] (Modification 1) The image processing device 1 does not need to perform evaluation regarding the second lesion using the second lesion detection model.

[0087] For example, in the flowchart shown in Fig. 6 , the image processing device 1 may perform display based on the evaluation result obtained in step S15 in step S17 without performing processing equivalent to step S16. Furthermore, the image processing device 1 may perform evaluation of the second lesion based on the inference result of the first lesion detection model. For example, the image processing device 1 generates an evaluation result in which a lesion area (area within bounding boxes 81 and 83 in Fig. 4 ) that is determined to be a lesion area different from the first lesion among lesion areas detected by the first lesion detection model is determined to be a lesion area that may be the second lesion.

[0088] (Variation 2) The image processing device 1 may perform an evaluation of a progressing second lesion only when the evaluation of the first lesion is equal to or lower than a predetermined criterion. The predetermined criterion is a criterion determined in advance as an evaluation indicating the possibility of the occurrence of a new lesion (metastasis), such as "progression" in the RECIST evaluation.

[0089] 6, the image processing device 1 performs an evaluation of the first lesion based on each image of interest, and if there is an image of interest that is evaluated below a predetermined standard, the image processing device 1 executes step S16 and then step S17. On the other hand, if there is no image of interest that is evaluated below the predetermined standard, the image processing device 1 executes step S17 without executing step S16.

[0090] According to this aspect, the image processing device 1 performs evaluation of the second lesion only when there is a possibility that a new lesion will occur, and can suitably reduce the processing load.

[0091] (Variation 3) At least one of the first lesion detection model information D1, the second lesion detection model information D2, the preliminary examination information D31, and the target examination information D32 may be stored in a storage device separate from the image processing device 1.

[0092] 8 is a schematic diagram of a lesion evaluation system 100A according to a modified example. Lesion evaluation system 100A includes a server device 2 that stores at least one of first lesion detection model information D1, second lesion detection model information D2, pre-examination information D31, and target examination information D32. Lesion evaluation system 100A also includes multiple image processing devices 1 (1A, 1B, ...) that are capable of data communication with server device 2 via a network.

[0093] In this case, each image processing device 1 references at least one of the first lesion detection model information D1, second lesion detection model information D2, pre-examination information D31, and target examination information D32 via the network. In this case, the interface 13 of each image processing device 1 includes a communication interface such as a network adapter for communication. In this configuration, each image processing device 1 can reference the first lesion detection model information D1, second lesion detection model information D2, pre-examination information D31, and target examination information D32, as in the above-described embodiment, and preferably executes processing related to lesion evaluation. Note that the server device 2 may instead execute at least some of the processing executed by each functional block of the processor 11 of the image processing device 1 shown in FIG. 2 .

[0094] 8 is a block diagram of an image processing device 1X. The image processing device 1X includes an image-of-interest extraction unit 31X and a lesion evaluation unit 32X. The image processing device 1X may be composed of a plurality of devices.

[0095] The image-of-interest extraction means 31X extracts an image of interest to be analyzed for the lesion from the second group of medical images based on a comparison between a first group of medical images of the patient, each annotated with information about the first lesion, and a second group of medical images of the patient that were generated after the first group of medical images. The image-of-interest extraction means 31X can be, for example, the image-of-interest extraction unit 31 in the first embodiment (including modified examples, the same applies hereinafter).

[0096] The lesion evaluation means 32X evaluates the first lesion in the second medical image group based on the image of interest. The lesion evaluation means 32X can be, for example, the first lesion evaluation unit 32 in the first embodiment.

[0097] 9 is an example of a flowchart showing the processing procedure executed by the image processing device 1X. The image-of-interest extraction means 31X compares a first group of medical images of a patient, each annotated with information about a first lesion, with a second group of medical images of the patient that were generated after the first group of medical images, and extracts an image of interest from the second group of medical images to be analyzed for a lesion (step S21). The lesion evaluation means 32X evaluates the first lesion in the second group of medical images based on the image of interest (step S22).

[0098] According to the second embodiment, the image processing device 1X can suitably extract, based on the image, an image of interest from the second medical image group for evaluation of the first lesion that has annotation information in the first medical image group, and can efficiently and accurately perform evaluation of the first lesion in the second medical image group.

[0099] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0100] In addition, part or all of the above-described embodiments (including variations, the same applies below) may also be described as, but are not limited to, the following supplementary notes. Furthermore, not only the devices, methods, and storage media described in the supplementary notes, but also various hardware, software, various recording means (including storage media) for recording software, or systems may be made to depend on part or all of the configurations described in the supplementary notes, as long as they do not deviate from the above-described embodiments.

[0101] [Supplementary Note 1] An image processing device comprising: an image-of-interest extraction means for extracting an image of interest to be subjected to a lesion analysis from a second group of medical images of a patient, the second group of medical images having annotation information related to a first lesion, based on a comparison between the first group of medical images of the patient and the second group of medical images of the patient generated after the first group of medical images; and a lesion evaluation means for evaluating the first lesion in the second group of medical images based on the image of interest. [Supplementary Note 2] The image processing device described in Supplementary Note 1, wherein the image-of-interest extraction means performs image-by-image matching between the first group of medical images and the second group of medical images, and extracts at least an image from the second group of medical images that corresponds to an image having the annotation information. [Supplementary Note 3] The image processing device described in Supplementary Note 1, wherein, when the first lesion is present in the image of interest, the lesion evaluation means evaluates a change in the first lesion based on the size of the first lesion in the image having the annotation information and the size of the first lesion in the image of interest. [Supplementary Note 4] The image processing device of Supplementary Note 1, wherein the lesion evaluation means evaluates that the first lesion has disappeared if the first lesion is not present in the image of interest. [Supplementary Note 5] The image processing device of Supplementary Note 3, wherein the lesion evaluation means determines whether the first lesion is present in the image of interest based on the annotation information and a detection result of a lesion area in the image of interest. [Supplementary Note 6] The image processing device of Supplementary Note 5, wherein the lesion evaluation means acquires the detection result based on the image of interest and a machine learning model, and the machine learning model is a model that machine-learns the relationship between an image input to the machine learning model and a lesion area present in the image. [Supplementary Note 7] The image processing device of Supplementary Note 5, wherein the lesion evaluation means aligns the area of ​​the patient in the image to which the annotation information is added with the area of ​​the patient in the image of interest, and determines whether the first lesion is present in the image of interest based on the degree of overlap after the alignment between the lesion area detected in the image of interest and the area of ​​the first lesion indicated by the annotation information. [Supplementary Note 8] The image processing device according to Supplementary Note 1, further comprising a display control means for controlling display of information based on the result of the evaluation together with the target image on a display device as information to support decision-making.[Supplementary Note 9] The image processing device according to Supplementary Note 8, wherein the evaluation results include at least one of a RECIST evaluation result and a growth rate of the first lesion, and wherein the display control means performs display control to display information regarding at least one of the RECIST evaluation result and the growth rate on the display device. [Supplementary Note 10] The image processing device according to Supplementary Note 8, wherein the display control means displays warning information regarding the first lesion based on the evaluation results. [Supplementary Note 11] The image processing device according to Supplementary Note 1, further comprising second lesion evaluation means that performs evaluation of a second lesion not indicated by the annotation information based on the annotation information and the second medical image group. [Supplementary Note 12] The image processing device according to Supplementary Note 11, wherein the second lesion evaluation means performs evaluation of the second lesion based on a detection result of a lesion area in each image of the second medical image group and a detection result of a lesion area in an image of the first medical image group corresponding to each image of the second medical image group. [Supplementary Note 13] The image processing device of Supplementary Note 12, wherein the lesion evaluation means acquires, based on a machine learning model, detection results for a lesion area in each image of the second medical image group and detection results for a lesion area in an image of the first medical image group corresponding to each image of the second medical image group, and the machine learning model is a model that has learned the relationship between an image input to the machine learning model and a lesion area present in the image. [Supplementary Note 14] The image processing device of Supplementary Note 11, further comprising display control means for controlling display on a display device of information based on the evaluation results for the second lesion together with the image of the second medical image group in which the second lesion is detected. [Supplementary Note 15] The image processing device of Supplementary Note 14, wherein the display control means highlights the area in which the second lesion is detected on the image of the second medical image group. [Supplementary Note 16] The image processing device of Supplementary Note 14, wherein the display control means displays warning information for the second lesion based on the evaluation results for the second lesion.[Supplementary Note 17] An image processing method in which a computer compares a first group of medical images of a patient annotated with annotation information related to a first lesion with a second group of medical images of the patient generated after the first group of medical images, extracting an image of interest from the second group of medical images to be analyzed for a lesion, and evaluating the first lesion in the second group of medical images based on the image of interest. [Supplementary Note 18] A storage medium storing a program that causes a computer to execute a process of comparing a first group of medical images of a patient annotated with annotation information related to a first lesion with a second group of medical images of the patient generated after the first group of medical images, extracting an image of interest from the second group of medical images to be analyzed for a lesion, and evaluating the first lesion in the second group of medical images based on the image of interest. [Supplementary Note 19] The image processing method according to Supplementary Note 17, matching the first group of medical images with the second group of medical images on an image-by-image basis, and extracting at least an image from the second group of medical images that corresponds to an image annotated with the annotation information as the image of interest. [Supplementary Note 20] The image processing method according to Supplementary Note 17, in which, if the first lesion is present in the image of interest, a change in the first lesion is evaluated based on the size of the lesion area of ​​the first lesion in the image to which the annotation information is added and the size of the lesion area of ​​the first lesion in the image of interest. [Supplementary Note 21] The image processing method according to Supplementary Note 17, in which, if the first lesion is not present in the image of interest, the first lesion is evaluated to have disappeared. [Supplementary Note 22] The image processing method according to Supplementary Note 20, in which it is determined whether the first lesion is present in the image of interest based on the annotation information and a detection result of the lesion area in the image of interest. [Supplementary Note 23] The image processing method according to Supplementary Note 22, in which the detection result is obtained based on the image of interest and a machine learning model, and the machine learning model is a model that has learned by machine learning the relationship between an image input to the machine learning model and a lesion area present in the image.[Supplementary Note 24] The image processing method of Supplementary Note 22, further comprising: aligning a region of the patient in the image to which the annotation information is added with a region of the patient in the image of interest; and determining whether the first lesion is present in the image of interest based on the degree of overlap between the lesion region detected in the image of interest and the region of the first lesion indicated by the annotation information after the alignment. [Supplementary Note 25] The image processing method of Supplementary Note 17, further comprising: display control for displaying information based on the results of the evaluation together with the image of interest on a display device as information to support decision-making. [Supplementary Note 26] The image processing method of Supplementary Note 25, further comprising: display control for displaying information related to at least one of the results of the RECIST evaluation and a growth rate of the first lesion on the display device. [Supplementary Note 27] The image processing method of Supplementary Note 25, further comprising display control for displaying warning information related to the first lesion based on the results of the evaluation. [Supplementary Note 28] The image processing method according to Supplementary Note 17, wherein an evaluation of a second lesion not indicated by the annotation information is performed based on the annotation information and the second medical image group. [Supplementary Note 29] The image processing method according to Supplementary Note 28, wherein an evaluation of the second lesion is performed based on a detection result of a lesion area in each image of the second medical image group and a detection result of a lesion area in an image of the first medical image group corresponding to each image of the second medical image group. [Supplementary Note 30] The image processing method according to Supplementary Note 29, wherein a detection result of a lesion area in each image of the second medical image group and a detection result of a lesion area in an image of the first medical image group corresponding to each image of the second medical image group are obtained based on a machine learning model, the machine learning model being a model that has learned the relationship between images input to the machine learning model and lesion areas present in the images. [Supplementary Note 31] The image processing method according to Supplementary Note 28, wherein display control is performed to display information based on the evaluation result of the second lesion on a display device together with images of the second medical image group in which the second lesion was detected. [Supplementary Note 32] The image processing method according to Supplementary Note 31, wherein the region in which the second lesion is detected is highlighted on the images of the second medical image group.[Supplementary Note 33] The image processing method according to Supplementary Note 32, further comprising displaying warning information regarding the second lesion based on the result of the evaluation regarding the second lesion.

[0102] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0103] 1, 1A, 1B, 1X Image processing device 2 Server device 3 Display device 4 Input device 11 Processor 12 Memory 13 Interface 100, 100A Lesion evaluation system

Claims

1. An image processing apparatus comprising: a target image extraction means for extracting a target image to be analyzed for a lesion from a second medical image group of a patient, based on a comparison between a first medical image group of the patient with annotation information attached to a first lesion and the second medical image group of the patient generated after the first medical image group; and a lesion evaluation means for evaluating the first lesion in the second medical image group based on the target image.

2. The image processing apparatus according to claim 1, wherein the target image extraction means performs matching on an image-by-image basis between the first medical image group and the second medical image group, and extracts at least an image of the second medical image group associated with the image with the annotation information as the target image.

3. The image processing apparatus according to claim 1, wherein when the first lesion exists in the target image, the lesion evaluation means evaluates a change in the first lesion based on the size of the first lesion in the image with the annotation information and the size of the first lesion in the target image.

4. The image processing apparatus according to claim 1, wherein when the first lesion does not exist in the target image, the lesion evaluation means evaluates that the first lesion has disappeared.

5. The image processing apparatus according to claim 3, wherein the lesion evaluation means determines whether or not the first lesion exists in the target image based on the annotation information and a detection result of a lesion area in the target image.

6. The image processing apparatus according to claim 5, wherein the lesion evaluation means obtains the detection result based on the target image and a machine learning model, and the machine learning model is a model obtained by machine learning the relationship between an image input to the machine learning model and a lesion area existing in the image.

7. The image processing apparatus according to claim 5, wherein the lesion evaluation means aligns the area of the patient in the image with the annotation information and the area of the patient in the target image, and determines whether or not the first lesion exists in the target image based on the degree of overlap after alignment between the lesion area detected from the target image and the area of the first lesion indicated by the annotation information.

8. The image processing apparatus according to claim 1, further comprising a display control means for performing display control to display information based on the result of the evaluation, together with the target image, on a display device as information for assisting in decision-making.

9. The result of the evaluation includes at least one of the result of the RECIST evaluation and the growth rate of the first lesion, and the display control means performs display control to display information regarding at least one of the result of the RECIST evaluation and the growth rate on the display device. The image processing apparatus according to claim 8.

10. The display control means displays warning information regarding the first lesion based on the result of the evaluation. The image processing apparatus according to claim 8.

11. The image processing apparatus according to claim 1, further comprising second lesion evaluation means for performing an evaluation regarding a second lesion not indicated by the annotation information based on the annotation information and the second group of medical images.

12. The second lesion evaluation means performs an evaluation regarding the second lesion based on the detection result of the lesion region in each image of the second group of medical images and the detection result of the lesion region in the image of the first group of medical images corresponding to each image of the second group of medical images. The image processing apparatus according to claim 11.

13. The lesion evaluation means acquires, based on a machine learning model, the detection result of the lesion region in each image of the second group of medical images and the detection result of the lesion region in the image of the first group of medical images corresponding to each image of the second group of medical images, and the machine learning model is a model that has learned the relationship between the image input to the machine learning model and the lesion region existing in the image. The image processing apparatus according to claim 12.

14. The image processing apparatus according to claim 11, further comprising display control means for performing display control to display information based on the result of the evaluation regarding the second lesion on the display device together with the image of the second group of medical images in which the second lesion is detected.

15. The display control means highlights the region in which the second lesion is detected on the image of the second group of medical images. The image processing apparatus according to claim 14.

16. The display control means displays warning information regarding the second lesion based on the result of the evaluation regarding the second lesion. The image processing apparatus according to claim 14.

17. A computer extracts a target image to be analyzed for lesions from a second group of medical images of a patient, which is generated after the first group of medical images of the patient with annotation information regarding a first lesion, based on a comparison between the first group of medical images and the second group of medical images, and performs an evaluation regarding the first lesion in the second group of medical images based on the target image. An image processing method.

18. A storage medium storing a program that causes a computer to execute a process of extracting a target image to be analyzed for lesions from a second group of medical images of a patient, which is generated after the first group of medical images of the patient with annotation information regarding a first lesion, based on a comparison between the first group of medical images and the second group of medical images, and performing an evaluation regarding the first lesion in the second group of medical images based on the target image.

19. The image processing method according to claim 17, wherein matching is performed on an image-by-image basis between the first group of medical images and the second group of medical images, and at least an image of the second group of medical images associated with the image with the annotation information is extracted as the target image.

20. The image processing method according to claim 17, wherein when the first lesion exists in the target image, a change in the first lesion is evaluated based on the size of the lesion area of the first lesion in the image with the annotation information and the size of the lesion area of the first lesion in the target image.

Images