Image generation device, method for generating image, method of learning, and program

Abstract

To provide a mechanism that can preferably acquire an image in which contrast effect corresponding to a contrast time period including contrast time at a certain point is visualized.SOLUTION: An image generation device 20 includes: an image acquisition part 251 for acquiring a medical image; and an output part 252 for outputting a contrast effect image in which contrast effect corresponding to a contrast time period including contrast time in at least one certain point is visualized on the basis of the medical image acquired by the image acquisition part 251.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generating device, an image generating method, a learning method, and a program. [Background technology]

[0002] In the medical field, in order to identify a disease of a subject or observe the extent of the disease, contrast images are obtained over time using a contrast agent that can emphasize the flow of blood, etc., and are used for diagnosis. For example, contrast examinations are performed using various imaging devices, such as fluorescein fundus angiography (FA) examinations using a fundus camera, multi-phase contrast examinations using X-ray computed tomography (CT) devices, and Sonazoid contrast ultrasound examinations using ultrasound examination diagnostic devices (echo). However, while contrast images obtained by contrast examinations are often useful as diagnostic information, contrast agents can cause serious symptoms in some subjects, and examinations using radiation can have adverse effects due to exposure. Therefore, taking this into consideration, contrast examinations may not be performed multiple times, or may not be performed at all.

[0003] In recent deep learning technology, it has also been proposed to convert an image of a certain domain into an image of a different domain. For example, Patent Document 1 proposes a method of generating a model that outputs an image reproducing a diagram showing an abnormal area when a fundus examination image is input. In addition, Non-Patent Document 1 proposes a method of generating a model that outputs an image similar to an FA examination image when a retinal fundus photograph without a contrast agent is input. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 142910 [Non-patent literature]

[0005] [Non-Patent Document 1] Alireza Tavakkoli, Sharif Amit Kamran, Khondker Fariha Hossain, Stewart Lee Zuckerbrod, "A novel deep learning conditional generative adversarial network for producing angiography images from retinal fundus photographs.",Sci Rep 10, 21580(2020),<https: / / doi.org / 10.1038 / s41598-020-78696-2> (Released on December 9, 2020) Summary of the Invention [Problem to be solved by the invention]

[0006] However, the techniques described in Patent Document 1 and Non-Patent Document 1 are insufficient for suitably acquiring an image depicting a contrast effect corresponding to a contrast time including a contrast time at a certain point in time.

[0007] The present invention has been made in consideration of such problems, and aims to provide a mechanism that can suitably acquire an image in which a contrast effect corresponding to a contrast time including a contrast time at a certain point in time is depicted. [Means for solving the problem]

[0008] The image generating device of the present invention includes an image acquiring unit that acquires a medical image, and an output unit that uses an image generation model that inputs the medical image and generates a contrast effect image that depicts a contrast effect, and outputs a contrast effect image that depicts a contrast effect corresponding to a contrast time including at least one contrast time point, based on the medical image acquired by the image acquiring unit. The image generating device of the present invention also includes a learning unit that uses learning data including a medical image group, a contrast image group related to the medical image group, and a group of imaging conditions related to the contrast image group, the group of imaging conditions including a contrast time including at least one contrast time point, to train an image generation model that generates a contrast effect image in which a contrast effect corresponding to the contrast time is depicted based on the medical image when a medical image related to the medical image group and the contrast time are input. Furthermore, the present invention includes an image generating method, a learning method, and a program for causing a computer to function as each part of the image generating device described above. Effect of the Invention

[0009] According to the present invention, it is possible to suitably obtain an image depicting a contrast effect corresponding to a contrast time including a contrast time at a certain point in time. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of an image generation system including an image generation device according to a first embodiment. [Diagram 2] 2 is a diagram for explaining the concept of an image generation model that an output unit has in the image generating device according to the first embodiment. FIG. [Diagram 3] 2 is a diagram for explaining learning of an image generation model included in an output unit in the image generating device according to the first embodiment. FIG. [Figure 4] 3 is a diagram for explaining a calculation target region of a loss calculated when training an image generation model that an output unit has in the image generating device according to the first embodiment. FIG. [Diagram 5] 3 is a diagram showing an example of a GUI screen displayed on a display in the image generating device according to the first embodiment. FIG. [Figure 6] 5 is a flowchart showing an example of a processing procedure in a control method for an image generating device according to the first embodiment. [Figure 7]FIG. 13 illustrates a first modified example of the first embodiment, and is a diagram for explaining a period of time (contrast time) during which FA examination images, which are moving images constituting teacher data used when the image generation model learns, are recorded. [Figure 8] FIG. 13 illustrates a first modified example of the first embodiment, showing an example of the relationship between a contrast effect image, which is a moving image output by an image generation model, and a correct answer image (FA test image), which is a moving image constituting training data. [Figure 9] FIG. 13 illustrates a second modified example of the first embodiment, showing an example of an OCTA image and an FA inspection image. [Figure 10] 13 is a flowchart illustrating an example of a processing procedure in a registration process between an OCTA image and an FA inspection image according to a second modified example of the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a schematic configuration of an image generation system including an image generation device according to a second embodiment. [Figure 12] FIG. 11 is a diagram showing an example of a GUI screen displayed on a display in an image generating device according to a second embodiment. [Figure 13] 10 is a flowchart showing an example of a processing procedure in a control method for an image generating device according to a second embodiment. [Figure 14] FIG. 13 is a diagram for explaining the concept of an image generation model that an output unit has in an image generating device according to a third embodiment. [Figure 15] FIG. 13 is a diagram for explaining the concept of an image generation model that an output unit has in an image generating device according to a third embodiment. [Figure 16] FIG. 13 illustrates the third embodiment and is a diagram showing an example of a period with and without FA inspection images of the left and right eyes constituting training data used when the image generation model is trained. [Figure 17] FIG. 13 is a diagram for explaining learning of an image generation model included in an output unit in an image generating device according to a third embodiment. [Figure 18] 13 is a flowchart showing an example of a processing procedure in a control method for an image generating device according to a first modified example of the third embodiment. [Figure 19]13 is a flowchart illustrating an example of a processing procedure in a generation process of an interpolated image, illustrating a third modified example of the third embodiment. [Figure 20] FIG. 13 illustrates a third modified example of the third embodiment, showing an example of a period with and without FA inspection images constituting training data used when the image generation model is trained. [Figure 21] FIG. 21 illustrates a third modified example of the third embodiment and is a diagram for explaining an effective pixel region common to the immediately preceding FA inspection image and the immediately following FA inspection image illustrated in FIG. 20. [Figure 22] FIG. 13 illustrates a third modified example of the third embodiment, showing an example of a period with and without FA inspection images constituting training data used when the image generation model is trained. [Figure 23] FIG. 23 illustrates a third modified example of the third embodiment and is a diagram for explaining an effective pixel region in a case where the immediately subsequent FA inspection image illustrated in FIG. 22 is the FA inspection image captured first in the FA inspection. [Figure 24] FIG. 13 is a diagram for explaining the concept of an image generation model that an output unit has in an image generating device according to a fourth embodiment. [Diagram 25] FIG. 13 illustrates a fourth embodiment and is a diagram for explaining the presence or absence of an FA inspection image constituting training data used when the image generation model is trained. [Figure 26] FIG. 13 is a diagram for explaining learning of an image generation model included in an output unit in an image generating device according to a fourth embodiment. [Figure 27] FIG. 13 is a diagram for explaining learning of an image generation model included in an output unit in an image generating device according to a fourth embodiment. [Figure 28] FIG. 13 is a diagram showing an example of a GUI screen displayed on a display in an image generating device according to a fourth embodiment. [Figure 29] 13 is a flowchart showing an example of a processing procedure in a control method for an image generating device according to a fourth embodiment. [Diagram 30] FIG. 13 is a diagram for explaining the concept of an image generation model that an output unit has in an image generating device according to a fifth embodiment. [Diagram 31] FIG. 13 is a diagram for explaining learning of an image generation model that an output unit has in an image generating device according to a fifth embodiment. [Diagram 32] 13 is a flowchart showing an example of a processing procedure in a control method for an image generating device according to a sixth embodiment. [Diagram 33] FIG. 13 is a diagram showing an example of a GUI screen displayed on a display in an image generating device according to a seventh embodiment. [Diagram 34] FIG. 13 is a diagram illustrating an example of a schematic configuration of an image generation model generating device according to an eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a mode (embodiment) for carrying out the present invention will be described with reference to the drawings. In addition, in the embodiment of the present invention described below, examples are given assuming still images or moving images in two-dimensional or three-dimensional images, but for ease of understanding, the drawings include explanations using still images in two-dimensional images. In other words, images handled in the embodiment of the present invention described below are not limited to still images in two-dimensional images.

[0012] (First embodiment) First, the first embodiment will be described.

[0013] FIG. 1 is a diagram showing an example of a schematic configuration of an image generation system 1 including an image generation device 20 according to a first embodiment. As shown in FIG. 1, the image generation system 1 includes a photographing device 10, an image generation device 20, and a network 30. The photographing device 10 and the image generation device 20 are communicatively connected via the network 30. Note that the schematic configuration of the image generation system 1 shown in FIG. 1 is an example, and the number of each device may be changed as desired. Furthermore, in the image generation system 1, devices not shown in FIG. 1 may be connected to the network 30.

[0014] In the first embodiment, the photographing device 10 is, for example, an optical coherence tomography (OCT) device capable of photographing the fundus of the subject's eye. Note that in the first embodiment, it is sufficient that the photographing device 10 can acquire an OCTA (Optical Coherence Tomography Angiography) image, which is a medical image derived from photographing with the OCT device. For this reason, for example, the photographing device 10 may be replaced by an image management system that stores and manages the OCTA images.

[0015] As shown in FIG. 1, the image generating device 20 includes a NW (network) interface 210, an input interface 220, a display 230 which is a display device, a memory circuit 240, and a processing circuit 250.

[0016] The NW interface 210 is communicatively connected to the input interface 220, the display 230, the storage circuit 240, and the processing circuit 250. The NW interface 210 controls the transmission and communication of various information and various data (including image data) between each device connected via the network 30. For example, the NW interface 210 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0017] The input interface 220 is communicatively connected to the NW interface 210, the display 230, the memory circuit 240, and the processing circuit 250. The input interface 220 converts an input operation received from an operator into an input signal, which is an electric signal, and inputs the input signal to the processing circuit 250, etc. For example, the input interface 220 can be realized by a trackball, a switch button, a mouse, a keyboard, etc. Also, for example, the input interface 220 can be realized by a touch pad that performs an input operation by touching the operation surface, a touch screen in which a display screen and a touch pad are integrated, a non-contact input circuit using an optical sensor, a voice input circuit, etc. Note that the input interface 220 is not limited to only those that include physical operation parts such as a mouse and a keyboard. For example, the input interface 220 also includes a component that receives an electric signal corresponding to an input operation from an external input device provided separately from the image generating device 20 and inputs the electric signal as an input signal to the processing circuit 250, etc.

[0018] The display 230 is communicatively connected to the NW interface 210, the input interface 220, the memory circuit 240, and the processing circuit 250. The display 230 displays various information and various data (including image data) output from the processing circuit 250. For example, the display 230 is realized by a liquid crystal display, a CRT (Cathode Ray Tube) display, an organic EL display, a plasma display, a touch panel, or the like.

[0019] The memory circuitry 240 is communicatively connected to the NW interface 210, the input interface 220, the display 230, and the processing circuitry 250. The memory circuitry 240 stores various types of information and various types of data (including image data). Furthermore, the memory circuitry 240 stores, for example, programs that the processing circuitry 250 reads out and executes to realize various functions. For example, the memory circuitry 240 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, or the like.

[0020] The processing circuitry 250 comprehensively controls the operation of the image generating device 20 and performs various processes. As shown in FIG. 1, the processing circuitry 250 includes an image acquisition unit 251, an output unit 252, and a display unit 253. In this embodiment, a program for causing each component (251 to 253) of the processing circuitry 250 to function is stored in the storage circuitry 240 in the form of a program executable by a computer. For example, the processing circuitry 250 is a processor that realizes the functions of each component (251 to 253) by reading and executing a program from the storage circuitry 240. Note that, in FIG. 1, the processing circuitry 250 has been described as a single processor that realizes the image acquisition unit 251, the output unit 252, and the display unit 253, but the processing circuitry 250 may be configured by combining multiple independent processors. In this case, each processor in the multiple independent processors that configure the processing circuitry 250 may realize the functions of each component (251 to 253) by executing a program.

[0021] 1, the description has been given on the assumption that the storage circuit 240 is a single storage circuit, but the storage circuit 240 may be distributed across a plurality of storage circuits. In this case, the processing circuit 250 may read out and execute a corresponding program from each storage circuit.

[0022] The term "processor" may mean, for example, a central processing unit (CPU) or a graphical processing unit (GPU). The term "processor" may mean, for example, an application specific integrated circuit (ASIC). The term "processor" may mean, for example, a programmable logic device (for example, a simple programmable logic device (SPLD)). The term "processor" may mean, for example, a complex programmable logic device (CPLD). The term "processor" may mean, for example, a field programmable gate array (FPGA). In this embodiment, the processor realizes the function of each component by reading and executing a program stored in the storage circuit 240. Note that instead of storing a program in the storage circuit 240, the processor may be configured to directly incorporate the program into the circuit of the processor. In this case, the processor realizes the function of each component by reading and executing the program incorporated in the circuit.

[0023] The image acquisition unit 251 has a function of acquiring a medical image, which is a still image of an object to be examined (in this embodiment, the subject's eye) acquired by the photographing device 10. Specifically, the medical image in this embodiment is, for example, an OCTA image, which is a fundus examination image of the fundus of the subject's eye. Here, the OCTA image will be described. The OCTA image is an image generated as a blood vessel image in the fundus of the subject's eye by projecting three-dimensional motion contrast data of the fundus of the subject's eye acquired by an OCT device applied as the photographing device 10 onto a two-dimensional plane. Here, the motion contrast data is data obtained by repeatedly photographing the same cross section of the measurement object (in this embodiment, the fundus of the subject's eye) with the OCT device and detecting a temporal change in the measurement object between the photographs. This motion contrast data is obtained, for example, by calculating the temporal change in phase, vector, and intensity of a complex OCT signal from a difference, ratio, correlation, or the like. A two-dimensional front image of the fundus of the subject's eye is generated as an OCTA image by specifying a depth range of a layer or the like in the fundus of the subject's eye from this motion contrast data. That is, by specifying different depth ranges in the fundus of the subject eye, it is possible to generate OCTA images of any range, such as the superficial layer, deep layer, outer layer, and choroidal vascular network. Note that the type of OCTA image is not limited to this, and OCTA images with different depth ranges set by changing the reference layer and offset value may be generated. Note that in this embodiment, an OCTA image of the superficial layer in the fundus of the subject eye and a fluorescein fundus angiography (FA) examination image are used as examples for explanation.

[0024] The output unit 252 has a function of outputting a contrast effect image depicting a contrast effect corresponding to a contrast time including at least one contrast time based on an OCTA image, which is a medical image acquired by the image acquisition unit 251. In particular, the output unit 252 outputs a contrast effect image equivalent to a still image when the contrast time includes only one contrast time, and outputs a contrast effect image equivalent to a moving image composed of multiple still images when the contrast time includes multiple contrast times. In this embodiment, the output unit 252 outputs a moving image as a contrast effect image corresponding to a contrast time including multiple contrast times. Specifically, the contrast effect image in this embodiment is a pseudo contrast image like an FA examination image in a moving image format depicting a change in contrast effect over time, as acquired in an FA examination. In addition, the output unit 252 in this embodiment sets a predetermined FPS (Frames Per Second), such as 10 frames per second, at which a change in contrast effect is easily observed, as the playback speed of the contrast effect image, which is a moving image. In addition, the output unit 252 may output the contrast effect image, for example, to the memory circuitry 240, or may output it to another device (not shown) via the NW interface 210 and the network 30, and may also be in a form that simultaneously outputs it to the display 230.

[0025] The display unit 253 has a function of displaying the contrast effect image output from the output unit 252 on the display 230 so that the operator can easily observe it.

[0026] In the present embodiment, the output unit 252 has an image generation model that inputs a medical image, which is a still image, and outputs a contrast effect image of a moving image in which the contrast effect corresponding to the contrast time including the contrast time of a plurality of points in time is depicted based on the medical image.

[0027] FIG. 2 is a diagram for explaining the concept of an image generation model 2520 included in the output unit 252 in the image generating device 20 according to the first embodiment.

[0028] 2 is a model including an image processing system that outputs a contrast effect image by, for example, a rule base or machine learning (particularly, deep learning technology). In this embodiment, the image generation model 2520 is a model trained using training data including, for example, a medical image group related to a medical image, a contrast image group related to the medical image group, and an imaging condition group related to the contrast image group. The image generation model 2520 including an image processing system by deep learning technology will be described below.

[0029] The image generation model 2520 shown in FIG. 2 has a network model 2521 based on U-Net as an image processing system using deep learning technology. Here, U-Net is a known network model using deep learning technology. Specifically, U-Net is trained with a dataset consisting of a pair of images of an input image and a corresponding output image. Then, when an image is input to the image generation model 2520 including a fully trained U-Net, a plausible image corresponding to the input image can be output according to the tendency of the dataset used for training. For example, it is known that it can be applied to image segmentation processing, image quality improvement, image domain conversion, etc. according to the dataset.

[0030] 2, the image generation model 2520 converts the input image St101, which is a still image, into a tensor and inputs it to the network model 2521, converts the tensor output by the network model 2521 into a moving image, and outputs the output image Mo111. In this case, when U-Net is adopted as the network model 2521, it is necessary to modify U-Net. Here, the tensor in the description of this embodiment refers to a format in which a pixel value group of an image or the like is expressed as a multidimensional array, and is a data input / output format to the network model 2521, and furthermore, images and tensors can be converted to each other.

[0031] A specific example will be described below. The total number of video frames of the output image Mo111, which is a video image to be output, is N, and the shape of the input image St101, which is a single still image, converted into a tensor is "Cin ×H in ×W in Let us consider the case where "C in " is the number of channels, and "H in ' is the height of the input tensor and 'W in " is the width of the input tensor, and in particular "C in If " is 1, the spatial axis of the number of channels can be ignored. In the network model 2521, which is a modified U-Net, the elements that make up the input tensor are increased, and shape deformation is performed up to the final layer to obtain "N × C out ×H out ×W out , where H out " is the height of the output tensor, and "W out " is the width of the output tensor. The tensor output from the network model 2521 is divided into N "C out ×H out ×W out ", and each of the divided tensors is converted into a video frame image. The converted video frame images are concatenated and output from the image generation model 2520 as an output image Mo111, which is a single video image. The shape of the tensor is not limited to the shape shown in the description of this embodiment, and may be other shapes that can achieve the same purpose. In addition, although U-Net is exemplified in this embodiment, other network models that can achieve the same purpose may be adopted. In addition, although two-dimensional images are handled in this embodiment, when three-dimensional images are handled in another embodiment, it is possible to handle it by further adding a depth space to the shape of the tensor described here.

[0032] A data set for learning the image generation model 2520 including the network model 2521 based on U-Net will be described below. The data set is configured as a group of teacher data acquired from a plurality of test subjects, with a pair of teacher data consisting of an OCTA image, which is a still image obtained by photographing the same test subject (i.e., the subject's eye), and an FA test image, which is a moving image during a period of a predetermined contrast time (contrast time). The contrast time is a time indicating the elapsed time from a reference time point (reference time point) such as the time when a contrast agent is injected into the subject, the time when the first image is taken, or the time when the contrast effect of an organ is first confirmed in the acquired image. The period of a predetermined contrast time (contrast time) is a period defined such that the contrast time is from 0 seconds to 60 seconds. If the FA test image is a moving image of 1 FPS, there are 61 moving image frame images corresponding to 61 contrast times for each second in the period. A part or all of the moving image frame images constituting the FA test image, which is a moving image, may be supplemented by the FA test image, which is a still image.

[0033] Here, the FA examination image, which is a moving image for a period of a predetermined contrast time (contrast time), may not be composed of the same number of moving image frame images depending on the type and settings of the imaging device 10. Therefore, sampling of moving image frame images is performed so that the number of moving image frame images constituting the FA examination image, which is a moving image constituting the teacher data, is constant for all teacher data. Then, by performing the above sampling as necessary, the FA examination image, which is a moving image constituting the final data set, is constructed with a constant number of moving image frame images. At this time, the number of moving image frame images matches the number of moving image frame images of the contrast effect image, which is a moving image output by the image generation model 2520.

[0034] Depending on the configuration of the network model 2521, better results may be obtained if the input image and the correct answer image in the teacher data are aligned. Specifically, in the network model 2521 based on U-Net, it is preferable that the OCTA image of the input image in the teacher data acquired by photographing the same test subject and each of the video frame images constituting the FA test image, which is the correct answer image, are aligned. For example, if the alignment is performed anatomically by manual image processing or image registration processing, the depiction mode of the contrast effect in the contrast effect image output by the image generation model 2520 becomes closer to the real FA test image. Note that since the OCTA image and the FA test image are images acquired by different types of imaging devices, the depiction mode is significantly different, and anatomical alignment may be difficult depending on conditions such as the contrast time. In this case, first, among the sets (pairs) of video frame images constituting an OCTA image and an FA examination image, for at least one pair that is relatively easy to anatomically align, the video frame images are deformed and aligned with reference to the anatomical position of the OCTA image. Next, the remaining video frame images are deformed and aligned with reference to the anatomical position of the deformed video frame images. As a result, even in a situation where anatomical alignment between an OCTA image and an FA examination image is difficult, better anatomical alignment can be performed. As a result, the depiction of the contrast effect in the contrast effect image output by the image generation model 2520 becomes closer to the real FA examination image.

[0035] Fig. 3 is a diagram for explaining the learning of the image generation model 2520 possessed by the output unit 252 in the image generating device 20 according to the first embodiment. In Fig. 3, the same components as those shown in Fig. 2 are given the same reference numerals, and detailed explanations thereof will be omitted. Hereinafter, learning of the image generation model 2520 using a certain set of teacher data, that is, updating processing of a group of parameters constituting a network model 2521 included in the image generation model 2520, will be explained with reference to Fig. 3.

[0036] First, in FIG. 3, when an input tensor Te102 obtained by tensorizing an OCTA image constituting the teacher data is input to a network model 2521, an output tensor Te112 corresponding to a contrast effect image of a moving image is output from the network model 2521. Next, the image generation model 2520 calculates a loss Lo132 which is an error between a correct answer tensor Te122 obtained by tensorizing an FA inspection image, which is a moving image constituting the same teacher data, and the output tensor Te112. Finally, the image generation model 2520 updates a group of parameters constituting the network model 2521 so that the loss Lo132 becomes small. This series of update processes is repeated using a group of teacher data allocated for training among the data set until the network model 2521 has learned sufficiently. Here, for the sake of explanation, an example is shown in which one set of teacher data is used in one update process, but multiple groups of teacher data may be used in one update process for the purpose of shortening the learning time, stabilizing the learning process, etc. In addition, if the image generation model 2520 has been sufficiently trained during the learning process, for example by performing an accuracy evaluation using training data for verification, it may be determined that the image generation accuracy is sufficiently high and the learning process may be interrupted (early stopping).

[0037] The calculation method based on the following method can be adopted for the accuracy evaluation and error (loss) calculation between the FA inspection image (or its tensor) in the teacher data assigned for training or verification and the contrast effect image (or its tensor) output by the image generation model 2520. Specifically, for example, a method of quantifying the error or similarity by a method such as MSE (Mean Squared Error) or SSIM (Structural Similarity) can be adopted. Here, in order to perform the accuracy evaluation and the error (loss) calculation for the moving image, a calculation method based on the method such as MSE or SSIM is used in either a moving image or still image form. In this case, as a form for the moving image, a form in which the calculation is performed on a multidimensional array of "width x height x time" of the moving image is considered. Also, as a form for the still image, a form in which the calculation is performed on a multidimensional array of "width x height" of the moving image frame images constituting the moving image is considered to be the average of the results obtained by the calculation.

[0038] In the learning of the image generation model 2520, the calculation target for the accuracy evaluation and the calculation of the error (loss) may be selected in consideration of a semantic area, which is an area in an image included in the learning data and can be divided according to the aspect depicted in the image or information related to the image. Specifically, the semantic area includes a masked area or a non-masked area depicted in the image included in the learning data, a printed area of ​​patient information or imaging information (date and time, imaging protocol name, etc.), and an area related to the part or state of an organ (normal tissue, abnormal tissue, bleeding, inflammation, white spot, treatment scar, etc.). Furthermore, the semantic area includes a bright area or a dark area, a high-image-quality area or a low-image-quality area, and an area where image processing such as alignment is successful or unsuccessful in the image included in the learning data. In this way, the semantic area is an area in an image included in the learning data and can be divided according to the aspect depicted in the image or information related to the image. For example, in a fundus photograph or an FA examination image acquired by a fundus camera, a masked area (an area filled in black, etc.) is depicted in the periphery of the image depending on the shooting angle. Since this masked region is a region in which organs are not depicted (does not affect diagnosis), when training the image generation model 2520, the performance and characteristics of the image generation model 2520 may be adjusted by using only the non-masked region that affects the diagnosis as the subject of accuracy evaluation and error (loss) calculation.

[0039] Fig. 4 is a diagram for explaining a calculation target region of a loss calculated when learning an image generation model 2520 of an output unit 252 in an image generating device 20 according to the first embodiment. In Fig. 4, the same components as those shown in Fig. 3 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0040] For example, in an FA inspection image, as shown in FIG. 4, a masked region Se151 may be depicted, and the masked region Se151 may be excluded from the accuracy evaluation and error (loss) calculation. Note that, when using a calculation method that considers the difference between pixels at the same coordinates between images, such as MSE, in the accuracy evaluation and error (loss) calculation between multiple images considering the semantic region, attention should be paid to making the pixel region to be calculated common between multiple images. A specific explanation will be given with reference to FIG. 4. When calculating the loss Lo133, the non-masked region Se152 of the correct tensor Te122 of the FA inspection image and the region Se142 in the output tensor Te112 of the contrast effect image that corresponds in coordinates to the non-masked region Se152 are set as the calculation target regions.

[0041] In addition, when the image to be subjected to the accuracy evaluation and the error (loss) calculation is a moving image, the position and type of the semantic area may change for each moving image frame image constituting the moving image. Therefore, the method of calculating the accuracy evaluation and the error (loss) and the calculation target area may be changed correspondingly for each moving image frame image. In particular, when the loss Lo132 for updating the parameter group constituting the network model 2521 is calculated, if only the non-mask area Se152 is targeted, the contrast effect image output by the image generation model 2520 will not depict a state corresponding to the mask area Se151. In other words, since the contrast effect is depicted in the area Se141, the contrast effect image is one in which the contrast effect for the entire area depicted in the OCTA image input to the image generation model 2520 can be observed. Conversely, by not considering the semantic area, a state corresponding to the mask area Se151 may be depicted, and an image closer to a real contrast image may be presented to the operator, thereby reducing the sense of incongruity. In addition, image processing using a known rule base or machine learning can be used to extract the semantic region to be the target of the accuracy evaluation and the error (loss) calculation. In addition, since the non-mask region in the FA inspection image is a fixed region determined by the imaging device 10, it may be mechanically extracted and used as the target of the accuracy evaluation and the error (loss) calculation.

[0042] Here, a method for updating (optimizing) parameters constituting the network model 2521 based on the error between the correct tensor Te122 and the output tensor Te112 output by the network model 2521 in order to learn the image generation model 2520 has been described. However, the present embodiment is not limited to this method. Parameters constituting the network model 2521 may be updated by applying a technique related to GAN with an image input, such as Conditional GAN ​​(Generative Adversarial Network), which is a well-known deep learning technique. For example, the parameters constituting the network model 2521 may be updated while making the following judgment on the contrast effect image generated by the network model 2521 corresponding to the Generator Network in the Conditional GAN. Specifically, the parameters constituting the network model 2521 may be updated while judging whether the contrast effect image is real (FA examination image) or fake (FA examination image-like image) by the Discriminator Network.

[0043] When an OCTA image is input, the image generation model 2520 trained by the above process can output a contrast effect image of a moving image depicting a plausible contrast effect based on a group of teacher data assigned for training among the data set. In other words, it can output a pseudo contrast image (contrast effect image) like an FA examination image in a moving image format depicting a change over time in the contrast effect as obtained in an FA examination.

[0044] FIG. 5 is a diagram showing an example of a GUI screen 400 displayed on the display 230 in the image generating device 20 according to the first embodiment.

[0045] The display unit 253 performs a process of displaying a GUI (Graphical User Interface) screen 400 as shown in FIG. 5 on the display 230. Specifically, the display unit 253 performs a process of displaying a medical image (an OCTA image in this embodiment) acquired by the image acquisition unit 251 in an image display area 410 of the GUI screen 400 shown in FIG. 5. The display unit 253 also performs a process of displaying a contrast effect image output from the output unit 252 in an image display area 420 of the GUI screen 400 shown in FIG. 5. More specifically, in this embodiment, the display unit 253 performs a process of displaying a contrast effect image of a moving image in the image display area 420. This allows the operator to observe the contrast effect image by visually checking the image display area 420 of the GUI screen 400. In addition, an operation tool that allows the operator to operate the moving image in the contrast effect image is provided in the image display area 420 of the GUI screen 400. The image display area 420 is provided with operation tools such as a play button 421 for starting playback of a moving image, a pause button 422 for pausing playback of the moving image, a stop button 423 for stopping playback of the moving image, and a seek bar 424 for changing the playback position of the moving image. Note that the playback of the moving image in the contrast effect image displayed in the image display area 420 may be started automatically, or may be stopped at a playback position corresponding to the contrast time useful for diagnosis. In addition, in the GUI screen 400 shown in FIG. 5, an OCTA image, which is a medical image acquired by the image acquisition unit 251, is displayed in the image display area 410 in order to improve diagnostic efficiency during observation by comparing it with the contrast effect image displayed in the image display area 420.

[0046] FIG. 6 is a flowchart showing an example of a processing procedure in a control method for the image generating device 20 according to the first embodiment.

[0047] When the process of the flowchart shown in FIG. 6 starts, first, in step S101, the image acquisition unit 251 acquires an OCTA image, which is a medical image, from the imaging device 10, for example.

[0048] Next, in step S102, the output unit 252 generates and outputs a contrast effect image depicting a contrast effect corresponding to a contrast time including contrast times at a plurality of time points based on the OCTA image acquired in step S101. Specifically, in this embodiment, the output unit 252 outputs a contrast effect image that is a pseudo contrast image like an FA examination image in a moving image format depicting a time-dependent change in the contrast effect corresponding to the contrast time.

[0049] Next, in step S103, the display unit 253 displays the OCTA image acquired in step S101 in the image display area 410 of the GUI screen 400 shown in FIG. 5, and displays the contrast effect image of the moving image output in step S102 in the image display area 420.

[0050] When the process of step S103 ends, the process of the flowchart shown in FIG. 6 ends.

[0051] As described above, in the image generating device 20 according to the first embodiment, the image acquiring unit 251 acquires, for example, an OCTA image, which is a medical image, from the imaging device 10. Then, the output unit 252 outputs a contrast effect image (a contrast effect image in a moving image format in which the contrast effect is depicted) depicting a contrast effect corresponding to a contrast time including contrast times at a plurality of time points, based on the OCTA image acquired by the image acquiring unit 251. For example, when the contrast time is composed of contrast times at a plurality of time points over time, a contrast effect image in a moving image format in which a change in the contrast effect over time is depicted is output. According to this configuration, an image depicting a contrast effect corresponding to a contrast time including contrast times at a plurality of points in time can be suitably acquired, which makes it possible to suitably acquire an image similar to an FA examination image depicting a contrast effect corresponding to a contrast time including a contrast time that the operator wishes to observe, thereby making it possible to support the operator in making a diagnosis.

[0052] [Modification 1 of the first embodiment] Next, as a modification of the above-described first embodiment, Modification 1 of the first embodiment will be described.

[0053] FIG. 7 shows a first modified example of the first embodiment, and is a diagram for explaining the period of time (contrast time) during which the FA examination images, which are moving images constituting the teacher data used when the image generation model 2520 learns, are recorded.

[0054] The FA examination images, which are moving images constituting the teacher data used when the image generation model 2520 learns, may be FA examination images of only a part of a period (contrast time) of a predetermined contrast time from time T1 seconds to time T2 seconds, as shown in FIG. 7. It is preferable that the recording periods of all FA examination images, when integrated, include the period (contrast time) of the predetermined contrast time. In addition, when the period (contrast time) of the contrast time to be clinically observed or the period (contrast time) of the contrast time that the operator particularly wants to observe can be specified, it is preferable to include FA examination images including the period (contrast time) of the contrast time in the teacher data group with a focus. That is, it is preferable that the FA examination image group (contrast image group) included in the learning data includes more FA examination images taken at the contrast time including the contrast time that the operator wants to observe than FA examination images taken at the contrast time including other contrast times. This is effective because it improves the image generation accuracy (likelihood of depicting a contrast effect image) of the image generation model 2520 for the period of the contrast time (contrast time). In this case, when calculating the accuracy evaluation and error (loss), only the contrast time corresponding to the playback position of the existing video frame image is targeted.

[0055] FIG. 8 shows a first modified example of the first embodiment, and is a diagram showing an example of the relationship between a contrast effect image, which is a moving image output by the image generation model 2520, and a correct answer image (FA test image), which is a moving image constituting the teacher data.

[0056] For example, when calculating the accuracy evaluation or error (loss) for the contrast effect image and correct image (FA test image) shown in FIG. 8, the calculation is performed for the period from contrast time t seconds to T2 seconds, which is the period (contrast time) of the contrast time of the video frame images present in the correct image.

[0057] In the first modification of the first embodiment, it is assumed that the FA examination image, which is a moving image constituting the teacher data, is not recorded so as to include a period of a predetermined contrast enhancement time (contrast time). According to the first modification of the first embodiment, even in such a case, a pseudo image (contrast effect image) resembling an FA examination image in a moving image format depicting a time-dependent change in contrast effect can be suitably acquired based on an OCTA image. This makes it possible to suitably acquire an image resembling an FA examination image depicting a contrast effect corresponding to the contrast time including the contrast time that the operator wants to observe, thereby making it possible to support the operator in making a diagnosis.

[0058] [Modification 2 of the first embodiment] Next, as a modification of the above-described first embodiment, Modification 2 of the first embodiment will be described.

[0059] In the first embodiment described above, the FA examination images of the teacher data group used when the image generation model 2520 learns may include FA examination images with different sizes of imaging ranges (i.e., angles of view). In this regard, when the difference in size of the imaging range between the OCTA image and the FA examination image is large, anatomical alignment may be difficult. For example, if the imaging ranges of the OCTA image and the FA examination image are almost the same, common parts and blood vessels of the test object (in this embodiment, the test eye) are both depicted, and anatomical alignment is likely to be successful.

[0060] FIG. 9 shows a second modified example of the first embodiment, and is a diagram showing an example of an OCTA image and an FA examination image. FIG. 9 shows a wide-area OCTA image Im10 obtained by capturing a wide area, a wide-area FA examination image Im20 obtained by capturing a wide area, and a narrow-area FA examination image Im30 obtained by capturing a narrow area. When anatomically aligning the wide-area OCTA image Im10 and the narrow-area FA examination image Im30 shown in FIG. 9, the difference in the appearance of the depicted parts and blood vessels is large, coupled with the fact that the imaging devices are different from the start, and anatomical alignment may be difficult. In this case, the result of anatomical alignment can be improved by using the wide-area FA examination image Im20 obtained by capturing a wider area of ​​the same examination subject.

[0061] FIG. 10 illustrates a second modification of the first embodiment and is a flowchart showing an example of a processing procedure for aligning an OCTA image with an FA inspection image.

[0062] When the process of the flowchart shown in Fig. 10 is started, first, in step S201, the image generation model 2520 anatomically aligns the wide-area FA inspection image Im20 and the narrow-area FA inspection image Im30 shown in Fig. 9. At this time, since both images are acquired from the same imaging device 10, anatomical alignment can be performed.

[0063] Next, in step S202, the image generation model 2520 anatomically aligns the wide-area FA test image Im20 and the wide-area OCTA image Im10. At this time, since both images are acquired by photographing a wide area, anatomical alignment can be performed.

[0064] Next, in step S203, the image generation model 2520 relatively aligns the wide-area OCTA image Im10 and the narrow-area FA test image Im30. Specifically, the image generation model 2520 performs the alignment in step S203 by combining deformation information obtained when performing the anatomical alignment in step S201 and deformation information obtained when performing the anatomical alignment in step S202.

[0065] According to the second modification of the first embodiment, even if the difference in the size of the imaging range between the OCTA image and the FA examination image is large, better anatomical alignment can be performed. As a result, the depiction state of the contrast effect in the contrast effect image output by the image generation model 2520 can be made closer to the real FA examination image. That is, based on the OCTA image, a pseudo image (contrast effect image) resembling an FA examination image in a moving image format depicting the change in contrast effect over time can be suitably acquired. This makes it possible to suitably acquire an image resembling an FA examination image depicting the contrast effect corresponding to the contrast time including the contrast time that the operator wants to observe, thereby making it possible to support the operator in making a decision in diagnosis.

[0066] [Third Modification of the First Embodiment] Next, as a modification of the above-described first embodiment, Modification 3 of the first embodiment will be described.

[0067] Regarding the dataset for training the image generation model 2520 in the first embodiment described above, the OCTA images (group of medical images) constituting the dataset may be replaced with other types of images recording the state of the fundus of the test eye.

[0068] For example, three-dimensional motion contrast data, two-dimensional OCT images, and three-dimensional OCT images acquired by an OCT device may be applied as other types of images.Furthermore, for example, fundus images acquired by a fundus camera and SLO (Scanning Laser Ophthalmoscope) images acquired by a scanning laser ophthalmoscope may be applied as other types of images.

[0069] Also, for example, the image may be a mixture of an OCTA image and other types of images described above. Specifically, for example, a fundus image, which is a three-channel RGB color image, and an OCTA image, which is a one-channel grayscale image, may be mixed on the channel axis to create a four-channel image. At this time, since it is preferable that the anatomical positions of the fundus image and the OCTA image match, an anatomical alignment process is performed. Alternatively, if the photographing device 10 has the functions of both a fundus camera and an OCT device, the anatomical positions of the acquired fundus image and the OCTA image may match in advance, so that it is not necessary to perform anatomical alignment again.

[0070] In addition, when the OCTA image is replaced with the other type of image described above, the "OCTA image" described in the first embodiment is replaced with the "other type of image" described above. From the above, based on the "other type of image" described above, a pseudo image (contrast effect image) resembling an FA examination image in a moving image format depicting the change in contrast effect over time can be suitably acquired. This makes it possible to suitably acquire an image resembling an FA examination image depicting the contrast effect corresponding to the contrast time including the contrast time that the operator wants to observe, thereby making it possible to support the operator in making a diagnosis.

[0071] Second embodiment Next, a second embodiment will be described. In the following description of the second embodiment, matters common to the first embodiment will be omitted, and only matters different from the first embodiment will be described.

[0072] Fig. 11 is a diagram showing an example of a schematic configuration of an image generation system 1 including an image generation device 20 according to a second embodiment. In Fig. 11, the same components as those shown in Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0073] The image generating device 20 according to the second embodiment shown in FIG. 11 has a configuration in which an imaging condition acquisition unit 254 is added to the processing circuit 250 in comparison with the configuration of the image generating device 20 according to the first embodiment shown in FIG.

[0074] The imaging condition acquisition unit 254 has a function of acquiring imaging conditions including an injection time including an injection time of at least one point in time.

[0075] First, the output unit 252 generates a contrast effect image for extraction of a moving image depicting a contrast effect corresponding to a contrast time including contrast times at a plurality of time points based on a medical image, which is a still image acquired by the image acquisition unit 251, as in the first embodiment. Furthermore, the output unit 252 extracts a moving image frame image corresponding to a contrast time included in the imaging conditions acquired by the imaging condition acquisition unit 254 from a group of moving image frame images constituting the contrast effect image for extraction, and outputs it as a final contrast effect image. Specifically, the contrast effect image in this embodiment is a pseudo contrast image like a FA examination image in a still image format depicting a contrast effect at a specified contrast time, as acquired in an FA examination. Here, for ease of understanding, the imaging condition acquisition unit 254 in this embodiment acquires only information on the contrast time as the imaging conditions.

[0076] Fig. 12 is a diagram showing an example of a GUI screen 400 displayed on the display 230 in the image generating device 20 according to the second embodiment. In Fig. 12, the same components as those shown in Fig. 5 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0077] The GUI screen 400 in the second embodiment shown in FIG. 12 has a configuration similar to that of the GUI screen 400 in the first embodiment shown in FIG. 5, with the main difference being that a contrast injection time designation slider 431 and a contrast injection time designation text box 432 are added.

[0078] The contrast time set as an imaging condition can be specified by the operator operating, for example, a contrast time specification slider 431 or a contrast time specification text box 432 shown in FIG. 12 via the input interface 220. For example, FIG. 12 shows an example in which a time "40 seconds" after a reference time point is specified as the contrast time. Note that the method of specifying the contrast time is not limited to the form described here, and may be replaced by other methods that can achieve the same purpose. Note that, although the GUI screen 400 for the operator to specify the contrast time is described here, a configuration in which a contrast time that is predetermined in the image generating system 1 according to the second embodiment is input may also be used.

[0079] FIG. 13 is a flowchart showing an example of a processing procedure in a control method for the image generating device 20 according to the second embodiment.

[0080] When the process of the flowchart shown in FIG. 13 starts, first, in step S301, the image acquisition unit 251 acquires an OCTA image, which is a medical image, from the imaging device 10, for example.

[0081] Next, in step S302, the imaging condition acquisition unit 254 acquires imaging conditions including an injection time including at least one injection time point. Specifically, in this embodiment, the injection time is acquired as the imaging condition.

[0082] Next, in step S303, the output unit 252 generates and outputs a contrast effect image depicting a contrast effect corresponding to the contrast time based on the OCTA image acquired in S301 and the imaging conditions (contrast time) acquired in S302. Specifically, in this embodiment, the output unit 252 outputs a contrast effect image that is a pseudo contrast image like an FA examination image in a still image format depicting a contrast effect corresponding to the contrast time.

[0083] Next, in step S304, the display unit 253 displays the OCTA image acquired in step S301 in the image display area 410 of the GUI screen 400 shown in FIG.

[0084] When the process of step S304 ends, the process of the flowchart shown in FIG. 13 ends.

[0085] As described above, in the image generating device 20 according to the second embodiment, the image acquiring unit 251 acquires an OCTA image, which is a medical image, from, for example, the imaging device 10. The imaging condition acquiring unit 254 acquires imaging conditions including a contrast time including at least one contrast time point. The output unit 252 outputs a contrast effect image depicting a contrast effect corresponding to the contrast time, based on the OCTA image acquired by the image acquiring unit 251 and the imaging conditions acquired by the imaging condition acquiring unit 254. According to this configuration, it is possible to suitably obtain an image depicting a contrast effect corresponding to a contrast time including a contrast time at a certain point in time (in this embodiment, the contrast time). More specifically, according to the image generating device 20 according to the second embodiment, it is possible to suitably obtain an image like an FA examination image depicting a contrast effect corresponding to the contrast time that the operator wants to observe, thereby making it possible to support the operator in making a diagnosis.

[0086] (Third embodiment) Next, a third embodiment will be described. In the following description of the third embodiment, the description of the matters common to the first and second embodiments will be omitted, and only the matters different from the first and second embodiments will be described.

[0087] The schematic configuration of an image generation system including an image generation device according to the third embodiment is similar to the schematic configuration of the image generation system 1 including an image generation device 20 according to the second embodiment shown in FIG.

[0088] The output unit 252 of the third embodiment outputs a contrast effect image, which is a still image depicting the contrast effect corresponding to the contrast time included in the imaging conditions acquired by the imaging condition acquisition unit 254, based on a medical image, which is a still image acquired by the image acquisition unit 251.

[0089] Fig. 14 is a diagram for explaining the concept of an image generation model 2520 included in the output unit 252 in the image generating device 20 according to the third embodiment. In Fig. 14, the same components as those shown in Fig. 2 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0090] The output unit 252 of the third embodiment has an image generation model 2520 shown in Fig. 14. The image generation model 2520 shown in Fig. 14 has a network model 2521 based on U-Net as an image processing system using deep learning technology. Note that, although U-Net is exemplified in this embodiment, other network models capable of achieving the same purpose may be adopted.

[0091] The image generation model 2520 in Fig. 14 receives an input image St301 corresponding to a still image medical image and an injection time Ti341, and generates a still image contrast effect image depicting a contrast effect corresponding to the injection time Ti341 based on the input image St301. Specifically, the image generation model 2520 in Fig. 14 receives a tensorized version of the input image St301, which is a still image, and a tensorized version of the injection time Ti341, and inputs this to the network model 2521. Then, the image generation model 2520 in Fig. 14 generates a still image from the tensor output by the network model 2521, and outputs the still image as the output image Mo311.

[0092] When U-Net is adopted as the network model 2521, it is necessary to modify U-Net. Specifically, a scalar value T indicating the injection time Ti341 is assigned to at least one tensor space axis among the number of channels, height, and width of at least one tensor generated in the intermediate layer of the network model 2521. Here, the "tensor generated in the intermediate layer" corresponds to tensors Te351 to Te357 in FIG. 14. In FIG. 14, the scalar value T is assigned to all of the tensors Te351 to Te357, but, for example, there may be a form in which the scalar value T is assigned only to the tensor Te351, or a form in which the scalar value T is assigned to the tensors Te355 to Te357, etc.

[0093] Moreover, the scalar value T is a scalar value determined based on the contrast time Ti341, for example, the contrast time Ti341 in milliseconds divided by a constant. As a specific method of assignment, for example, consider a case where the shape of the original tensor before the scalar value T is assigned is "B×C×H×W". Here, B is the mini-batch size, C is the number of channels, H is the height, and W is the width. In the case of this shape, the number of channels is expanded to the shape of "B×(C+1)×H×W", and a process of filling the value of the expanded tensor region with the scalar value T is added, and the structure of the network model 2521 is changed so that the expanded tensor can be processed. Alternatively, when the number of channels is 2 or more, instead of expanding the tensor, the value of the tensor region for any one channel may be filled with the scalar value T. Note that in order to improve the image generation accuracy (likelihood of the output image Mo311) and calculation efficiency of the image generation model 2520, the network model 2521 that normalizes and handles the input and output tensors may be used. For example, a large value such as 40000 representing 40000 milliseconds may be set as the scalar value T indicating the injection time Ti341 for the range of values ​​(e.g., a range of −10.0 to 10.0) of the tensor generated by this network model 2521. In this case, since a model with low image generation accuracy may be trained, the scalar value T may be normalized, for example, by dividing it by the maximum value that can be input to the image generation model 2520 to convert it to a value of 0 to 1.

[0094] The purpose of the operation on the tensor described with reference to Fig. 14 is to input information on the contrast injection time to the image generation model 2520 and have the network model 2521 process the OCTA image, which is the input image St301, and the contrast injection time Ti341. Therefore, the present embodiment is not limited to the method described with reference to Fig. 14. An example of another method will be described with reference to Fig. 15.

[0095] Fig. 15 is a diagram for explaining the concept of an image generation model 2520 included in the output unit 252 in the image generating device 20 according to the third embodiment. In Fig. 15, the same components as those shown in Fig. 2 and Fig. 14 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0096] For example, as another method, as shown in FIG. 15, a method in which the network model 2521 is configured with an unmodified U-Net and a known Decoder network can be adopted. Specifically, first, a scalar value T indicating the contrast enhancement time Ti341 is input to the Decoder network. Then, the upsampled tensor Te361 output from the Decoder network is connected to the tensor of the OCTA image input to the U-Net, and the U-Net outputs the tensor of the contrast effect image. Even with the configuration shown in FIG. 15, the OCTA image, which is the input image St301, and the contrast enhancement time Ti341 can be processed by the network model 2521 to obtain a contrast effect image as an output image Mo311 from the image generation model 2520.

[0097] By the above-mentioned operation on the tensor, the information of the contrast time Ti341 is input to the network model 2521, and the image generation model 2520 can output a contrast effect image, which is a still image depicting a contrast effect corresponding to an arbitrary contrast time. Note that the method of inputting the information of the contrast time Ti341 to the network model 2521 is not limited to the method described in the present embodiment, and other methods that can achieve the same purpose may be used. For example, a method of operating the pixel value of the input image St301 with a value related to the contrast time Ti341, or adding a new image channel to the input image St301 and setting a pixel value related to the contrast time Ti341 may also be applied. Furthermore, a method of adding and inputting an image generated based on the contrast time Ti341 to the network model 2521 may also be applied.

[0098] A data set for learning the image generation model 2520 including the network model 2521 based on the U-Net described here will be described below. The data set is configured as a group of teacher data acquired from a plurality of test objects, with a still image of an OCTA image acquired by photographing the same test object, an FA test image photographed at a certain contrast time, and the contrast time of the FA test image being one set (pair). Here, the test object is the subject's eye in this embodiment. Note that, for one OCTA image, there may be a plurality of FA test image groups (contrast image groups) photographed and acquired over time, and a contrast time group (imaging condition group) corresponding to the FA test image groups.

[0099] FIG. 16 shows the third embodiment, and is a diagram showing an example of a period with and without FA examination images of the left and right eyes constituting the teacher data used when the image generation model 2520 learns. In an FA examination, the left and right eyes are alternately photographed after a contrast agent is injected, so that the time periods in which FA examination images exist may be distributed, for example, as shown in FIG. 16. In a long time period such as time period TF311 among these time periods, a moving image may be photographed as an FA examination image. In this embodiment, when a moving image is acquired, a group of moving image frame images constituting the moving image may be extracted as a group of still images, and a contrast time group corresponding to each moving image frame image may be identified, and the group may be paired with a corresponding OCTA image of the subject's eye and used as the teacher data.

[0100] Fig. 17 is a diagram for explaining the learning of the image generation model 2520 possessed by the output unit 252 in the image generating device 20 according to the third embodiment. In Fig. 17, the same components as those shown in Figs. 14 and 15 are given the same reference numerals, and detailed explanations thereof will be omitted. Hereinafter, learning of the image generation model 2520 using a certain set of teacher data, that is, updating processing of a parameter group constituting a network model 2521 included in the image generation model 2520, will be explained with reference to Fig. 17.

[0101] First, in FIG. 17, an input tensor Te302 obtained by tensorizing an OCTA image constituting the teacher data and a scalar value Se342 indicating the contrast injection time Ti341 constituting the same teacher data are input to the network model 2521. Then, an output tensor Te312 corresponding to a contrast effect image, which is a still image, is output from the network model 2521. Next, the image generation model 2520 calculates a loss Lo332, which is an error between the correct tensor Te322 obtained by tensorizing an FA examination image taken at the contrast injection time Ti341, which is a still image constituting the same teacher data, and the output tensor Te312. Finally, the image generation model 2520 updates a group of parameters constituting the network model 2521 so that the loss Lo332 becomes small. This series of update processes is repeated using a group of teacher data allocated for training among the data set until the network model 2521 has learned sufficiently.

[0102] When an OCTA image is input, the image generation model 2520 trained by the above process can output a still image contrast effect image depicting a plausible contrast effect based on a group of teacher data assigned for training among the data set. In other words, it can output a pseudo image (contrast effect image) like an FA examination image in the form of a still image depicting a contrast effect corresponding to a specified contrast time, as obtained in an FA examination.

[0103] The processing procedure in the control method of the image generating device 20 according to the third embodiment is similar to the flowchart showing the processing procedure in the control method of the image generating device 20 according to the second embodiment shown in Fig. 13. Hereinafter, the processing procedure in the control method of the image generating device 20 according to the third embodiment will be described with reference to the flowchart shown in Fig. 13.

[0104] In the third embodiment, when the process of the flowchart shown in FIG. 13 is started, first, in step S301, the image acquisition unit 251 acquires an OCTA image, which is a medical image, from the imaging device 10, for example.

[0105] Next, in step S302, the imaging condition acquisition unit 254 acquires imaging conditions including an injection time including at least one injection time point. Specifically, in this embodiment, the injection time is acquired as the imaging condition.

[0106] Next, in step S303, the output unit 252 generates and outputs a contrast effect image depicting a contrast effect corresponding to the contrast time based on the OCTA image acquired in S301 and the imaging conditions (contrast time) acquired in S302. Specifically, in this embodiment, the output unit 252 outputs a contrast effect image that is a pseudo contrast image like an FA examination image in a still image format depicting a contrast effect corresponding to the contrast time.

[0107] Next, in step S304, the display unit 253 displays the OCTA image acquired in step S301 in the image display area 410 of the GUI screen 400 shown in FIG.

[0108] When the process of step S304 ends, the process of the flowchart shown in FIG. 13 ends.

[0109] As described above, in the image generating device 20 according to the third embodiment, the image acquiring unit 251 acquires an OCTA image, which is a medical image, from, for example, the imaging device 10. The imaging condition acquiring unit 254 acquires imaging conditions including a contrast time including at least one contrast time point. The output unit 252 outputs a contrast effect image depicting a contrast effect corresponding to the contrast time, based on the OCTA image acquired by the image acquiring unit 251 and the imaging conditions acquired by the imaging condition acquiring unit 254. According to this configuration, it is possible to suitably obtain an image depicting a contrast effect corresponding to a contrast time including a contrast time at a certain point in time (in this embodiment, the contrast time). More specifically, according to the image generating device 20 according to the third embodiment, it is possible to suitably obtain an image like an FA examination image depicting a contrast effect corresponding to the contrast time that the operator wants to observe, thereby making it possible to support the operator in making a diagnosis.

[0110] Moreover, the image generating device 20 according to the third embodiment consumes less time and calculation costs for the output unit 252 because it does not output moving images, and is more useful in an environment with performance limitations, compared to the image generating device 20 according to the first embodiment. Furthermore, training of the image generating model 2520 of the output unit 252 does not require teacher data that is a moving image that satisfies a period of a predetermined contrast time (contrast time), that is, it does not matter if the FA examination images included in the teacher data have different contrast times. This makes it easier to collect teacher data, and therefore makes it possible to increase the possibility of depicting a contrast effect that is more similar to a real contrast image.

[0111] [Modification 1 of the third embodiment] Next, as a modification of the above-mentioned third embodiment, a first modification of the second embodiment will be described.

[0112] Fig. 18 is a flowchart showing an example of a processing procedure in a control method of the image generating device 20 according to the first modified example of the third embodiment. According to the processing of the flowchart shown in Fig. 18, a contrast effect image in a moving image format can also be output.

[0113] 18 starts, first, in step S401, the image acquisition unit 251 acquires a medical image from, for example, the imaging device 10. In the first modification of the third embodiment, an OCTA image is acquired as the medical image.

[0114] Next, in step S402, the imaging condition acquisition unit 254 acquires a group of imaging conditions in which the contrast time is changed so as to correspond to a period of a predetermined contrast time (contrast time). For example, when it is desired to observe the contrast effect at intervals of 1 second with the period of a predetermined contrast time (contrast time) being "from 0 seconds to 200 seconds", a group of 201 imaging conditions (contrast time) is acquired, which is generated by changing the contrast time from 1 second, 2 seconds, ..., to 200 seconds.

[0115] Next, in step S403, the output unit 252 outputs a group of contrast effect images corresponding to each of the imaging condition groups (contrast enhancement time groups) acquired in step S402 based on the OCTA image acquired in step S401. Specifically, in step S403, a group of contrast effect images, which are pseudo contrast images like FA examination images in a still image format depicting the contrast effects corresponding to each of the contrast enhancement time groups, are output.

[0116] Next, in step S404, the output unit 252 outputs the contrast effect images, which are moving images, by using the contrast effect image group output in step S403 as moving image frame images.

[0117] Next, in step S405, the display unit 253 displays the OCTA image acquired in step S401 in the image display area 410 of the GUI screen 400 shown in FIG. 5, and displays the contrast effect image of the moving image output in step S404 in the image display area 420.

[0118] When the process of step S405 ends, the process of the flowchart shown in FIG. 18 ends.

[0119] According to the first modification of the third embodiment, a pseudo FA examination image (contrast effect image) in a moving image format depicting a time-dependent change in contrast effect based on an OCTA image can be suitably acquired. This makes it possible to suitably acquire an FA examination image-like image depicting a contrast effect corresponding to a contrast time including a contrast time that the operator wishes to observe, thereby making it possible to support the operator in making a diagnosis.

[0120] [Modification 2 of the third embodiment] Next, as a modification of the above-mentioned third embodiment, a second modification of the third embodiment will be described.

[0121] Regarding the data set for training the image generation model 2520 in the third embodiment described above, the FA examination images constituting the data set may be replaced with other types of images that show the state of the contrast effect in the subject of examination.

[0122] For example, other types of images may be applied, such as a region division image illustrating the contrast agent leakage range identified from an FA examination image acquired at a certain contrast time, a contour image of the leakage range, or an image in which an FA examination image is colored using a color lookup table.

[0123] According to the second modification of the third embodiment, the above-mentioned other types of images can be suitably acquired as contrast effect images in which the contrast effect corresponding to the enhancement time is depicted based on the OCTA image. This makes it possible to suitably acquire an image showing the state of the contrast effect corresponding to the enhancement time that the operator wants to observe, and to support the operator in making a diagnosis.

[0124] [Modification 3 of the third embodiment] Next, as a modification of the above-mentioned third embodiment, Modification 3 of the third embodiment will be described.

[0125] Regarding the data set for learning the image generation model 2520 in the third embodiment, the FA test images constituting the data set may be interpolated FA test images generated by interpolating a group of FA test images acquired by photographing the same test object over time. More specifically, as shown in FIG. 16, in an FA test, there is a "period without FA test images" during which no FA test images are acquired. By generating and adopting an image equivalent to the FA test image during this "period without FA test images" by an interpolation process, the image generation accuracy of the image generation model 2520 (likelihood of depicting a contrast effect image) is improved.

[0126] FIG. 19 illustrates a third modification of the third embodiment, and is a flowchart showing an example of a processing procedure in generating an interpolated image.

[0127] When the process of the flowchart shown in FIG. 19 is started, first, in step S501, the image generation model 2520 identifies an interpolatable "period without FA examination images". This interpolatable "period without FA examination images" is a period in which FA examination images exist immediately before and after the "period without FA examination images". FIG. 20 shows a third modified example of the third embodiment, and is a diagram showing an example of a period with and without FA examination images constituting teacher data used when the image generation model 2520 learns. In FIG. 20, the time zone TF3302 (from contrast injection time T1 seconds to T2 seconds) is the interpolatable "period without FA examination images" in step S501.

[0128] Now, let us return to the explanation of FIG. When the process of step S501 ends, the process proceeds to step S502. When proceeding to step S502, the image generation model 2520 identifies the immediately preceding FA inspection image and the immediately succeeding FA inspection image in the "period without an FA inspection image" that can be interpolated and that was identified in step S501. In the example shown in Fig. 20, the immediately preceding FA inspection image Im3312 and the immediately succeeding FA inspection image Im3313 are identified in step S502.

[0129] Next, in step S503, the image generation model 2520 specifies an effective pixel area common to the immediately preceding FA inspection image and the immediately succeeding FA inspection image specified in step S502. Here, the effective pixel area is a pixel area in which the contrast effect is depicted. FIG. 21 shows a modified example 3 of the third embodiment, and is a diagram for explaining an effective pixel area Re3332 common to the immediately preceding FA inspection image Im3312 and the immediately succeeding FA inspection image Im3313 shown in FIG. 20. In FIG. 21, for example, the masked area existing around the immediately preceding FA inspection image Im3312 is not an effective pixel area because the contrast effect is not depicted, and the non-masked area existing in the center is an effective pixel area Re3322 because the contrast effect is depicted. In the same manner, the effective pixel area Re3323 in the immediately succeeding FA inspection image Im3313 can be specified. In the example shown in FIG. 21, the overlapping area between the effective pixel area Re3322 of the immediately preceding FA inspection image Im3312 and the effective pixel area Re3323 of the immediately following FA inspection image Im3313 is the common effective pixel area Re3332 identified in step S503.

[0130] Now, let us return to the explanation of FIG. When the process of step S503 ends, the process proceeds to step S504. When proceeding to step S504, the image generation model 2520 generates an interpolated image. Specifically, the image generation model 2520 generates an interpolated image using a pixel value group of a common effective pixel area Re3332 in the immediately preceding FA examination image Im3312 and a pixel value group of a common effective pixel area Re3332 in the immediately following FA examination image Im3313. In the example shown in Fig. 20, the interpolated image is generated by linearly interpolating the FA examination image in the "period without FA examination image (time zone TF3302)" from the contrast injection time T1 second to T2 second.

[0131] Specifically, in step S504 in FIG. 19, the following process is carried out to generate an interpolated image. The pixel value at pixel coordinates (x, y) of the previous FA inspection image Im3312 is A ij Then, the pixel value at the pixel coordinate (x, y) of the immediately following FA inspection image Im3313 is Bij In this case, the pixel value I at pixel coordinates (x, y) of the interpolated image at time t seconds is ij is given by the following equation (1). I ij = (1-α)×A ij + α×B ij (1) However, in equation (1), α = t ÷ (T2-T1) Note that, for the area other than the common effective pixel area Re3332, pixel values ​​that are always treated as a mask area, such as pixel values ​​of 0, are applied.

[0132] When the process of step S504 ends, the process of the flowchart shown in Fig. 19 ends. By the generation process of the interpolated image in the flowchart shown in Fig. 19, for example, for the time period TF3302 which is a "period without FA inspection images" in Fig. 20, it becomes possible to generate an interpolated image every second and add it to a data set.

[0133] Further application examples will be described with reference to FIGS. 22 and 23. FIG. FIG. 22 shows a third modified example of the third embodiment, and is a diagram showing an example of a period with and without an FA inspection image constituting the teacher data used when the image generation model 2520 learns. FIG. 23 shows a third modified example of the third embodiment, and is a diagram for explaining an effective pixel area Re3331 in the case where the immediately following FA inspection image Im3311 shown in FIG. 22 is the FA inspection image captured first in the FA inspection. In FIG. 22, consider a case where the FA inspection image Im3311 immediately following the "period without an FA inspection image" such as the time period TF3301 is the FA inspection image captured first in the FA inspection. In this case, the FA inspection image at the time when the contrast is injected at 0 seconds in FIG. 22 may be set as an FA inspection image Im3310 which is a completely dark image (for example, an image filled with the same pixel value as the mask area) and in which the entire image is an effective pixel area, as shown in FIG. 23. Specifically, the FA inspection image Im3310 shown in FIG. 23 may be set as the FA inspection image immediately before the virtual "period without an FA inspection image". In the example shown in FIG. 23, a common effective pixel region Re3331 between a virtual immediately preceding FA inspection image Im3310 and a virtual immediately succeeding FA inspection image Im3311 is the same region as an effective pixel region Re3321 of the immediately succeeding FA inspection image Im3311.

[0134] According to the third modification of the third embodiment, the training data of the data set is augmented by interpolating the FA examination images in the "period without FA examination images", which is effective in improving the image generation accuracy (likelihood of depicting the contrast effect image) of the image generation model 2520. This makes it possible to suitably obtain an image depicting the contrast effect corresponding to the contrast time that the operator wants to observe, and to support the operator in making a decision in diagnosis.

[0135] (Fourth embodiment) Next, a fourth embodiment will be described. In the following description of the fourth embodiment, matters common to the first to third embodiments will be omitted, and only matters different from the first to third embodiments will be described.

[0136] The schematic configuration of an image generation system including an image generation device according to the fourth embodiment is similar to the schematic configuration of the image generation system 1 including an image generation device 20 according to the first embodiment shown in FIG.

[0137] The output unit 252 of the fourth embodiment outputs a group of contrast effect images of still images depicting contrast effects corresponding to contrast times including a plurality of predetermined contrast time groups, based on medical images which are still images acquired by the image acquisition unit 251.

[0138] Fig. 24 is a diagram for explaining the concept of an image generation model 2520 included in the output unit 252 in the image generating device 20 according to the fourth embodiment. In Fig. 24, the same components as those shown in Fig. 2 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0139] The output unit 252 of the fourth embodiment has an image generation model 2520 shown in Fig. 24. The image generation model 2520 shown in Fig. 24 is an image generation model 2520 including an image processing system using a deep learning technique.

[0140] The image generation model 2520 shown in Fig. 24 inputs an input image St401 corresponding to a still image medical image. Then, based on the input image St401, the image generation model 2520 shown in Fig. 24 outputs output images Mo411a to Mo411c as a contrast effect image group of still images in which contrast effects corresponding to a plurality of predetermined enhancement time groups are depicted.

[0141] The image generation model 2520 shown in FIG. 24 includes a network model 2521 based on U-Net as an image processing system using deep learning technology, and outputs a group of contrast effect images depicting contrast effects at N predetermined contrast time groups. Here, the N predetermined contrast time groups are, for example, a set of contrast times such as times "30 seconds, 60 seconds, and 200 seconds" after a reference time point. Here, it is preferable that the predetermined contrast time is a clinically useful contrast time. For example, it may be selected from contrast times such as a time "before 60 seconds (early contrast)" after a reference time point, a time "60 seconds to 200 seconds (mid contrast)" after a reference time point, and a time "after 200 seconds (late contrast)" that are considered important in FA examinations.

[0142] The image generation model 2520 shown in Fig. 24 converts the input image St401, which is a still image, into a tensor and inputs it to the network model (2521). Then, the image generation model 2520 shown in Fig. 24 converts the tensor output by the network model (2521) into a still image and outputs it as output images Mo411a to Mo411c. In this case, if U-Net is adopted as the network model (2521), it is necessary to modify U-Net.

[0143] A specific example will be described below. When the input image St401, which is a still image, is converted into a tensor, the shape of the tensor is the same as that of the "C in ×H in ×W in In the network model (2521) modified from U-Net, the number of elements that make up the input tensor is increased, and shape deformation is performed up to the final layer to obtain "N × C out ×H out ×W out The network model (2521) outputs a tensor with the shape of N "C out ×H out ×W out". Then, the divided tensor group is turned into still images, and output images Mo411a to Mo411c are output from the image generation model 2520 as a contrast effect image group. Note that the shape of the tensor is not limited to the shape shown in the description of this embodiment, and may be other shapes that can achieve the same purpose. Also, although U-Net is exemplified in this embodiment, other network models that can achieve the same purpose may be adopted.

[0144] A data set for training the image generation model 2520 shown in Fig. 24, including a network model (2521) based on U-Net, will be described below. The data set is configured as a group of teacher data acquired from a plurality of test subjects, with a pair of teacher data consisting of an OCTA image, which is a still image obtained by photographing the same test subject, and an FA test image (group) obtained at one or more injection times among N predetermined injection time groups. Here, the test subject is the subject's eye in this embodiment.

[0145] For ease of understanding, the N predetermined contrast injection time groups will be described below as three time points "30 seconds, 60 seconds, and 200 seconds" after a reference time point. Fig. 25 shows the fourth embodiment and is a diagram for explaining the presence or absence of FA examination images constituting training data used when an image generation model 2520 learns. As described with reference to Fig. 16 in the third embodiment, there may be a time period during which imaging cannot be performed in an FA examination, and therefore, for example, the collection status of FA examination images constituting training data may be as shown in Fig. 25.

[0146] 26 and 27 are diagrams for explaining the learning of the image generation model 2520 possessed by the output unit 252 in the image generating device 20 according to the fourth embodiment. Here, in this embodiment, the image generation model 2520 possessed by the output unit 252 has a network model 2521 shown in Figs. 26 and 27. Hereinafter, learning of the image generation model 2520 using a certain set of teacher data, that is, an update process of a parameter group constituting the network model 2521 included in the image generation model 2520, will be explained with reference to Figs. 26 and 27.

[0147] First, in FIG. 26, an input tensor Te402 obtained by tensorizing an OCTA image constituting the teacher data is input to a network model 2521. Then, output tensors Te412a to Te412c corresponding to three contrast effect images that are still images are output from the network model 2521. Next, the image generation model 2520 calculates a loss group by excluding the missing contrast time in the FA examination image group that is the still image constituting the same teacher data, and the average of the loss group is set as the final loss. For example, if the FA examination image constituting the teacher data is only the one with the contrast time of 60 seconds, the process shown in FIG. 26 is performed. That is, in this case, as shown in FIG. 26, a loss Lo432b that is an error between a correct tensor Te422b obtained by tensorizing an FA examination image with the contrast time of 60 seconds and the corresponding output tensor Te412b is calculated and set as the final loss. As another pattern, when the FA examination images constituting the teacher data are only two images, one at an injection time of 30 seconds and the other at an injection time of 200 seconds, the process shown in FIG. 27 is performed. That is, in this case, as shown in FIG. 27, the losses Lo432a and Lo432c for the injection times of 30 seconds and 60 seconds are calculated in the same manner, and the average value of these is set as the final loss, which is (Lo432a+Lo432c) / 2. Finally, the image generation model 2520 updates the parameter group constituting the network model 2521 so as to reduce the final loss. This series of update processes is repeated using the teacher data group allocated for training from the data set until the network model 2521 has learned sufficiently.

[0148] When an OCTA image is input, the image generation model 2520 trained by the above process can output a contrast effect image group of a plurality of still images depicting a plausible contrast effect based on a training data group allocated for training among the data set. Specifically, it can output a contrast effect image group of three still images depicting a plausible contrast effect corresponding to contrast times of 30 seconds, 60 seconds, and 200 seconds. In other words, it can output a pseudo contrast image group (contrast effect image group) like an FA examination image in the form of a still image depicting the contrast effect at three contrast times as obtained in an FA examination.

[0149] FIG. 28 is a diagram showing an example of a GUI screen 400 displayed on the display 230 in the image generating device 20 according to the fourth embodiment.

[0150] The display unit 253 performs a process of displaying a GUI screen 400 as shown in Fig. 28 on the display 230. Specifically, the display unit 253 performs a process of displaying a medical image (in this embodiment, an OCTA image) acquired by the image acquisition unit 251 in an image display area 410 of the GUI screen 400 shown in Fig. 28. The display unit 253 also performs a process of displaying three contrast effect images output from the output unit 252 in image display areas 420a to 420c of the GUI screen 400 shown in Fig. 28. In this embodiment, a contrast effect image corresponding to an enhancement time of 30 seconds is displayed in the image display area 420a, a contrast effect image corresponding to an enhancement time of 60 seconds is displayed in the image display area 420b, and a contrast effect image corresponding to an enhancement time of 200 seconds is displayed in the image display area 420c. This allows the operator to view the image display areas 420a to 420c of the GUI screen 400 and observe the contrast effect images corresponding to the respective enhancement times.

[0151] FIG. 29 is a flowchart showing an example of a processing procedure in a control method for the image generating device 20 according to the fourth embodiment.

[0152] When the process of the flowchart shown in FIG. 29 is started, first, in step S601, the image acquisition unit 251 acquires an OCTA image, which is a medical image, from the imaging device 10, for example.

[0153] Next, in step S602, the output unit 252 generates and outputs a group of contrast effect images depicting contrast effects corresponding to contrast times including a plurality of predetermined contrast enhancement time groups, based on the OCTA images acquired in step S601. Specifically, in this embodiment, the output unit 252 outputs a group of contrast effect images that are pseudo contrast images resembling FA examination images in a still image format, depicting contrast effects corresponding to a plurality of predetermined contrast enhancement time groups.

[0154] Next, in step S603, the display unit 253 displays the OCTA image acquired in step S601 in the image display area 410 of the GUI screen 400 shown in Fig. 28, and displays the contrast effect images output in step S602 in the image display areas 420a to 420c. That is, like the GUI screen 400 shown in Fig. 28, the OCTA image acquired in step S601 and the contrast effect images output in step S602 are displayed side by side.

[0155] When the process of step S603 ends, the process of the flowchart shown in FIG. 29 ends.

[0156] As described above, in the image generating device 20 according to the fourth embodiment, the image acquiring unit 251 acquires, for example, OCTA images, which are medical images, from the imaging device 10. Then, based on the OCTA images acquired by the image acquiring unit 251, the output unit 252 outputs a group of contrast effect images (a group of pseudo contrast images like FA examination images in the form of still images) depicting contrast effects corresponding to a plurality of contrast time groups. According to this configuration, it is possible to suitably obtain a group of images depicting contrast effects corresponding to contrast times including contrast times at a plurality of time points. As a result, it is possible to suitably obtain a group of images similar to FA examination images depicting contrast effects corresponding to a group of contrast times that the operator wishes to observe, and it is possible to support the operator in making a diagnosis. In addition, according to the image generating device 20 of the fourth embodiment, it is possible to observe a group of contrast effect images of a group of contrast times useful for diagnosis at once, compared to a case where contrast effect images in a moving image format are output, and therefore time efficiency is good. In addition, according to the image generating device 20 of the fourth embodiment, it is only necessary to collect images related to the group of contrast times that the operator wishes to observe as teacher data, so that the load when creating a data set is low.

[0157] [Modification of the fourth embodiment] Next, a modification of the above-mentioned fourth embodiment will be described.

[0158] The output unit 252 in the fourth embodiment may include a plurality of image generation model groups, and each image generation model 2520 may output a pseudo contrast effect image resembling a still image-format FA examination image depicting a contrast effect corresponding to one of a plurality of contrast time groups. That is, in a modification of the fourth embodiment, each image generation model 2520 in the plurality of image generation model groups inputs a single OCTA image and outputs a contrast effect image corresponding to a single contrast time.

[0159] Fifth embodiment Next, a fifth embodiment will be described. In the following description of the fifth embodiment, matters common to the first to fourth embodiments will be omitted, and only matters different from the first to fourth embodiments will be described.

[0160] The schematic configuration of an image generation system including an image generation device according to the fifth embodiment is similar to the schematic configuration of the image generation system 1 including an image generation device 20 according to the second embodiment shown in FIG.

[0161] In the fifth embodiment, the imaging conditions acquired by the imaging condition acquisition unit 254 include other conditions in addition to the contrast time including the contrast time, and can affect the contrast effect image output by the output unit 252 in accordance with the other conditions included in the imaging conditions. The other conditions included in the imaging conditions include one or more pieces of information related to the FA examination, such as the presence or absence of individual image processing (optional image quality improvement processing, etc.) of the FA examination image, the photographing angle of view of the FA examination image, subject information (gender, age, photographed part, presence or absence of treatment, etc.), and the model of the FA examination device.

[0162] The imaging condition acquisition unit 254 of the fifth embodiment acquires imaging conditions including other conditions including one or more of the above-mentioned information related to the FA examination in addition to the contrast time including the contrast time at least one time point. That is, the imaging condition acquisition unit 254 of the fifth embodiment acquires imaging conditions including the above-mentioned contrast time and information other than the contrast time different from the contrast time. The output unit 252 of the fifth embodiment outputs a contrast effect image which is a still image depicting a contrast effect based on a medical image which is a still image acquired by the image acquisition unit 251 and imaging conditions acquired by the imaging condition acquisition unit 254. At this time, the medical image, the contrast time which is an imaging condition, and information other than the contrast time are input to the image generation model 2520 of the output unit 252.

[0163] Fig. 30 is a diagram for explaining the concept of an image generation model 2520 included in the output unit 252 in the image generating device 20 according to the fifth embodiment. In Fig. 30, the same components as those shown in Figs. 2, 14, and 15 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0164] The output unit 252 of the fifth embodiment has an image generation model 2520 shown in Fig. 30. The image generation model 2520 shown in Fig. 30 has a network model 2521 based on U-Net as an image processing system using deep learning technology. Note that, although U-Net is exemplified in this embodiment, other network models capable of achieving a similar purpose may be adopted.

[0165] Fig. 31 is a diagram for explaining learning of an image generation model 2520 included in an output unit 252 in an image generating device 20 according to the fifth embodiment. In Fig. 31, the same components as those shown in Fig. 30 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0166] The image generation model 2520 shown in Fig. 30 receives an input image St501 corresponding to a still medical image and imaging conditions Co541, and generates a contrast effect image, which is a still image in which a contrast effect is depicted based on the input image St501. Specifically, the image generation model 2520 in Fig. 30 inputs an input tensor Te502 in Fig. 31 obtained by tensorizing the input image St501 in Fig. 30, and an input (Sc542) obtained by tensorizing the imaging conditions Co541 in Fig. 30, to the network model 2521. The image generation model 2520 in Fig. 30 generates a still image from the tensor output by the network model 2521, and outputs an output image Mo511 as a contrast effect image.

[0167] When U-Net is adopted as the network model 2521, it is necessary to modify U-Net. The method of this modification is roughly similar to that of the third embodiment, but there are differences, so the differences from the third embodiment will be described below.

[0168] Specifically, a scalar value group Sc542 indicating the imaging condition Co541 is assigned to at least one tensor space axis of the number of channels, height, and width of at least one tensor generated in the intermediate layer of the network model 2521. Here, the scalar value group Sc542 is a set of scalar values ​​determined based on a group of information related to the FA examination constituting the imaging condition Co541. For example, information expressed by a continuous value such as contrast time and age is made into a scalar value divided by a constant, for example, in the same manner as in the third embodiment. Furthermore, information that can be expressed by a Boolean value such as the presence or absence of individual image processing and the presence or absence of treatment is made into a scalar value with, for example, False being 0 and True being 1. Furthermore, information that can be expressed as a category such as gender, imaging site, imaging angle of view (30 degrees, 55 degrees, etc.), and model of FA examination device is made into a scalar value obtained by dividing the corresponding category value by a constant, for example. As a specific example, if the category value for gender information is 0 for male and 2 for unknown or others, each category value may be divided by a constant, which is the maximum category value, 2, to obtain scalar values ​​of 0, 0.5, and 1, respectively. Note that the purpose is to input a group of information related to the FA examination into the network model 2521, and the conversion to scalar values ​​may not necessarily be performed in the manner described here. For example, in the example of converting age information into a scalar value described above, age is treated as a continuous value, but it may be treated as a discrete value and a category. Alternatively, it may be treated as an age group and converted to a scalar value based on a category value corresponding to "20s", "30s", "40s", etc. As a specific method of assignment, for example, consider a case where the shape of the original tensor before the scalar value group Sc542 is assigned is "B×C×H×W", and the number of pieces of information included in the imaging condition Co541 (i.e., the number of scalar value groups Sc542) is M. In this case, the number of channels is expanded to a shape of "B×(C+M)×H×W". Then, a process is added in which each of the scalar values ​​constituting the scalar value group Sc542 is filled in for each channel region of the expanded tensor region, and the structure of the network model 2521 is modified so that the expanded tensor can be processed.Alternatively, when the number of channels is M+1 or more, instead of expanding the tensor, values ​​of a tensor region for any M channels may be filled with each of the scalar values ​​constituting the scalar value group Sc542. Note that, since the purpose is to input a group of information related to the FA inspection to the network model 2521, the scalar value group Sc542 does not necessarily have to be input to the network model 2521 in the manner described here. For example, for a group of tensors generated in an intermediate layer of the network model 2521, each of the scalar values ​​constituting the scalar value group Sc542 may be assigned to a different tensor.

[0169] A data set for learning the image generation model 2520 including the network model 2521 based on the U-Net described here will be described below. The data set is configured as a group of teacher data acquired from a plurality of test objects, with a pair of teacher data consisting of an OCTA image, which is a still image acquired by photographing the same test object, an FA test image, and an imaging condition including at least the contrast time of the FA test image. Here, the test object is the subject's eye in this embodiment. Learning of the image generation model 2520 using a certain pair of teacher data, that is, an update process of a parameter group constituting the network model 2521 included in the image generation model 2520 will be described below with reference to FIG. 31.

[0170] First, in FIG. 31, an input tensor Te502 obtained by tensorizing an OCTA image constituting the teacher data and a scalar value group Sc542 indicating an imaging condition Co541 constituting the same teacher data are input to a network model 2521. Then, an output tensor Te512 corresponding to a contrast effect image, which is a still image, is output from the network model 2521. Next, the image generation model 2520 calculates a loss Lo532, which is an error between a correct answer tensor Te522 obtained by tensorizing an FA inspection image, which is a still image constituting the same teacher data, and the output tensor Te512. Finally, the image generation model 2520 updates a parameter group constituting the network model 2521 so that the loss Lo532 becomes small. This series of update processes is repeated using a teacher data group allocated for training among the data set until the network model 2521 has learned sufficiently.

[0171] When an OCTA image is input, the image generation model 2520 trained by the above process can output a still image contrast effect image depicting a plausible contrast effect based on a group of teacher data assigned for training among the data set. In other words, it can output a pseudo contrast image (contrast effect image) like an FA examination image in the form of a still image depicting a contrast effect corresponding to a specified contrast time, such as that obtained in an FA examination.

[0172] The processing procedure in the control method of the image generating device 20 according to the fifth embodiment is similar to the flowchart showing the processing procedure in the control method of the image generating device 20 according to the second embodiment shown in Fig. 13. Hereinafter, the processing procedure in the control method of the image generating device 20 according to the fifth embodiment will be described with reference to the flowchart shown in Fig. 13.

[0173] In the fifth embodiment, when the process of the flowchart shown in FIG. 13 is started, first, in step S301, the image acquisition unit 251 acquires an OCTA image, which is a medical image, from the imaging device 10, for example.

[0174] Next, in step S302, the imaging condition acquisition unit 254 acquires imaging conditions including an injection time including at least one injection time point and information other than the injection time.

[0175] Next, in step S303, the output unit 252 generates and outputs a contrast effect image in which the contrast effect is depicted, based on the OCTA image acquired in S301 and the imaging conditions acquired in S302. Specifically, in this embodiment, the output unit 252 outputs the contrast effect image, which is a pseudo contrast image in the form of a still image in which the contrast effect is depicted, similar to an FA examination image.

[0176] Next, in step S304, the display unit 253 displays the OCTA image acquired in step S301 in the image display area 410 of the GUI screen 400 shown in FIG.

[0177] When the process of step S304 ends, the process of the flowchart shown in FIG. 13 ends.

[0178] As described above, in the image generating device 20 according to the fifth embodiment, the image acquiring unit 251 acquires an OCTA image, which is a medical image, from, for example, the imaging device 10. The imaging condition acquiring unit 254 acquires an enhancement time including an enhancement time of at least one point in time and imaging conditions including information other than the enhancement time. Then, the output unit 252 outputs a contrast effect image (a pseudo image like an FA examination image in a still image format) in which a contrast effect is depicted based on the OCTA image acquired by the image acquiring unit 251 and the imaging conditions acquired by the imaging condition acquiring unit 254. According to this configuration, it is possible to suitably obtain an image similar to an FA examination image depicting a contrast effect corresponding to the enhancement time that the operator wants to observe, and it is possible to support the operator in making a decision in diagnosis. Furthermore, according to the image generating device 20 according to the fifth embodiment, compared to, for example, the third embodiment, it is possible to affect the contrast effect image in accordance with information other than the enhancement time, which is another condition included in the imaging conditions.

[0179] [Modification of the fifth embodiment] Next, a modification of the above-mentioned fifth embodiment will be described.

[0180] The imaging condition acquisition unit 254 in the above-mentioned fifth embodiment may acquire information related to an OCTA examination as imaging conditions in addition to information related to an FA examination, and may also include information related to the OCTA examination in the teacher data. Here, the information related to the OCTA examination includes the model of the OCTA examination device, whether or not individual image processing of the OCTA image is performed, the depth range for generating the OCTA image (superficial layer, deep layer, outer layer, choroidal vascular network, etc.), and the photographing angle of view of the OCTA image. Furthermore, the information related to the OCTA examination includes the resolution of the OCTA image and the scan mode of the OCTA image (Cross, Radial).

[0181] According to the modification of the fifth embodiment, information related to the OCTA examination can also be reflected in the image generation process by the image generating device 20, and a contrast effect image in which the contrast effect is depicted based on more detailed features of the input OCTA image can be obtained. This makes it possible to suitably obtain a contrast effect image in which the contrast effect corresponding to the contrast time including the contrast time that the operator wants to observe is depicted, thereby making it possible to support the operator in making a diagnosis.

[0182] Sixth embodiment Next, a sixth embodiment will be described. In the following description of the sixth embodiment, matters common to the first to fifth embodiments will be omitted, and only matters different from the first to fifth embodiments will be described.

[0183] The schematic configuration of an image generation system including an image generation device according to the sixth embodiment is similar to the schematic configuration of the image generation system 1 including an image generation device 20 according to the second embodiment shown in FIG.

[0184] The imaging condition acquisition unit 254 of the sixth embodiment acquires imaging conditions including information other than the contrast time in addition to the contrast time including at least one contrast time point. In this embodiment, the information other than the contrast time included in the imaging conditions includes one or more pieces of information related to an OCTA examination or an FA examination and that can be interpreted as a category.

[0185] The output unit 252 of the sixth embodiment has a group of image generation models in a plurality of image generation models 2520. Each image generation model in the group of image generation models is constructed corresponding to a type of information that can be interpreted as a category included in the imaging conditions acquired by the imaging condition acquisition unit 254, and has a different quality in terms of depicting the contrast effect. For example, when the imaging conditions include "depth range information (superficial layer, deep layer, outer layer, choroidal vascular network, etc.)" for generating an OCTA image as information related to an OCTA examination, the output unit 252 has a plurality of image generation models for each depth range. Specifically, the group of image generation models is, for example, an "image generation model for superficial layer", an "image generation model for deep layer", an "image generation model for outer layer", an "image generation model for choroidal vascular network", etc.

[0186] The output unit 252 of the sixth embodiment selects an appropriate image generation model from among a plurality of image generation models 2520 based on information other than the contrast time included in the imaging conditions. Then, the output unit 252 of the sixth embodiment uses the selected image generation model to output a contrast effect image based on the medical image acquired by the image acquisition unit 251 and the imaging conditions acquired by the imaging condition acquisition unit 254. Specifically, the output unit 252 of the sixth embodiment selects an appropriate image generation model based on the above-mentioned depth range information included in the imaging conditions, and executes image generation processing of the contrast effect image.

[0187] Also, for example, in a case where the imaging conditions include "presence or absence of individual image processing (optional high-quality processing, etc.)" as information related to the FA examination, the output unit 252 includes two image generation models 2520 corresponding to the presence or absence of individual image processing. Specifically, the image generation model 2520 for "with individual image processing" and the image generation model 2520 for "without individual image processing". In this case, the output unit 252 selects an appropriate image generation model 2520 according to the presence or absence of individual image processing included in the imaging conditions, and executes image generation processing of the contrast effect image. Note that continuous values ​​included in the imaging conditions may also be interpreted as categories. For example, categories may be determined such as "before 100 seconds", "after 100 seconds, before 200 seconds", and "after 200 seconds" according to the value of the contrast time. Note that, as in this example, when new category information can be generated from the contrast time, the information included in the imaging conditions may be only the contrast time.

[0188] The image generation models in the plurality of image generation models 2520 each include a network model 2521 trained with a data set suitable for the imaging conditions used. Specifically, the configuration of the data set in the training of the network model 2521 used when the "depth range information" for OCTA image generation is "superficial layer" is as follows: A group of teacher data obtained from a plurality of test subjects, with an OCTA image generated for the depth range "superficial layer" as a still image obtained by photographing the same test subject, an FA test image, and imaging conditions including at least the contrast time of the FA test image as one teacher data pair. Here, the test subject is the subject's eye in this embodiment.

[0189] Here, it is not necessary to input the imaging conditions that caused the image generation model 2520 to be selected (hereinafter, imaging conditions for selecting an image generation model) to the selected image generation model 2520. Therefore, imaging conditions from which the imaging conditions for selecting an image generation model have been removed are input to the image generation model 2520. That is, the imaging conditions are composed of an injection time including an injection time at least at one time point, and other imaging conditions required by the selected image generation model 2520. For example, it is not necessary to input "depth range information" to the "image generation model for surface layer" that is used when the above-mentioned "depth range information" is "surface layer". Therefore, the imaging conditions input to the "image generation model for surface layer" do not include "depth range information", but include an injection time including an injection time at least at one time point.

[0190] FIG. 32 is a flowchart showing an example of a processing procedure in a control method for the image generating device 20 according to the sixth embodiment.

[0191] When the process of the flowchart shown in FIG. 32 starts, first, in step S701, the image acquisition unit 251 acquires an OCTA image, which is a medical image, from the imaging device 10, for example.

[0192] Next, in step S702, the imaging condition acquisition unit 254 acquires imaging conditions including information other than the contrast time in addition to the contrast time including at least one contrast time point. In this embodiment, the information other than the contrast time included in the imaging conditions includes one or more pieces of information related to an OCTA examination or an FA examination and that can be interpreted as a category.

[0193] Next, in step S703, the output unit 252 selects an appropriate image generation model from among the multiple image generation models 2520 based on information other than the contrast enhancement time included in the imaging conditions (information that can be interpreted as a category).

[0194] Next, in step S704, the output unit 252 generates and outputs a contrast effect image in which a contrast effect is depicted based on the OCTA image acquired in step S701, using the image generation model 2520 selected in step S703. Specifically, in this embodiment, the output unit 252 outputs the contrast effect image, which is a pseudo contrast image like an FA examination image in a still image format.

[0195] Next, in step S705, the display unit 253 displays the OCTA image acquired in step S701 in the image display area 410 of the GUI screen 400 shown in FIG. 12, and displays the contrast effect image output in step S704 in the image display area 420.

[0196] When the process of step S705 ends, the process of the flowchart shown in FIG. 32 ends.

[0197] As described above, in the image generating device 20 according to the sixth embodiment, the image acquiring unit 251 acquires an OCTA image, which is a medical image, from, for example, the imaging device 10. The imaging condition acquiring unit 254 sets imaging conditions including information other than the contrast time in addition to the contrast time including the contrast time at least one time point. The output unit 252 selects an appropriate image generation model from among a plurality of image generation models 2520 based on information other than the contrast time included in the imaging conditions (information that can be interpreted as a category). The output unit 252 uses the selected image generation model 2520 to output a contrast effect image in which the contrast effect is depicted based on the OCTA image acquired by the image acquiring unit 251. According to this configuration, an image resembling an FA examination image in which a contrast effect corresponding to the enhancement time that the operator wants to observe can be suitably acquired, and it becomes possible to support the operator in making a decision in diagnosis. Furthermore, according to the image generating device 20 according to the sixth embodiment, the image generation model 2520 can be switched according to the imaging conditions, and as a result, it is possible to increase the possibility of acquiring a contrast effect image in which a contrast effect more similar to a real contrast image is acquired.

[0198] [Modification 1 of the sixth embodiment] Next, as a modification of the above-mentioned sixth embodiment, a first modification of the sixth embodiment will be described.

[0199] The output unit 252 in the sixth embodiment described above has an image generation model group of a plurality of image generation models 2520, but the following modified examples are applicable. Specifically, instead of selecting an image generation model 2520 based on information that can be interpreted as a category included in the imaging conditions, all of the image generation model groups may output contrast effect images. Note that in modified example 1 of the sixth embodiment, since the image generation model 2520 is not selected, the imaging conditions described above do not need to include "information that can be interpreted as a category."

[0200] The contrast effect image groups output by the image generation model groups can be displayed on the GUI screen 400 or stored in the memory circuitry 240 for use in other processes. Furthermore, the contrast effect image groups output by the image generation model groups can be transferred to another device (not shown) via the NW interface 210 and the network 30 for use.

[0201] [Modification 2 of the sixth embodiment] Next, as a modification of the above-mentioned sixth embodiment, a second modification of the sixth embodiment will be described.

[0202] The output unit 252 in the sixth embodiment described above has an image generation model group in a plurality of image generation models 2520, but the following modified example is applicable. Specifically, instead of having an image generation model group, the output unit 252 may be provided with a single image generation model 2520 capable of outputting a contrast effect image group corresponding to all category values ​​defined in the "information that can be interpreted as a category" included in the imaging conditions.

[0203] For example, a case will be described in which "superficial layer", "deep layer", "outer layer", and "choroidal vascular network" are defined as category values ​​corresponding to the "depth range information" described in the sixth embodiment. In this case, in the second modification of the sixth embodiment, the image generation model 2520 of the output unit 252 can output contrast effect images with depth ranges of "superficial layer", "deep layer", "outer layer", and "choroidal vascular network". In the image generation process of the output unit 252, a group of contrast effect images corresponding to each of "superficial layer", "deep layer", "outer layer", and "choroidal vascular network" is output according to at least the contrast time included in the imaging conditions. In this example, since the selection of the image generation model 2520 is not performed, it is not necessary for the imaging conditions to include "depth range information" which is "information that can be interpreted as a category". Alternatively, the imaging conditions may include "depth range information", and the image generation model 2520 described above may be processed to output only contrast effect images corresponding to the "depth range information".

[0204] Seventh embodiment Next, a seventh embodiment will be described. In the following description of the seventh embodiment, matters common to the first to sixth embodiments will be omitted, and only matters different from the first to sixth embodiments will be described.

[0205] The schematic configuration of an image generation system including an image generation device according to the seventh embodiment is similar to the schematic configuration of the image generation system 1 including an image generation device 20 according to the second embodiment shown in FIG.

[0206] The output unit 252 of the seventh embodiment simply inputs a radiation image, which is a three-dimensional image, as a medical image, and outputs a contrast effect image, which is a pseudo contrast image like a contrast 4DCT image in a moving image format in which a contrast effect is depicted based on the radiation image.

[0207] The image acquisition unit 251 of the seventh embodiment acquires a radiological image, which is a three-dimensional image, as a medical image in which the object of examination is a still image acquired by the imaging device 10. Note that, although the medical image in this embodiment is specifically assumed to be a three-dimensional CT image, it may be a radiological image acquired by another imaging device 10. Also, in this embodiment, it is sufficient that the imaging device 10 can acquire a radiological image. For this reason, for example, the imaging device 10 may be replaced by an image management system that stores and manages the radiological images.

[0208] The output unit 252 of the seventh embodiment has one or more image generation models 2520. The image generation models 2520 are constructed corresponding to the types of information that can be interpreted as categories included in the imaging conditions acquired by the imaging condition acquisition unit 254, and may have different qualities related to the depiction of the contrast effect. For example, in a situation where the imaging conditions include "imaging site information (head, chest, abdomen, etc.)" as information related to a CT examination, the output unit 252 has a group of image generation models in the multiple image generation models 2520 for each imaging site. Specifically, the image generation model group here is, for example, an "image generation model for head", an "image generation model for chest", an "image generation model for abdomen", etc.

[0209] The output unit 252 of the seventh embodiment selects an image generation model 2520 according to imaging site information included in the imaging conditions, executes image generation processing, and outputs a contrast effect image, which is a still image. When a plurality of imaging condition groups are specified, the output unit 252 of the seventh embodiment outputs a contrast effect image group, which is a plurality of still images, corresponding to each imaging condition. Furthermore, the output unit 252 of the seventh embodiment outputs a contrast effect image, which is a moving image, by using the contrast effect image group as a moving image frame image. The contrast effect image, which is a moving image generated here, is a moving image of a three-dimensional image, and is a pseudo contrast image like a contrast 4DCT image. As an example of interpreting the value included in the imaging conditions as a category, for example, a category may be determined according to the value of the subject's age, such as "under 20's", "20's to 30's", and "40's or older".

[0210] Each of the image generation models in the plurality of image generation models 2520 includes a network model 2521 trained with a data set suitable for the imaging conditions used. Specifically, the configuration of the data set in the training of the network model 2521 used when the "imaging part information" is "head" is as follows: Specifically, the training data is a group of training data acquired from a plurality of subjects, with a CT image and a contrast CT image acquired by photographing the "head" of the same examination subject, and imaging conditions including at least the contrast time of the contrast CT image, as one training data set (pair).

[0211] The imaging condition acquisition unit 254 of the seventh embodiment acquires a group of imaging conditions in which the contrast time is changed so as to correspond to a period of a predetermined contrast time (contrast time). For example, when it is desired to observe the contrast effect at intervals of 1 second by setting the period of a predetermined contrast time (contrast time) to "0 seconds to 1000 seconds", a group of 1001 imaging conditions (contrast time) is acquired by changing the contrast time from 1 second, 2 seconds, ..., to 1000 seconds. The imaging conditions may include information that can be interpreted as a category such as "imaging site information".

[0212] The display unit 253 of the seventh embodiment displays the contrast effect image output by the output unit 252 on a GUI screen so that the operator can easily observe it. FIG. 33 is a diagram showing an example of a GUI screen 400 displayed on the display 230 in the image generating device 20 according to the seventh embodiment. In FIG. 33, the same components as those shown in FIG. 5 are denoted by the same reference numerals, and detailed description thereof will be omitted. The display unit 253 performs a process of displaying a medical image (a radiation image in this embodiment) acquired by the image acquiring unit 251 in an image display area 410 of the GUI screen 400 shown in FIG. 33. The display unit 253 also performs a process of displaying a contrast effect image output from the output unit 252 in an image display area 420 of the GUI screen 400 shown in FIG. 33. In particular, when a pseudo contrast effect image resembling a contrast 4DCT image is output from the output unit 252, the display unit 253 may perform the following display. In this case, the display unit 253 may display a tomographic position operation slider 425 and its text box 426 for operating a three-dimensional image, together with GUI screen components (421 to 424) capable of performing playback operations and seek operations in a moving image, as shown in Fig. 33. Furthermore, in the GUI screen 400 shown in Fig. 33, a slider 415 and its text box 416 may also be displayed in the image display area 410.

[0213] The processing procedure in the control method of the image generating device 20 according to the seventh embodiment is similar to the flowchart showing the processing procedure in the control method of the image generating device 20 according to the first modified example of the third embodiment shown in Fig. 18. Hereinafter, the processing procedure in the control method of the image generating device 20 according to the seventh embodiment will be described with reference to the flowchart shown in Fig. 18.

[0214] 18 starts, first, in step S401, the image acquisition unit 251 acquires a medical image from, for example, the imaging device 10. In the seventh embodiment, a three-dimensional CT image is acquired as the medical image.

[0215] Next, in step S402, the imaging condition acquisition unit 254 acquires an imaging condition group (contrast time group) in which the contrast time is changed so as to correspond to a period of a predetermined contrast time (contrast time).

[0216] Next, in step S403, the output unit 252 outputs a group of contrast effect images corresponding to each of the imaging condition groups obtained in step S402 based on the three-dimensional CT image obtained in step S401. Specifically, in step S403, the output unit 252 outputs a group of contrast effect images which are pseudo contrast images resembling contrast CT images in a still image format depicting the contrast effect corresponding to the imaging condition group (contrast time group) obtained in step S402.

[0217] Next, in step S404, the output unit 252 outputs the contrast effect images, which are moving images, by using the contrast effect image group output in step S403 as moving image frame images.

[0218] Next, in step S405, the display unit 253 displays the CT image acquired in step S401 in the image display area 410 of the GUI screen 400 shown in FIG. 33, and displays the contrast effect image, which is the moving image output in step S404, in the image display area 420.

[0219] When the process of step S405 ends, the process of the flowchart shown in FIG. 18 ends.

[0220] According to the seventh embodiment, a contrast CT image in a moving format that allows observation of the change in contrast effect over time, i.e., a pseudo image (contrast effect image) resembling contrast 4DCT can be acquired based on a CT image. This makes it possible to suitably acquire a contrast 4DCT image corresponding to the contrast time that the operator wishes to observe, thereby making it possible to support the operator in making a diagnosis.

[0221] Eighth embodiment Next, an eighth embodiment will be described. In the following description of the eighth embodiment, matters common to the first to seventh embodiments will be omitted, and only matters different from the first to seventh embodiments will be described.

[0222] In the above-described first to seventh embodiments, the image generating device 20 is provided as the generating device. In the eighth embodiment, however, a configuration in which an image generation model generating device is provided will be described.

[0223] Fig. 34 is a diagram showing an example of a schematic configuration of an image generation model generating device 50 according to an eighth embodiment. In Fig. 34, the same components as those shown in Fig. 1 and Fig. 11 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0224] As shown in Fig. 34, the image generation model generating device 50 includes a storage circuit 240 and a processing circuit 250. In Fig. 34, a device including the storage circuit 240 and the processing circuit 250 is shown as the image generation model generating device 50, but it may be configured as the image generating device 50 as in the first to seventh embodiments described above.

[0225] The processing circuitry 250 shown in Fig. 34 comprehensively controls the operation of the image generation model generating device 50 and performs various processes. As shown in Fig. 34, the processing circuitry 250 includes a learning unit 255. In this embodiment, a program for causing the processing circuitry 250 to function as the learning unit 255 is stored in the storage circuitry 240 in the form of a program executable by a computer. For example, the processing circuitry 250 is a processor that realizes the function of the learning unit 255 by reading and executing the program from the storage circuitry 240.

[0226] The learning unit 255 has a function of acquiring a group of teacher data included in a data set for learning an image generation model stored in the storage circuitry 240, and learning the image generation model. The learning unit 255 learns the image generation model using learning data including the medical image group described in the first to seventh embodiments, the contrast image group related to the medical image group, and the imaging condition group related to the contrast image group. Here, the imaging condition group is an imaging condition group including an enhancement time including at least one enhancement time point. Specifically, the learning unit 255 uses the above-mentioned learning data to learn an image generation model that generates a contrast effect image in which a contrast effect corresponding to the enhancement time is depicted based on the medical image when a medical image related to the medical image group and a contrast time are input.

[0227] (Other embodiments) In the above-mentioned first to sixth embodiments, the OCTA image of the surface layer and the FA examination image are described as examples of images in the field of ophthalmology, but the present invention is not limited to this form. For example, the same processing may be performed on the OCTA image of the choroidal vascular network and the indocyanine green fluorescent fundus angiography (IA) examination image. In addition, the same processing may be performed on the Enface image of the choroidal vascular network generated from OCT and the IA examination image, not limited to the OCTA image of the choroidal vascular network.

[0228] In the seventh embodiment, the CT image and the contrast CT image are described as examples of images in the radiology field, but the present invention is not limited to this form. For example, the same processing may be performed on a contrast CT image of a certain phase and a contrast CT image of a different phase. In addition, the same processing may be performed on, for example, an MRI image and a contrast CT image obtained from different types of imaging devices.

[0229] The contrast effect image output by the output unit 252 may be processed into another type of image that allows the user to understand the state of the contrast effect as described in the modified example 2 of the third embodiment and then displayed. That is, the contrast effect image output by the output unit 252 does not have to be displayed as it is.

[0230] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. This program and a computer-readable storage medium storing the program are included in the present invention.

[0231] It should be noted that the above-mentioned embodiments of the present invention are merely illustrative examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0232] The disclosure of the embodiments of the present invention includes the following configurations, methods, and programs. [Configuration 1] an image acquisition unit for acquiring medical images; an output unit that uses an image generation model that receives a medical image and generates a contrast effect image in which a contrast effect is depicted, and outputs a contrast effect image in which a contrast effect corresponding to a contrast time including at least one contrast time point is depicted based on the medical image acquired by the image acquisition unit; An image generating device comprising: [Configuration 2] An imaging condition acquisition unit that acquires imaging conditions including the contrast time, The output unit outputs the contrast effect image based on the medical image and the imaging conditions. 2. The image generating device according to configuration 1. [Configuration 3] The output unit outputs a moving image composed of a plurality of the contrast effect images. 3. The image generating device according to configuration 1 or 2. [Configuration 4] the image generation model has a function of inputting the medical image and the contrast time to generate the contrast effect image; The image generation model is a model trained using training data including a medical image group related to the medical image, a contrast image group related to the medical image group, and an imaging condition group related to the contrast image group. 4. The image generating device according to any one of configurations 1 to 3. [Configuration 5] The image generating device according to configuration 4, wherein the image generation model is a model trained based on semantic regions, which are regions in an image included in the training data and can be divided according to the manner in which the image is depicted or information related to the image. [Configuration 6] 6. The image generating device according to configuration 4 or 5, wherein the learning data includes a group of contrast images acquired over time from the same examination subject as the group of contrast images. [Configuration 7] 7. The image generating device according to any one of configurations 4 to 6, wherein pairs of medical images and contrast images acquired from the same examination subject included in the learning data are anatomically aligned. [Configuration 8] 8. The image generating device according to any one of Configurations 4 to 7, wherein the group of contrast images included in the learning data includes more contrast images taken at contrast times including a contrast time that an operator wants to observe than contrast images taken at contrast times including other contrast times. [Configuration 9] an imaging condition acquisition unit that acquires imaging conditions including the contrast enhancement time and information other than the contrast enhancement time that is different from the contrast enhancement time; 9. The image generating device according to any one of configurations 1 to 8, wherein the image generation model inputs the medical image, the contrast enhancement time, and information other than the contrast enhancement time. [Configuration 10] The output unit has a plurality of the image generation models, The output unit selects an appropriate image generation model from among the plurality of image generation models based on information other than the contrast enhancement time, and outputs the contrast effect image based on the medical image and the imaging conditions using the selected image generation model. 10. The image generating device according to claim 9, [Configuration 11] 9. The image generating device according to any one of configurations 4 to 8, wherein the learning data is augmented based on an effective pixel area in the group of contrast images acquired from the same examination subject included in the learning data. [Configuration 12] 12. The image generating device according to any one of claims 1 to 11, wherein the medical image is an ophthalmoscopy image. [Configuration 13] 12. The image generating device according to any one of configurations 1 to 11, wherein the medical image is a radiation image. [Configuration 14] The image generating device according to any one of configurations 1 to 13, wherein the output unit generates a moving image in which the contrast effect is depicted based on the medical image, and outputs a moving image frame image corresponding to the contrast time in the moving image as the contrast effect image. [Configuration 15] 15. The image generating device according to any one of configurations 1 to 14, further comprising a display unit that displays the contrast effect image on a display device. [Configuration 16] a learning unit that uses learning data including a medical image group, a contrast image group related to the medical image group, and an imaging condition group related to the contrast image group, the imaging condition group including an imaging time including at least one imaging time point, to learn an image generation model that generates a contrast effect image in which a contrast effect corresponding to the contrast time is depicted based on the medical image when a medical image related to the medical image group and the contrast time are input. An image generating device comprising: [Method 1] An image acquisition step for acquiring a medical image; an output step of outputting a contrast effect image depicting a contrast effect corresponding to a contrast time including at least one contrast time point based on the medical image acquired in the image acquisition step, using an image generation model that receives a medical image and generates a contrast effect image depicting a contrast effect; An image generating method comprising: [Method 2] a learning step of learning an image generation model that uses learning data including a medical image group, a contrast image group related to the medical image group, and an imaging condition group related to the contrast image group, the imaging condition group including an imaging time including at least one imaging time point, to generate a contrast effect image in which a contrast effect corresponding to the contrast time is depicted based on the medical image when a medical image related to the medical image group and the contrast time are input. A learning method comprising: [Program 1] A program for causing a computer to function as each unit of the image generating device according to any one of configurations 1 to 16. [Explanation of symbols]

[0233] 1: image generating system, 10: photographing device, 20: image generating device, 30: network, 210: NW (network) interface, 220: input interface, 230: display, 240: memory circuit, 250: processing circuit, 251: image acquisition unit, 252: output unit, 253: display unit

Claims

1. an image acquisition unit for acquiring medical images; a condition acquisition unit that acquires information including at least one injection time point as a condition; an output unit that uses an image generation model that receives a medical image as an input and generates a contrast effect image that depicts a contrast effect corresponding to the contrast time, and outputs a contrast effect image that depicts a contrast effect corresponding to the contrast time, based on the medical image acquired by the image acquisition unit and the contrast time acquired by the condition acquisition unit; An image generating device comprising:

2. The condition includes information on a plurality of different imaging times, The output unit outputs a plurality of the contrast effect images corresponding to the plurality of different contrast enhancement times.

2. The image generating device according to claim 1.

3. The output unit outputs a moving image in which the plurality of contrast effect images are used as frame images.

3. The image generating device according to claim 2.

4. the image generation model has a function of generating the contrast effect image by inputting the medical image and the contrast enhancement time, The image generation model is a model trained using training data including a medical image group related to the medical image, a contrast-enhanced image group related to the medical image group, and an imaging condition group related to the contrast-enhanced image group.

2. The image generating device according to claim 1.

5. The image generation model is a model trained based on semantic regions, which are regions in an image included in the training data and can be divided according to the manner in which the image is depicted or information related to the image, and during training, parameters included in the image generation model are updated based on the type of the semantic region.

5. The image generating device according to claim 4.

6. The learning data includes, as the contrast image group, a group of contrast images acquired over time from the same examination subject.

5. The image generating device according to claim 4.

7. A pair of a medical image and a contrast-enhanced image acquired from the same examination subject included in the training data is anatomically aligned.

5. The image generating device according to claim 4.

8. The group of contrast images included in the training data includes more contrast images taken at contrast times including a contrast time that the operator wants to observe than contrast images taken at contrast times including other contrast times.

5. The image generating device according to claim 4.

9. The condition acquisition unit acquires the conditions further including information other than the contrast time, The output unit has a plurality of the image generation models, selects an appropriate image generation model from the plurality of image generation models based on information other than the contrast injection time, and outputs the contrast effect image based on the medical image and the conditions using the selected image generation model.

2. The image generating device according to claim 1.

10. The image generation model is a trained model trained using training data including a medical image group including a plurality of medical images, a contrast image group including a plurality of contrast images corresponding to the medical image group and acquired over time, and an imaging condition group including contrast times corresponding to the contrast image group, The contrast-enhanced image group includes an interpolated image generated based on the contrast-enhanced image.

5. The image generating device according to claim 4.

11. The learning data is augmented based on effective pixel regions in the contrast-enhanced images acquired from the same examination subject included in the learning data.

5. The image generating device according to claim 4.

12. The medical image is an ophthalmoscopy image.

2. The image generating device according to claim 1.

13. The medical image is a radiological image.

2. The image generating device according to claim 1.

14. a learning unit that uses learning data including a medical image group, a contrast image group related to the medical image group, and an imaging condition group related to the contrast image group, the imaging condition group including an enhancement time including at least one enhancement time point, to learn an image generation model that generates a contrast effect image depicting a contrast effect corresponding to the enhancement time based on the medical image when the medical image related to the medical image group and the enhancement time are input; An image generating device comprising:

15. an image acquisition step of acquiring a medical image; an output step of outputting a contrast effect image depicting a contrast effect corresponding to an enhancement time including at least one enhancement time point based on the medical image acquired in the image acquisition step, using an image generation model that receives a medical image as an input and generates a contrast effect image depicting a contrast effect; An image generating method comprising:

16. a learning step of learning an image generation model that generates a contrast effect image depicting a contrast effect corresponding to the contrast time based on a medical image when the medical image related to the medical image group and the contrast time are input, using learning data including a medical image group, a contrast image group related to the medical image group, and an imaging condition group related to the contrast image group, the imaging condition group including an imaging time including at least one imaging time point; A learning method comprising:

17. A program for causing a computer to function as each unit of the image generating device according to any one of claims 1 to 14.

Images