Learning system, learning method, inference system, inference method, and program

Abstract

To provide an image generation model capable of depicting a mode of a partial region such as a mode of a relatively small region with respect to the whole image and a mode of a specific region.SOLUTION: A training system according to an aspect of the present invention is a training system that trains an image generation model that outputs a generated image corresponding to an input image and having a sparkle point area, the training system including a training data acquisition unit that acquires training data including the input image and a correct image having the sparkle point area corresponding to the input image, a generated image acquisition unit that inputs the input image to the image generation model and acquires a generated image, a sparkle point image acquisition unit that acquires a first sparkle point image including the sparkle point area based on the correct image and acquires a second sparkle point image corresponding to the first sparkle point image based on the generated image obtained by inputting the input image to the image generation model, and an update unit that updates the image generation model based on an error between the first sparkle point image and the second sparkle point image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosure of this specification relates to a learning system, a learning method, an inference system, an inference method, and a program. [Background technology]

[0002] In recent deep learning technologies, methods have been proposed for generating images that mimic images in a different domain while preserving the structural features of images in a given domain. For example, Non-Patent Document 1 below proposes a method for inputting a retinal fundus image without using a contrast agent and outputting a generated image resembling a fluorescein fundus angiography (FA) examination image. Patent Document 1 also proposes a method for inputting a fundus examination image and outputting an image that reproduces an abnormal region generated based on a contrast examination image. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Alireza Tavakkoli, Sharif Amit Kamran, Khondker Fariha Hossain, Stewart Lee Zuckerbrod, “A novel deep learning conditional generative adversarial network for producing angiography images from retinal fundus photographs.”, Sci Rep 10, 21580(2020),<https: / / doi.org / 10.1038 / s41598-020-78696-2> (Released December 9, 2020) [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-180466 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the techniques disclosed in Non-Patent Document 1 and Patent Document 1 may not be able to accurately depict the state of some regions, particularly the state of a relatively small region relative to the entire image or the state of a specific region. [Means for solving the problem]

[0006] The learning system of the present invention is a learning system that trains an image generation model that outputs a generated image having a bright spot region corresponding to an input image, and is equipped with a teacher data acquisition means that acquires teacher data including an input image and a correct answer image having one or more bright spot regions corresponding to the input image, a generated image acquisition means that inputs the input image into the image generation model and acquires a generated image, a bright spot image acquisition means that acquires a first bright spot image including at least one of the bright spot regions based on the correct answer image, and acquires a second bright spot image corresponding to the first bright spot image based on the generated image obtained by inputting the input image into the image generation model, and an update means that updates the image generation model based on the error between the first bright spot image and the second bright spot image. [Effects of the Invention]

[0007] It is possible to obtain an image generation model that can depict the appearance of a portion of an image, such as the appearance of a relatively small area relative to the entire image or the appearance of a specific area. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the configuration of a learning system according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing an example of training data according to the present invention. [Figure 3] FIG. 2 is a diagram showing types of regions according to the present invention. [Figure 4] FIG. 10 is a diagram showing an example of a leakage region according to the present invention. [Figure 5] FIG. 10 is a diagram showing an example of bright spot area information (coordinates of a rectangular area) according to the present invention. [Figure 6] FIG. 1 is a diagram showing an example of the configuration of a network model according to the present invention. [Figure 7] FIG. 3 is a diagram showing an example of types of loss according to the first embodiment. [Figure 8] 4 is a flowchart showing a learning process of an image generation model according to the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of foreground and background settings according to Modification 1 of the first embodiment. [Figure 10] FIG. 10 is a diagram showing the configuration of a learning system according to a second embodiment. [Figure 11] 10 is a flowchart showing a learning process of an image generation model according to the second embodiment. [Figure 12] FIG. 10 is a diagram showing the configuration of an inference system according to a third embodiment. [Figure 13] 10 is a flowchart illustrating an inference process of an image generation model according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] For ease of understanding, the following description and drawings will be directed to two-dimensional images, but the following embodiment can handle not only two-dimensional images but also three-dimensional images.

[0011] In addition, in medical images used in the description of the embodiments, generally, areas where blood vessels or contrast effects are present are often displayed brighter than other areas, but depending on the imaging device, the settings of the imaging device, and the display device, they may be displayed darker. However, for ease of understanding, in the following description, it is assumed that areas where blood vessels or contrast effects are present are recorded so as to be displayed brightly on the image. Specifically, bright pixels have high pixel values, and dark pixels have low pixel values.

[0012] Furthermore, the term "bright spot area" used in the description of the embodiment refers to a connected area that is brighter than its surroundings (i.e., has a pixel value that is a predetermined pixel value or higher than the surrounding pixel values) and is relatively small (less than a predetermined maximum area / volume) relative to the entire image.

[0013] For ease of understanding, the term "bright spot area" in the description of the embodiments refers to a connected area in which the subject's appearance is depicted, and does not refer to a connected area in which noise caused by the imaging device, image processing, etc. is depicted, unless otherwise specified.

[0014] Furthermore, a process may be performed to distinguish the connected region in which the "bright spot region" is depicted from a depiction similar to so-called bright spot noise on the image. Specifically, the "bright spot region" may be not only a connected region having a predetermined maximum area / maximum volume or less, but also a connected region having a predetermined minimum area / minimum volume or more, that is, a connected region having an area / volume within a predetermined range.

[0015] One of the problems that the embodiments disclosed in this specification and the drawings aim to solve is to improve the visualization performance of features in generated images, for example, to improve the visualization performance of contrast agent leakage regions in generated images of medical images.

[0016] However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of the configurations shown in the embodiments described below can also be positioned as other problems.

[0017] First Embodiment An example of the configuration of a learning system 10 according to this embodiment will be described with reference to FIG.

[0018] 1 is an example, and the number of each device may be changed as desired. Devices not shown in FIG. 1 may also be connected to network 20.

[0019] For example, the learning system 10 includes a network interface 110, a memory circuit 120, and a processing circuit 130.

[0020] The NW interface 110 is connected to the processing circuit 130 and controls the transmission and communication of various data between each device connected via the network 20. For example, the NW interface 110 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0021] The memory circuitry 120 is connected to the processing circuitry 130 and stores various data. The memory circuitry 120 also stores various programs that are read and executed by the processing circuitry 130 to realize various functions. For example, the memory circuitry 120 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, or the like.

[0022] The processing circuitry 130 controls the operation of the entire learning system 10. The processing circuitry 130 has, for example, a teacher data acquisition function 131, a generated image acquisition function 132, a bright spot image acquisition function 133, and an update function 134. In this embodiment, each processing function that is a component of the processing circuitry 130 is stored in the storage circuitry 120 in the form of a program executable by a computer. The processing circuitry 130 is a processor that realizes the function corresponding to each program by reading and executing the program from the storage circuitry 120. In other words, when each program has been read, the processing circuitry 130 has each processing function shown in the processing circuitry 130 of FIG. 1.

[0023] In FIG. 1, each function is described as being realized by a single processor. The functions are, for example, a teacher data acquisition function 131, a generated image acquisition function 132, a bright spot image acquisition function 133, and an update function 134. However, the processing entity does not have to be a single processor; a processing circuit 130 may be configured by combining multiple independent processors, and each processor may execute a program to realize each processing function. In addition, in FIG. 1, a single storage circuit such as the storage circuit 120 is described as storing a program corresponding to each processing function, but multiple storage circuits may be distributed and the processing circuit 130 may read corresponding programs from the individual storage circuits.

[0024] The term "processor" used in the above description refers to a circuit such as a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). The processor realizes each processing function by reading and executing a program stored in the memory circuit 120. Note that instead of storing the program in the memory circuit 120, the program may be configured to be directly embedded in the processor circuit. In this case, the processor realizes each processing function by reading and executing the program embedded in the circuit.

[0025] A data set is stored in the memory circuitry 120. Alternatively, the data set may not be stored in the memory circuitry 120, but may be acquired by the learning system 10 from a system (not shown) via the network 20 and used.

[0026] A dataset is composed of one or more sets of training data, each set consisting of a pair of an input image and a correct image. The input image according to this embodiment is an OCTA (OCT Angiography) image acquired by an OCT (Optical Coherence Tomography) device, as shown in Im101 of FIG. 2 . The correct image is an FA image (fluorescein fundus angiography image), as shown in FA image Im102 of FIG. 2 , acquired by capturing the same subject as the corresponding input image, the OCTA image. The FA image is assumed to have its anatomical structure aligned (deformed) with respect to the corresponding input image, the OCTA image. The correct image may also be a segment image showing a contrast agent leakage region, as depicted in the FA image. The contrast duration of the FA images included in the dataset may be limited to a predetermined range, such as 60 to 70 seconds, to reduce the variation in the contrast effect of the correct images, resulting in a stable contrast effect expressed in the generated images. Furthermore, alignment is achieved by manually processing images or performing image registration processing, for example, by deforming the FA image so that the vascular structure in the OCTA image and the vascular structure in the FA image approximately coincide with each other.

[0027] Here, terms relating to FA images and derived data used in the description of each embodiment including this embodiment will be described with reference to FIGS. 3 and 4. FIG.

[0028] The region in an FA image where blood vessels are depicted is called the vascular region. The vascular region can be determined manually or by image segmentation processing on the FA image. A segmented image in which the vascular region is the foreground and the non-vascular region is the background is called a vascular region image. For example, the vascular region image corresponding to FA image Im102 is an image like Im2011 in Figure 3. Note that in FA images, the vascular region and the contrast agent leakage region overlap, making it difficult to identify the vascular region. However, the vascular region can be roughly determined by estimating it from the vascular course in other regions, by referencing images of the same subject acquired by a fundus camera or OCT device, or by using a known vascular extraction algorithm.

[0029] A bright region in an FA image where contrast medium has leaked out without following the blood vessel path is called a leakage region. A leakage region can be determined, for example, by manually determining the FA image, by image segmentation processing, or by performing image processing that primarily removes blood vessel regions from the FA image.

[0030] Furthermore, a segmented image in which the leakage region is the foreground and the rest of the image is the background is called a leakage region image. For example, the leakage region image corresponding to FA image Im102 is an image like Im2021 in Figure 3, in which seven independent, i.e., connected, leakage regions (regions Re2021 to Re2027 in Figure 4) can be seen.

[0031] Among the leakage regions in FA images, the areas that appear as small, locally bright bright spots due to the presence of capillary pools, etc., are called bright spot regions.

[0032] The bright spot area can be determined, for example, by a person manually determining it on the FA image, by an image segmentation process, or by selecting an area from the leakage area that has a predetermined range of area (or volume if a three-dimensional image is being used).

[0033] In addition, a segmented image in which the bright spot regions are the foreground and the regions other than the bright spot regions are the background is called a bright spot region image. For example, the bright spot region image corresponding to the FA image Im102 is an image like Im2031 in Figure 3, in which four independent, i.e., connected, bright spot regions (regions Re2031 to Re2034 in Figure 4) can be seen.

[0034] For the purpose of explanation, leakage regions in an FA image other than bright spot regions may be referred to as large leakage regions. Large leakage regions can be determined, for example, in the same way as bright spot regions, by manually determining them in an FA image, by image segmentation processing, or by selecting a region with a predetermined area from the leakage region.

[0035] In addition, a segmented image in which the large leakage region is the foreground and the rest of the image is called a large leakage region image. For example, the large leakage region image corresponding to the FA image Im102 is an image like Im2041 in Figure 3, in which three independent, i.e., connected, large leakage regions (regions Re2045 to Re2047 in Figure 4) can be seen.

[0036] A bright spot region is at least a part of a connected region in an image that has pixel values ​​equal to or greater than a predetermined value or equal to or less than a predetermined value and has a size within a predetermined range. The bright spot region may be acquired using a bright spot image acquisition function, which will be described later. A bright spot region is an area of ​​a leakage region image originating from a capillary reservoir that is smaller than a predetermined value (area).

[0037] The dataset according to this embodiment further includes bright spot region information, which is information relating to the coordinates of rectangular regions containing each bright spot region group of the FA image, linked to each FA image included in the dataset. The rectangular region corresponding to the bright spot region information (coordinates of the rectangular region) contains at least one bright spot region. The rectangular region is set taking into consideration the area ratio or volume ratio when the bright spot region is the foreground and the region other than the bright spot region is the background. Specifically, the coordinates may be adjusted so that the area ratio (volume ratio when a three-dimensional image is targeted) between the foreground and the background is approximately 1:1.

[0038] If the background includes an area that is as bright as or brighter than the bright spot area, such as a blood vessel area, the coordinates of the rectangular area may be adjusted to use the area excluding that area as a new background. As a specific example, the following description will be given using the partial bright spot area image Im3031 in Figure 5, which is an enlarged view of the area Re2031 in the bright spot area image Im2031 in Figure 4.

[0039] The partial bright spot region image Im3031 includes the bright spot region Re2031, and a rectangular region Rect3031 is set to include the bright spot region Re2031. Within the rectangular region Rect3031, the white part that is the bright spot region Re2031 is the foreground, and the other black part is the background, and the coordinates of the rectangular region Rect3031 are adjusted so that the area ratio between the foreground and the background is approximately 1:1.

[0040] Although there may be many candidates for the coordinates of a rectangular area where the area ratio between the foreground and background is approximately 1:1, it is preferable to automatically determine the coordinates of the rectangular area according to a predetermined rule for the sake of reproducibility, etc. For example, under the condition that the center of gravity of the foreground and the center of gravity of the rectangular area coincide, and the width and height of the rectangle coincide, the coordinates of the rectangular area where the area ratio between the foreground and background is closest to 1:1 may be adopted.

[0041] Also, for example, the coordinates of the rectangular area in which the area ratio between the foreground and background is approximately 1:1 may be adopted while sequentially expanding the rectangular area by moving the leftmost coordinate, topmost coordinate, rightmost coordinate, and bottommost coordinate to identify the rectangular area from the state of the smallest rectangular area containing the bright spot area.

[0042] The above-described processing may be executed by the bright spot image acquisition function 133 of the learning system 10 as processing for generating bright spot region information from acquired teacher data.

[0043] That is, the bright spot image acquisition function 133 constituting the learning system 10 generates bright spot region information based on the ratio between the foreground and the background. Specifically, a rectangular region may be set in the FA image so that the area ratio between the foreground and the background is 1:1. In other words, the bright spot image acquisition function is characterized by determining the sizes of the first and second bright spot images based on the relationship between the size of the bright spot region depicted in the first bright spot image and the size of the background region, which is at least a portion of the region other than the bright spot region of the first bright spot image.

[0044] In addition, if moving a certain coordinate for identifying a rectangular area causes the rectangular area to extend outside the image space, the bright spot image acquisition function 133 may move other coordinates to prevent the rectangular area from extending outside the image space, or may stop moving the coordinates.

[0045] In this embodiment, the execution details of the learning process for the learning system 10 will be described, in which a rectangular region is used as the shape of the region of interest including the bright spot region, but as long as there is no contradiction, the region of interest may be another shape, such as an elliptical region or a triangular region. In this case, the bright spot region information may include coordinates indicating the shape of the region of interest.

[0046] Next, the functional configuration of the learning system 10 will be described in detail with reference to Fig. 1. Here, the learning system 10 is a system that performs learning of an image generation model that outputs a generated image having a bright spot region corresponding to an input image.

[0047] The learning system 10 according to this embodiment is a learning system for training an image generation model that outputs a generated image having a bright spot region corresponding to an input image. The learning system 10 includes a training data acquisition function 131 that acquires training data including an input image and a ground truth image having one or more bright spot regions corresponding to the input image. The learning system 10 also includes a generated image acquisition function 132 that inputs the input image into the image generation model and acquires a generated image. The learning system 10 also includes a bright spot image acquisition function 133 that acquires a first bright spot image including a bright spot region based on the ground truth image, and acquires a second bright spot image corresponding to the first bright spot image based on the generated image obtained by inputting the input image into the image generation model. The learning system 10 further includes an update function 134 that updates the image generation model based on the error between the first bright spot image and the second bright spot image.

[0048] By using the training data and performing each of the above-described functional steps, an image generation model is trained that outputs a generated image having a bright spot region corresponding to the input image. In this embodiment, the learning system 10 is configured as described above, and is therefore able to train an image generation model that can plausibly depict the appearance of a portion of an image, such as the appearance of a relatively small region relative to the entire image or the appearance of a unique region. The following is a detailed explanation of each functional configuration.

[0049] The teacher data acquisition function 131 acquires teacher data including an input image and a correct answer image from the memory circuitry 120 or the like. For ease of understanding, it is assumed that the size of the input image (number of pixels indicating width and height) and the size of the correct answer image (number of pixels indicating width and height) are the same. The teacher data acquisition function 131 is an example of a teacher data acquisition unit. Here, the correct answer image is an image having one or more bright spot regions corresponding to the input image.

[0050] The generated image acquisition function 132 outputs a generated image in which a contrast effect is depicted based on the features depicted in the input image. For ease of understanding, it is assumed that the size of the input image (number of pixels indicating width and height) and the size of the generated image (number of pixels indicating width and height) are the same. The generated image acquisition function 132 is an example of a generated image acquisition unit.

[0051] Specifically, the generated image generated by the image generation model according to this embodiment is a pseudo contrast-enhanced image resembling an FA image in which a contrast effect is depicted, such as that acquired in an FA examination. More specifically, the generated image acquisition function 132 has an image generation model 1320 that receives an OCTA image as input data and outputs an FA image-resembling generated image in which a contrast effect is depicted based on the anatomical features depicted in the OCTA image.

[0052] An image generation model is a model that includes an image processing system that outputs a generated image using, for example, a rule base or machine learning (particularly, deep learning technology). As a specific example, an image generation model that includes an image processing system using deep learning technology will be explained using Figure 6.

[0053] Figure 6 shows an example in which an image generation model 1320 includes a U-Net type network model 1321 as an image processing system using deep learning technology. Here, U-Net is a well-known encoder-decoder type network model with a skip connection mechanism.

[0054] Once U-Net has been sufficiently trained on a dataset consisting of a group of paired images, each consisting of an input image and a corresponding output image, it can output a plausible image corresponding to the input image according to the trends of the dataset used for training.

[0055] For example, U-Net is known to be applicable to image segmentation, image quality improvement, image domain transformation, etc., depending on the dataset.

[0056] The image generation model 1320 converts the input image Im101 into a tensor and inputs it to the network model 1321, and causes the network model 1321 to output the tensor as the output image Im102.

[0057] Here, the tensor in the description of this embodiment refers to a format in which a group of pixel values ​​of an image, etc., is expressed as a multidimensional array, and is a data input / output format for the network model, and images and tensors can be converted to each other. Also, although U-Net is used as an example in this embodiment, other network models that can achieve the same purpose may be adopted.

[0058] The bright spot image acquisition function 133 acquires a first bright spot image group, which is a first partial image group of the correct image, and a second bright spot image group, which is a second partial image group of the generated image. The bright spot image acquisition function 133 is an example of a bright spot image acquisition unit. That is, the bright spot image acquisition function 133 is characterized by acquiring a first partial image included in the correct image as the first bright spot image, and acquiring a second partial image positionally corresponding to the first partial image from the generated image as the second bright spot image.

[0059] Specifically, the bright spot image acquisition function 133 refers to each group of bright spot area information (coordinates of rectangular areas) associated with the correct image included in the data set, and acquires a first partial image that matches the coordinates of the rectangular area in the correct image as the first bright spot image.

[0060] In addition, the bright spot image acquisition function 133 also performs the same for the generated image by referring to each group of bright spot area information (coordinates of rectangular areas) linked to the corresponding correct image, and acquires a second partial image in the generated image that matches the coordinates of the rectangular area as a second bright spot image.

[0061] That is, the bright spot image acquisition function 133 acquires a group of partial images at the same coordinates from both the correct image and the generated image as a bright spot image group. Note that in this embodiment, an example will be described in which a plurality of bright spot image groups are acquired from each of the correct image and the generated image, but if the number of bright spot region information associated with the correct image is one, only one bright spot image will be acquired.

[0062] Furthermore, if the coordinates of the rectangular area that is the bright spot area information match the entire coordinate space of the associated correct answer image, the entire correct answer image is acquired as the bright spot image.

[0063] Next, a case will be described in which the coordinates of the rectangular area that is the bright spot area information protrude from the entire coordinate space of the associated correct image. In this case, the bright spot image acquisition function 133 acquires the bright spot image by padding the protruding partial area with pixel values ​​of the correct image or filling it with a predetermined value so that the size of the acquired bright spot image matches the size of the rectangular area.

[0064] Furthermore, since the group of bright spot images obtained from the correct image is not changed during the learning process performed by the learning system 10 of this embodiment, the bright spot image acquisition function 133 may acquire images that have been previously included in the data set as static data.

[0065] The bright spot image acquisition function 133 may further adjust the pixel values ​​of the acquired bright spot image to adjust the sensitivity to the error between the bright spot image of the correct image calculated by the update function 134 and the bright spot image of the generated image in step S14 described below.

[0066] Specifically, the bright spot image acquisition function 133 may perform Min-Max normalization on each bright spot image, add a predetermined bias value, multiply by a weight value to vary the pixel values, and then limit (clip) the range of pixel values.

[0067] In other words, the bright spot image acquisition function 133 may adjust the error calculated by the update function 134 in step S14 described below by processing the bright spot image, and as a result, adjust the degree to which the parameters constituting the network model 1321 are updated.

[0068] In the description of this embodiment, the degree to which the parameters constituting the network model are updated refers to a so-called loss. The loss is, for example, the average value of a group of values ​​indicating errors for an output data group output by the network model based on one or more sets of training data groups (mini-batches) selected from a dataset. The loss may also be, for example, a value calculated by weighting a group of values ​​indicating errors for the output data group. The loss may also be, for example, a value calculated by further weighting the loss.

[0069] The update function 134 updates the parameters that make up the network model 1321 and changes the performance of the image generation model 1320. Specifically, the update function 134 updates (optimizes) the parameters that make up the network model 1321 included in the image generation model 1320 so as to reduce the loss described above. The update function 134 is an example of an update unit.

[0070] The learning process executed by the learning system 10 according to this embodiment will be described with reference to the flowchart in Fig. 8. Fig. 8 is a flowchart showing an example of the learning process of the image generation model 1320 executed by the learning system 10 according to this embodiment.

[0071] For ease of understanding, the procedure for updating the parameters constituting the network model 1321 using one set of training data and executing the learning process (i.e., a situation where the mini-batch size is 1) will be described. However, in the actual learning process performed by the learning system 10, the learning process may be executed using multiple sets of training data for the purpose of shortening the time required to complete the learning process, stabilizing the learning process, etc.

[0072] In step S11, the training data acquisition function 131 acquires an input image, which is an OCTA image, and a correct image, which is an FA image, included in a set of training data that constitutes a data set.

[0073] In step S12, the generated image acquisition function 132 inputs the input image acquired in step S11 into the image generation model 1320, and acquires a generated image resembling an FA image in which the contrast effect is depicted based on the features depicted in the input image.

[0074] Specifically, as shown in FIG. 7, when an OCTA image, which is an input image constituting the training data, is input as an input tensor Te101 to a network model 1321, an output tensor Te103, which is a generated image similar to an FA image, can be obtained.

[0075] In step S13, the bright spot image acquisition function 133 acquires a first bright spot image group of the correct image and a second bright spot image group of the generated image.

[0076] 7, the bright spot image acquisition function 133 acquires first bright spot images Te1021-Te1024 from the correct tensor Te102, which is the correct image, by referring to each of the first bright spot region information groups associated with the correct image. Also, from the output tensor Te103, which is a generated image like an FA image, it acquires second bright spot images Te1031-Te1034 by referring to each of the second bright spot region information groups associated with the corresponding correct images.

[0077] In step S14, the update function 134 updates the parameters constituting the network model 1321 so as to reduce the error between the first bright spot image of the correct image and the second bright spot image of the generated image that correspond to each other in the image space.

[0078] 7, the update function 134 first acquires a first bright spot image Te1021 obtained from the correct tensor Te102, which is a correct image derived from the common bright spot region information. The update function 134 then acquires a second bright spot image Te1031 obtained from the output tensor Te103, which is a generated image, and calculates the error value between the first bright spot image Te1021 and the second bright spot image Te1031.

[0079] Next, the update function 134 similarly calculates error values ​​for the first bright point image Te1022 and the second bright point image Te1032, the first bright point image Te1023 and the second bright point image Te1033, and the first bright point image Te1024 and the second bright point image Te1034. The update function 134 uses each of the calculated error value groups as losses Lo1011 to Lo1014 for updating the parameters that make up the network model 1321.

[0080] At this time, the update function 134 may use the average value of the losses Lo1011 to Lo1014 as the loss, for reasons such as ensuring that the degree to which the parameters constituting the network model 1321 are updated is not affected by the number of brightness regions in the correct image.

[0081] Note that the update function 134 may use pixel values ​​between spatially corresponding pixels to calculate the error value. For example, it may use a function that calculates a large value when the error is large and a small value when the error is small by applying an index such as mean square error, mean absolute error, or SSIM (Structural Similarity).

[0082] For example, when applying the mean square error or mean absolute error, the update function 134 can simply use the index value as is, but when applying an index such as SSIM that becomes large when the error is small, the SSIM value can be multiplied by a negative value to obtain the error value.

[0083] Furthermore, the update function 134 may be configured to compare processed images of each bright spot image, rather than directly comparing the bright spot images of the correct image and the generated image. For example, the update function 134 may be configured to generate processed images by applying a filter process to each bright spot image to emphasize the bright spots, and then evaluate the error of the processed images. This allows for the definition of a loss that emphasizes the reproduction of bright spots.

[0084] Furthermore, it is preferable that the update function 134 takes into account not only the error between the bright spot images but also the error between the correct image and the generated image, and adjusts the areas other than the bright spot areas in the generated image, as well as the overall brightness and contrast of the generated image, so that they resemble the correct image.

[0085] 7, the update function 134 also calculates an error value between the correct tensor Te102, which is the correct image, and the output tensor Te103, which is the generated image. The update function 134 then uses the calculated error value as the loss Lo101 for updating the parameters that configure the network model 1321.

[0086] At this time, the update function 134 may multiply a predetermined weight value by the loss based on the error between the group of bright spot images of the correct image and the group of bright spot images of the generated image (Lo1011 to Lo1014 in Figure 7), and the loss based on the error between the correct image and the generated image (Lo101 in Figure 7).

[0087] That is, the update function 134 may adjust the degree to which each of these weight values ​​influences the update of the parameters that make up the network model 1321 by multiplying them.

[0088] As shown in the correct tensor Te102 in Fig. 7, a mask region (black region) exists in the peripheral region of the FA image, which is the correct image. Therefore, when the loss Lo101 that reflects the error with the output tensor Te103 based on this mask region is used, the generated image output by the network model 1321 is also trained so that the peripheral region is depicted in black like the mask region.

[0089] To avoid the learning tendency described above, a loss Lo101 that ignores the error with the output tensor Te103 based on this masked region may be used. In other words, the error value may be calculated between a pixel group in the unmasked region of the correct image and a pixel group in the generated image that corresponds to the unmasked region in image space, and used as the loss.

[0090] In the above case, the update function 134 can learn to prevent the entire image, including the peripheral area, of the generated image output by the network model 1321 from depicting a mask area similar to that of the correct image. In other words, the update function 134 is characterized by reducing the influence of some errors on the error based on the size of the first bright spot area and the size of the background area. Specifically, the update function 134 is characterized by not calculating errors for some areas in the first bright spot image based on the size of the bright spot area and the size of the background area, thereby reducing the influence of the error.

[0091] In step S15, the learning system 10 determines whether the generated image satisfies predetermined conditions. Specifically, the learning system 10 evaluates the quality of the generated image using training data for verification that is not used in step S11. When the image generation model 1320 has been sufficiently learned by the learning system 10, that is, when it is determined that the quality of the generated image is sufficiently high, the learning process may be stopped early.

[0092] The learning process of the network model 1321 can be advanced by repeating the series of processes from steps S11 to S15 a predetermined number of times while changing the training data in step S11.

[0093] In this embodiment, the update function 134 is a learning processing system that updates parameters constituting the network model 1321 included in the image generation model 1320 based on the error between the first group of bright spot images of the correct image and the second group of bright spot images of the generated image. However, other learning processing systems may be employed.

[0094] For example, the update function 134 may update parameters constituting the network model 1321 by applying a technique related to GAN that uses an image as input, such as Conditional GAN ​​(Generative Adversarial Network), which is a well-known deep learning technique, and execute a learning process. That is, the update function 134 may update parameters constituting the network model 1321 while having a Discriminator Network determine whether or not a group of bright spot images of generated images generated by the network model 1321, which corresponds to a Generator Network in a Conditional GAN, are similar to real bright spot images (FA images that are correct images).

[0095] The image generation model 1320 trained by the learning process executed by the learning system 10 according to the present embodiment can output an OCTA image based on the trends of the training data used in the learning process when an OCTA image is input. This learning process enables the image generation model 1320 to output an FA-like generated image that plausibly depicts even small bright spot regions that have previously been difficult to depict.

[0096] <Modification 1 of the First Embodiment> It has been explained that the rectangular area corresponding to the bright spot area information in the first embodiment contains at least one bright spot area by the bright spot image acquisition function 133, etc., and the coordinates are adjusted so that the area ratio between the bright spot area in the foreground and the area other than the bright spot area in the background is approximately 1:1.

[0097] However, in the method described in the first embodiment, depending on the state of the bright spot regions, it may be difficult to make the area ratio approximately 1:1.

[0098] As a specific example, one example will be described using FIG. 9. Partial image Im401a is a part of the first bright spot region image of the FA image. An upright (non-rotated) minimum rectangle Rect401a that contains the bright spot region Re401 is set for partial image Im401a. It can be seen that in partial image Im401a, the area other than the bright spot region, which is the background, is significantly larger than the bright spot region, which is the foreground. In this state, even if the coordinates of the rectangular region are adjusted using the method described in the first embodiment, it is not possible to make the area ratio between the foreground and the background approximately 1:1.

[0099] To cope with this situation, the bright spot image acquisition function 133 may set a rotated rectangular area Rect401b shown in the partial image Im401b, thereby making the area ratio of the foreground to the background approximately 1:1. However, because the rectangular area 4012 is rotated with respect to the image space coordinate system, the upright bright spot image acquired by the bright spot image acquisition function 133 will include interpolated pixels.

[0100] As another method, when calculating the error value between the bright spot image of the correct image and the bright spot image of the generated image, the bright spot image acquisition function 133 may set a disregard area in which the error is not calculated by the update function 134. Specifically, the bright spot image acquisition function 133 sets a disregard area IgRe401c (hatched area) in a rectangular area Rect401c in the partial image Im401c.

[0101] At this time, the background region for which the error is calculated by the update function 134 is the region (black region around the bright spot region) obtained by excluding the bright spot region Re401 and the ignore region IgRe401c, which are the foreground, from the rectangular region Rect401c. Here, the outer contour of the background region matches the rotated rectangular region Rect401b, that is, the area ratio between the foreground and the background is approximately 1:1.

[0102] As another example of setting an ignored area in which no error is calculated using the bright spot image acquisition function 133, an ignored area IgRe401d (hatched area) is set in the rectangular area Rect401d in the partial image Im401d. In this case, the background area in which the error is calculated using the update function 134 is a free area that includes the bright spot area Re401. In this case, the bright spot image acquisition function 133 also expands and adjusts the background area so that the area ratio between the foreground and the background is approximately 1:1. One example of an expansion method using the bright spot image acquisition function 133 is to adjust the distance from the outline of the bright spot area Re401 to the outer outline of the background area as a margin.

[0103] In the method for setting an ignored area in which errors are not calculated, the bright spot image acquisition function 133 includes in the bright spot area information not only information about the coordinates of a rectangular area containing the bright spot area, but also coordinate information of a pixel group to be set as the ignored area (or a segment image from which equivalent information can be obtained). Furthermore, in step S14 according to the first embodiment, the update function 134 does not calculate an error in the ignored area by referring to the coordinate information of the pixel group to be set as the ignored area, which is included in the bright spot area information. Alternatively, the error corresponding to the ignored area is not reflected in the loss.

[0104] As mentioned above, when the bright spot image acquisition function 133 sets the background other than the leakage region in the correct image, the background may include a region with high pixel values ​​similar to the bright spot region, such as a blood vessel region. In such a case, it is preferable that the bright spot image acquisition function 133 sets in advance the region having characteristics similar to the aspect for which the imaging performance is to be improved as an ignorance region for which no error is calculated, and then performs the above procedure.

[0105] As a result, the learning process can be carried out in a state where the area ratio between the bright spot region, which is the foreground, and the region other than the bright spot region, which is the background, is approximately 1:1.

[0106] <Modification 2 of the First Embodiment> In the above embodiment, a configuration in which one partial image contains one bright spot area has been described, but a configuration in which one partial image contains multiple bright spot areas may also be used. Furthermore, the bright spot image acquisition function 133 may be configured to set only one partial image that contains all the bright spot areas for each input image, or may be configured to treat the entire range of the input image as equivalent to the above partial image without creating a partial image.

[0107] In this case, similar to Modification 1, it is desirable that the bright spot image acquisition function 133 sets a disregard area where no error is calculated so that the area ratio between the foreground and background is approximately 1 to 1. This also allows the learning process to proceed in a state where the area ratio between the bright spot area, which is the foreground, and the area other than the bright spot area, which is the background, is approximately 1 to 1.

[0108] <Modification 3 of the First Embodiment> In the above embodiment, the bright spot image acquisition function 133 is configured to set the foreground to background area ratio for each partial image to be approximately 1:1. However, it is sufficient if the bright spot image acquisition function 133 sets the foreground to background area ratio for the entire partial image to be approximately 1:1. For example, the bright spot image acquisition function 133 may be configured not to set the foreground to background area ratio for each partial image to be approximately 1:1.

[0109] First, the bright spot image acquisition function 133 calculates the sum of the areas of all bright spot regions, Si, for each input image. Next, an appropriate value for the sum of the areas of the partial images is calculated as a constant multiple (usually double) of the sum of the areas, Si. Furthermore, the sum of the areas of the partial images may be divided by the number of partial images, M, to obtain a value, Sm (Sm = 2 × Si ÷ M), which may be set as the area common to all the partial images.

[0110] Furthermore, only when the bright spot area does not fit in the partial image obtained in this way, the size of the partial image may be enlarged so that the bright spot area fits in by the bright spot image acquisition function 133. This also allows the learning process to proceed with an area ratio of approximately 1:1 between the bright spot area in the foreground and the area other than the bright spot area in the background.

[0111] <Fourth Modification of the First Embodiment> In the above embodiment, the configuration has been described in which the area ratio of the foreground to the background is set to approximately 1:1 by dynamically setting the size of the partial image based on each of the input images used for learning using the bright spot image acquisition function 133. However, it is sufficient that the area ratio of the foreground to the background is set to approximately 1:1, and the size of the partial image may not be determined based on each of the input images.

[0112] For example, the bright spot image acquisition function 133 may use a partial region whose size is preset to a predetermined value based on the standard size of the bright spot region, such that the area is a constant multiple (usually twice) of the standard size.

[0113] Here, the size of the partial image may be a fixed value, or only if the bright spot area does not fit, the size of the partial image may be enlarged so that the bright spot area fits in. This also allows the learning process to proceed with an approximately 1:1 area ratio between the bright spot area, which is the foreground, and the area other than the bright spot area, which is the background.

[0114] <Second embodiment> The learning system 100 according to this embodiment omits the adjustment of the coordinates of the rectangular area that is the bright spot area information in the first embodiment (adjustment to make the area ratio of the foreground and background approximately 1:1), and instead performs a learning process to adjust the loss. Note that this does not exclude the combination of the methods described in the first and second embodiments.

[0115] The present embodiment will be described below, but explanations of parts that are the same as those in the first embodiment will be omitted.

[0116] First, an example of the configuration of a learning system 100 according to this embodiment will be described with reference to FIG.

[0117] The processing circuit 1300 of this embodiment has the same configuration as the processing circuit 130 of the first embodiment, and further includes a loss adjustment information acquisition function 135. Furthermore, the update function 134 adjusts the degree to which the image generation model 1321 is updated based on the loss adjustment information.

[0118] The loss adjustment information acquisition function 135 acquires loss adjustment information used to adjust the loss value for updating the parameters constituting the network model 1321, based on information about the luminance distribution of the first bright spot image. Here, the loss adjustment information is specifically a scalar value that weights each loss. For example, the scalar value increases as the area ratio of the foreground to the background in a bright spot image deviates from 1:1, that is, as the balance between the area of ​​the foreground (bright spot region) and the area of ​​the background (other than the bright spot region, other than the bright spot region and blood vessel region, etc.) worsens (the proportion of the foreground is low). The loss adjustment information acquisition function 135 is an example of a loss adjustment information acquisition unit.

[0119] As in the first embodiment, the dataset of this embodiment includes bright spot area information, which is information about the coordinates of rectangular areas that contain each of the bright spot area groups of the FA images, linked to each of the FA images included in the dataset.

[0120] However, the rectangular area corresponding to the bright spot area information (coordinates of the rectangular area) does not have to be coordinates such that when the bright spot area is the foreground and the area other than the bright spot area is the background, the area ratio between the foreground and the background (volume ratio when targeting a three-dimensional image) is approximately 1:1, as in the first embodiment. For example, it can be a rectangle of a preset size. The size of the rectangle can be any size, but can be determined, for example, based on the standard size of a bright spot.

[0121] Next, the learning process executed by the learning system 100 according to this embodiment will be described with reference to FIG.

[0122] Steps S11 to S13 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0123] In step S140, the update function 134, similarly to the first embodiment, acquires losses Lo1011 to Lo1014 and loss Lo101 for updating the parameters that make up the network model 1321. Here, in this embodiment, the loss adjustment information acquisition function 135 further acquires loss adjustment information corresponding to each loss. The update function 134 adjusts losses Lo1011 to Lo1014 using the loss adjustment information, and then updates the parameters that make up the network model 1321.

[0124] The loss adjustment information acquisition function 135 acquires a scalar value, which is loss adjustment information corresponding to a certain loss, by referencing the first bright spot image, which is the correct image used to calculate the error value from which the loss is derived. The loss adjustment information acquisition means 135 acquires information regarding loss adjustment based on at least one of the size of the bright spot region or the size of the background region in the first bright spot image. Here, the information regarding the brightness distribution is a statistical value of the brightness values ​​of the pixel group that constitutes the first bright spot image.

[0125] For example, the loss adjustment information acquisition function 135 can acquire a scalar value α, which is the loss adjustment information, using Equation 1, which uses the area A of the bright spot image of the correct image and the area b of the bright spot region (foreground in this embodiment) in that bright spot image. Note that if the value of the scalar value α exceeds a predetermined value, the loss adjustment information acquisition function 135 may limit the predetermined value as a maximum value.

[0126]

number

[0127] As another example of an acquisition method by the loss adjustment information acquisition function 135, for example, the area A of the bright spot image of the correct image and the area d of the area other than the bright spot area in that bright spot image (the background in this embodiment) may be used. Specifically, the scalar value α, which is the loss adjustment information, can be acquired by Equation 2. Note that, similar to Equation 1, if the value of the scalar value α exceeds a predetermined value, the loss adjustment information acquisition function 135 may limit the value to the predetermined value as a maximum value.

[0128]

number

[0129] Furthermore, the loss adjustment information acquisition function 135 may use other calculation formulas other than the above, such that the lower the proportion of the foreground, the larger α is set.

[0130] As mentioned above, when the background is the area other than the leakage area in the correct image, the background may include an area with high pixel values ​​similar to the bright spot area, such as a blood vessel area. In such a case, it is preferable not to allow the area with similar characteristics to the area for which the visualization performance is desired to be improved to affect the calculation of the scalar value α by the loss adjustment information acquisition function 135. Specifically, the loss adjustment information acquisition function 135 can subtract the area of ​​the area from the area A of the bright spot image and the area d of the area other than the bright spot area, and then calculate α using Equation 2.

[0131] Furthermore, the loss adjustment information acquisition function 135 can acquire a new loss (that is, the adjusted loss) L' by using Equation 3, which uses the loss L to be adjusted and a scalar value α, which is loss adjustment information. (Formula 3) L´=αL Step S15 is the same as in the first embodiment.

[0132] The learning system 100 can proceed with the learning process of the network model 1321 by repeating the series of processes from steps S11 to S15 described above a predetermined number of times while changing the training data in step S11.

[0133] The loss adjustment information acquisition function 135 may calculate α individually for each partial image. Alternatively, a common α may be set for each partial image by using the sum of the areas of all partial images and the sum of the foreground areas for each input image. Either configuration allows for learning processing that emphasizes the reproduction of bright spot regions.

[0134] The image generation model 1320 trained by the learning process executed by the learning system 10 according to the present embodiment can output an OCTA image based on the trends of the training data used in the learning process when an OCTA image is input. This learning process enables the image generation model 1320 to output an FA-like generated image that plausibly depicts even small bright spot regions that have previously been difficult to depict.

[0135] <Modification 1 of the Second Embodiment> In the second embodiment, the scalar value α, which is the loss adjustment information, is calculated using area information about the bright spot image of the correct image. In this modification, the loss adjustment information acquisition function 135 uses statistical information about pixel values ​​about the bright spot image of the correct image to achieve an approximately similar purpose.

[0136] The loss adjustment information acquisition function 135 may use a method in which the scalar value α is increased as the average value of a certain bright spot image deviates from the average pixel value expected when the foreground and background area ratio is 1:1. As a specific example, the loss adjustment information acquisition function 135 may use a minimum value V determined from the range of pixel values ​​of the correct image group. min , maximum value V max The loss adjustment information, scalar value α, can be obtained by Equation 4, which uses the average pixel value μ of the bright spot image of the correct image. In Equation 4, the expected average pixel value when the area ratio between the foreground and background is 1:1 is the minimum value V min and the maximum value V max The intermediate value between (minimum value V min +V max)÷2". Furthermore, similar to the second embodiment, when the value of the scalar value α exceeds a predetermined value, the loss adjustment information acquisition function 135 may limit the value by setting the predetermined value as the maximum value.

[0137]

number

[0138] As mentioned above, when the background is a region other than the leakage region in the correct image, the background may include a region with high pixel values ​​similar to the bright spot region, such as a blood vessel region. In such a case, it is preferable not to allow the region, which has characteristics similar to the mode in which the visualization performance is desired to be improved, to affect the calculation of the scalar value α by the loss adjustment information acquisition function 135. Specifically, when calculating the average pixel value μ by the loss adjustment information acquisition function 135, the pixel values ​​of the region may be ignored, and α may be calculated using Equation 3.

[0139] <Modification 2 of the Second Embodiment> In the above embodiment, a configuration was described in which one partial image contains one bright spot area, but as in variant example 2 of the first embodiment, the bright spot image acquisition function 133 may also be configured to contain multiple bright spot areas in one partial image.

[0140] <Modification 3 of the Second Embodiment> In the above embodiment, the value of α is dynamically set based on the input image used for learning. However, the loss adjustment information acquisition function 135 may be configured to use a fixed value for α. For example, the loss adjustment information acquisition function 135 may calculate the ratio between the area of ​​the partial image and the area of ​​the bright spot region based on the standard size of the bright spot region, and set a fixed value of α according to this ratio. This also allows the learning process to proceed with an emphasis on reproducing the bright spot region.

[0141] <Third embodiment> This embodiment will be described with reference to FIG. 12. In this embodiment, a memory circuit 120 stores an image generation model trained by at least one of the learning systems of the first and second embodiments. The inference system 30 may be configured as the same system by the same entity as the learning system 10, or may be a system in which an image generation model trained by a different entity is acquired via a network 20 or the like. Systems configured from different devices by the same entity are also within the scope of this embodiment.

[0142] The inference system 30 has an inference target image acquisition function 311 that acquires an inference target image from the memory circuitry 120 or the like. Furthermore, the inference system 30 is an inference system that uses a trained image generation model trained by the learning method described in the first or second embodiment. The inference system 30 includes an inference function 312 that performs inference processing on the inference target image using the trained image generation model, and a display control function 313 that displays the inference results on a display unit or the like. The trained image generation model is generated by training using training data that includes an input image and a ground truth image having one or more bright spot regions corresponding to the input image. Specifically, the trained image generation model is a model trained based on the error between a first bright spot image based on the ground truth image and a second bright spot image based on a generated image output from the image generation model by inputting the input image.

[0143] The inference process performed by the inference system 30 according to this embodiment will be described with reference to FIG.

[0144] In step S131, the inference target image acquisition function 311 acquires the inference target image from the storage unit 120 or the like, and then proceeds to the next step.

[0145] In step S132, the inference function 312 acquires an image generation model trained based on the error between a first bright spot image based on the correct image and a second bright spot image based on a generated image output from the image generation model by inputting an input image. The inference function 312 executes inference processing by applying the trained image generation model to the image to be inferred, and then proceeds to the next step.

[0146] In step S133, the display control function 313 causes the display unit or the like to display the inference result acquired by executing the inference function 312, and then ends the inference process.

[0147] In this embodiment, the trained image generation model acquired through the above-described training process can be used in the inference process, thereby obtaining highly accurate inference results that plausibly depict the appearance of a portion of the image, such as the appearance of a relatively small area or a specific area relative to the entire image.

[0148] <Other embodiments> A generated image may be generated from an OCTA image by a learning system 10 equipped with an image generation model having a network model that has completed learning processing according to the above-described embodiment and modified examples, or by another device. The generated image may also be displayed on a display device. The generated image may also be stored in a storage device or the like. Alternatively, the generated image may be further processed and then displayed on a display device or stored in a storage device or the like.

[0149] Furthermore, while the above-described embodiments and modifications have been described with respect to bright spot regions, the "range" in the condition for "connected regions having a predetermined area / volume range" for identifying bright spot regions may be changed. This configuration may improve visualization performance for all leakage regions. Alternatively, it may improve visualization performance for only the large leakage regions described above.

[0150] Although several embodiments have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. [Explanation of symbols]

[0151] 10 Learning System 20 Network 110 Network Interface 120 Memory circuit 130 Processing circuit 131 Teacher data acquisition function 132 Acquisition function of generated images 133 Bright spot image acquisition function 134 Update function 1320 Image Generation Model 1321 Network Model

Claims

1. A learning system that learns an image generation model that outputs a generated image having a bright spot region corresponding to an input image, A training data acquisition means for acquiring training data including an input image and a ground truth image having one or more bright point regions corresponding to the input image; a generated image acquisition means for inputting the input image into the image generation model and acquiring a generated image; A bright spot image acquisition means for acquiring a first bright spot image including at least one of the bright spot regions based on the correct image, and acquiring a second bright spot image corresponding to the first bright spot image based on the generated image obtained by inputting the input image into the image generation model; an updating means for updating the image generation model based on an error between the first bright spot image and the second bright spot image; A learning system that includes:

2. The learning system described in claim 1, characterized in that the bright spot image acquisition means determines the size of the first and second bright spot images based on the relationship between the size of the bright spot area depicted in the first bright spot image and the size of the background area, which is at least a part of the area of ​​the first bright spot image other than the bright spot area.

3. The learning system described in claim 1, characterized in that the update means reduces the influence of errors in some areas on the error based on the size of the bright spot area and the size of the background area in the first bright spot image.

4. The learning system described in claim 3, characterized in that the update means reduces the effect of the error by not calculating the error for some areas in the first bright spot image based on the size of the bright spot area and the size of the background area.

5. further comprising a loss adjustment information acquisition means for acquiring information regarding loss adjustment based on information regarding the luminance distribution of the first bright point image; The learning system according to claim 1 , wherein the update means further adjusts the degree to which the image generation model is updated based on the loss adjustment information.

6. The learning system described in claim 5, characterized in that the loss adjustment information acquisition means acquires information regarding the loss adjustment based on at least one of the size of the bright spot area or the size of the background area in the first bright spot image.

7. 6. The learning system according to claim 5, wherein the information about the luminance distribution is a statistical value of the luminance values ​​of a group of pixels that constitute the first bright-point image.

8. 2. The learning system according to claim 1, wherein the updating means further updates the image generation model based on an error between the correct image and the generated image.

9. A learning system described in any one of claims 1 to 3, characterized in that the bright spot image acquisition means acquires a first partial image included in the correct image as the first bright spot image, and acquires a second partial image positionally corresponding to the first partial image from the generated image as the second bright spot image.

10. The learning system described in claim 1, characterized in that the means for acquiring the bright spot image acquires as the bright spot area at least a portion of a connected area in the image that has pixel values ​​greater than or equal to a predetermined value or less than a predetermined value and has a size within a predetermined range.

11. The learning system according to claim 1 , wherein the bright spot region originates from a capillary pool in the subject of the correct image.

12. The learning system according to claim 11, characterized in that the bright spot image acquisition means acquires an area of ​​the leakage area image originating from the capillary reservoir that is smaller than a predetermined value as the bright spot area.

13. The learning system described in claim 1, characterized in that the acquisition means for the bright spot image adjusts the error calculated by the update means by adjusting the pixel values ​​of the first bright spot image and the second bright spot image.

14. an inference means for performing inference processing using the image generation model trained by the learning system according to any one of claims 1 to 3, further comprising an acquisition means for an inference target image; a display control means for displaying the result of the inference by the inference means; A learning system further comprising:

15. an inference target image acquisition means for acquiring an inference target image; an inference means for performing inference on the inference target image using an image generation model trained based on an error between a first bright spot image based on the correct answer image and a second bright spot image based on a generated image output from the image generation model by inputting the input image, using training data consisting of an input image and a correct answer image having one or more bright spot regions corresponding to the input image; a display control means for displaying the inference result by the inference means on a display unit; An inference system comprising:

16. A learning method for learning an image generation model that outputs a generated image having a bright spot region corresponding to an input image, a training data acquisition step of acquiring training data including an input image and a ground truth image having one or more bright point regions corresponding to the input image; a generated image acquisition means for inputting the input image into the image generation model and acquiring a generated image; a bright spot image acquisition step of acquiring a first bright spot image including at least one of the bright spot regions based on the correct image, and acquiring a second bright spot image corresponding to the first bright spot image based on the generated image obtained by inputting the input image into the image generation model; an updating step of updating the image generation model based on an error between the first bright spot image and the second bright spot image; A learning method that includes:

17. an acquisition step of an inference target image for acquiring an inference target image; an inference step of performing inference on the inference target image using an image generation model trained based on the error between a first bright spot image based on the correct answer image and a second bright spot image based on a generated image output from the image generation model by inputting the input image, using training data consisting of an input image and a correct answer image having one or more bright spot regions corresponding to the input image; a display control step of displaying the inference result by the inference step on a display unit; An inference method comprising:

18. A program for executing the learning method according to claim 16 or the inference method according to claim 17 on a computer.

Images